Upstream Proxying

Upstream proxying and caching allows you to upload and use the packages you own, while Cloudsmith fetches and caches other packages (such as dependencies).

This enables you to use Cloudsmith as a first-class cache and a central source of truth for packages, to protect you from outages of external services (which is especially important when running behind your firewall).

Upstream Concepts

Cloudsmith upstream support centers around several key concepts:

  • Proxying: The act of transparently allowing access to a package that exists on an upstream repository. Package managers see the remote package as one which belongs to the Cloudsmith repository.
  • Caching: An extension to the proxying functionality, where requested packages from an upstream are fetched and permanently stored in your Cloudsmith repository. This helps to ensure package dependencies are always available and helps to protect from upstream outages or security breaches.
  • Indexing: In order to be aware of the packages available from an upstream, Cloudsmith builds an index. This process occurs when an upstream is first added to your Cloudsmith repository and is scheduled for a resync on a regular basis.

Indexing

Index availability is a critical factor for upstream handling on Cloudsmith, helping to ensure deterministic performance for upstream requests and a deeper insight into the availability of packages.

The indexing process can differ, depending on the package format and upstream itself. Where possible, Cloudsmith will determine the availability of all packages on an upstream repository ahead-of-time, which generally means that an upstream repository is unavailable when first added, until this indexing process has occurred.

For package formats that do not maintain a centralized mechanism for retrieval of all packages, Cloudsmith employs a just-in-time indexing mechanism. In this approach, awareness of packages is made the first time a package is successfully cached from an upstream repository. Going forwards, Cloudsmith maintains a list of all versions available on the source upstream for the package and ensures this is kept in sync.

When neither indexing mechanism is available for an upstream, Cloudsmith falls back to a real-time unindexed approach. When requests are made for upstream packages, Cloudsmith determines availability across each upstream in your repository, in real-time. This is the least performant approach.

We strive to ensure that at least just-in-time indexing is available for each package format and upstream source, although this is not always possible.

Priority

When defining upstreams for a repository, a priority can be specified. The priority of an upstream is used to determine the order in which upstream requests are resolved. Cloudsmith evaluates upstreams by the order of 1..n.

A good approach when determining what priority to apply to upstreams is to ensure that the lowest value is specified for the upstream which is most likely to contain upstream packages you request. This helps to improve performance in the event that an upstream source does not support any of our indexing mechanisms.

Supported Formats

Format

Fixed Proxy
Configurable Proxy
Caching
Indexing
Indexing Type

Debian

N/A
Ahead-of-Time

RedHat

N/A
Ahead-of-Time

Gradle

N/A
Just-in-Time

Maven

N/A
Just-in-Time

sbt

N/A
Just-in-Time

npm

N/A

Python

N/A

Cargo

N/A

Dart

N/A
Just-in-Time

NuGet

N/A
Ahead-of-Time

Ruby (Beta)

N/A
Ahead-of-Time

📘

All formats are on the roadmap for Configurable Proxy and Caching.

Create an Upstream Proxy

Click the green "Create Upstream" button, and then select the format you want to create an upstream for:

13201320

Create Upstream button

Create a Maven Upstream

593593

Create Maven Upstream Form

Form Field

Description

Name

A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream.

Priority

The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date.

Upstream URL

The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository.

Proxy Only

(Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository.

Cache and Proxy

Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests.

Cache Only

Requests made for packages that aren't yet in this repository will self-redirect until available. This mode ensures that packages served are guaranteed to be signed with the associated repository signing key

Verify SSL Certificates

If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks.

GPG key

The source of a package signing key. When a signing key is provided, the Cloudsmith setup script will ensure this signing key is deployed to allow packages available on this upstream to be installed

Authentication (optional)

Optional credentials that can be provided if the upstream is not publicly accessible

Headers (optional)

Optional Key-Value headers that can be passed to upstreams with each request.

📘

Note

Package caching is only supported for for Maven packages that have a .pom file present on the upstream source.

Create a Debian Upstream

652652

Create Debian Upstream Form

Form Field

Description

Name

A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream.

Priority

The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date.

Upstream URL

The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository.

Proxy Only

(Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository.

Cache and Proxy

Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests.

Distribution Version

The distribution version that packages from the upstream will be associated with.

Upstream Component

The component to fetch from the upstream.

Upstream Distribution

(optional) The distribution to fetch from the upstream. Useful for repositories that have custom naming schemes. If left blank, the Distribution Version will be used.

Source Packages

If selected, source packages will be available from the upstream.

Verify SSL Certificates

If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks.

Authentication (optional)

Optional credentials that can be provided if the upstream is not publicly accessible

Headers (optional)

Optional Key-Value headers that can be passed to upstreams with each request.

Create an RPM Upstream

590590

Create RedHat Upstream Form

Form Field

Description

Name

A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream.

Weighting

The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date.

Upstream URL

The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository.

Proxy Only

(Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository.

Cache and Proxy

Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests.

Distribution

The distribution version to index from the upstream, such as el/8r or fedora/32.

Source Packages

If selected, source packages will be available from the upstream.

Verify SSL Certificates

If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks.

GPG Key

The source of a package signing key. When a signing key is provided, the Cloudsmith setup script will ensure this signing key is deployed to allow packages available on this upstream to be installed.

Authentication (optional)

Optional credentials that can be provided if the upstream is not publicly accessible

Headers (optional)

Optional Key-Value headers that can be passed to upstreams with each request.

Create a NuGet Upstream

655655

Create NuGet Upstream Form

Form Field

Description

Name

A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream.

Priority

The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date.

Upstream URL

The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository.

Proxy Only

(Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository.

Cache and Proxy

Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests.

Verify SSL Certificates

If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks.

Authentication (optional)

Optional credentials that can be provided if the upstream is not publicly accessible

Headers (optional)

Optional Key-Value headers that can be passed to upstreams with each request.

Create a Dart Upstream

651651

Create Dart Upstream Form

Form Field

Description

Name

A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream.

Priority

The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date.

Upstream URL

The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository.

Proxy Only

Proxy requests through to upstream sources in order to match assets that are not present in this repository.

Verify SSL Certificates

If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks.

Authentication (optional)

Optional credentials that can be provided if the upstream is not publicly accessible

Headers (optional)

Optional Key-Value headers that can be passed to upstreams with each request.

Create a Ruby Upstream

📘

Ruby Upstreams are currently a beta feature. If you'd like to be included in early access to this feature please contact us.

656656

Create Ruby Upstream Form

Form Field

Description

Name

A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream.

Priority

The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date.

Upstream URL

The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository.

Proxy Only

Proxy requests through to upstream sources in order to match assets that are not present in this repository.

Cache and Proxy

Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests.

Verify SSL Certificates

If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks.

Authentication (optional)

Optional credentials that can be provided if the upstream is not publicly accessible

Headers (optional)

Optional Key-Value headers that can be passed to upstreams with each request.

Edit an Upstream Proxy

Click the blue "Edit Upstream" button to edit an upstream source:

962962

Disable an Upstream Proxy

Click the orange "Disable Upstream" button to disable an upstream source:

956956

Delete an Upstream Proxy

Click the red "Disable Upstream" button to disable an upstream source:

958958

Did this page help you?