Upstream Proxying
Upstream proxying and caching allows you to upload and use the packages you own, while Cloudsmith fetches and caches other packages (such as dependencies).
This enables you to use Cloudsmith as a first-class cache and a central source of truth for packages, to protect you from outages of external services (which is especially important when running behind your firewall).
Upstream Concepts
Cloudsmith upstream support centers around several key concepts:
- Proxying: The act of transparently allowing access to a package that exists on an upstream repository. Package managers see the remote package as one which belongs to the Cloudsmith repository.
- Caching: An extension to the proxying functionality, where requested packages from an upstream are fetched and permanently stored in your Cloudsmith repository. This helps to ensure package dependencies are always available and helps to protect from upstream outages or security breaches.
- Indexing: In order to be aware of the packages available from an upstream, Cloudsmith builds an index. This process occurs when an upstream is first added to your Cloudsmith repository and is scheduled for a resync on a regular basis.
Indexing
Index availability is a critical factor for upstream handling on Cloudsmith, helping to ensure deterministic performance for upstream requests and a deeper insight into the availability of packages.
The indexing process can differ, depending on the package format and upstream itself. Where possible, Cloudsmith will determine the availability of all packages on an upstream repository ahead-of-time, which generally means that an upstream repository is unavailable when first added, until this indexing process has occurred.
For package formats that do not maintain a centralized mechanism for retrieval of all packages, Cloudsmith employs a just-in-time indexing mechanism. In this approach, awareness of packages is made the first time a package is successfully cached from an upstream repository. Going forwards, Cloudsmith maintains a list of all versions available on the source upstream for the package and ensures this is kept in sync.
When neither indexing mechanism is available for an upstream, Cloudsmith falls back to a real-time unindexed approach. When requests are made for upstream packages, Cloudsmith determines availability across each upstream in your repository, in real-time. This is the least performant approach.
We strive to ensure that at least just-in-time indexing is available for each package format and upstream source, although this is not always possible.
Priority
When defining upstreams for a repository, a priority can be specified. The priority of an upstream is used to determine the order in which upstream requests are resolved. Cloudsmith evaluates upstreams by the order of 1..n.
A good approach when determining what priority to apply to upstreams is to ensure that the lowest value is specified for the upstream which is most likely to contain upstream packages you request. This helps to improve performance in the event that an upstream source does not support any of our indexing mechanisms.
Supported Formats
| Format | Fixed Proxy | Configurable Proxy | Caching | Indexing | Indexing Type |
|---|---|---|---|---|---|
| Cargo |  |  | | | N/A |
| CRAN | N/A | | | | Ahead-of-Time |
| Dart | N/A | | | | Just-in-Time |
| Debian | N/A | | | | Ahead-of-Time |
| Docker | N/A | | | | Just-in-Time |
| Gradle | N/A | | | | Just-in-Time |
| Helm | N/A | | | | Ahead-of-Time |
| Maven | N/A | | | | Just-in-Time |
| npm | N/A | | | | Just-in-Time |
| NuGet | N/A | | | | Ahead-of-Time |
| Python | N/A | | | | Ahead-of-Time |
| RedHat | N/A | | | | Ahead-of-Time |
| Ruby | N/A | | | | Ahead-of-Time |
| sbt | N/A | | | | Just-in-Time |
| Swift | N/A | | | | Just-in-Time |
You can can also create and manage upstreams via the Cloudsmith API
Create an Upstream Proxy
Click the green "Create Upstream" button, and then select the format you want to create an upstream for:
Create Upstream button
Create a Maven Upstream
Create Maven Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | (Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Cache Only | Requests made for packages that aren't yet in this repository will self-redirect until available. This mode ensures that packages served are guaranteed to be signed with the associated repository signing key |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| GPG key | The source of a package signing key. When a signing key is provided, the Cloudsmith setup script will ensure this signing key is deployed to allow packages available on this upstream to be installed |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Note
Package caching is only supported for for Maven packages that have a
.pomfile present on the upstream source.
Create a Debian Upstream
Create Debian Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | (Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Distribution Version | The distribution version that packages from the upstream will be associated with. |
| Upstream Component | The component to fetch from the upstream. |
| Upstream Distribution | (optional) The distribution to fetch from the upstream. Useful for repositories that have custom naming schemes. If left blank, the Distribution Version will be used. |
| Source Packages | If selected, source packages will be available from the upstream. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a Docker Upstream
Create Docker Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible. Note: Docker Hub requires that requests to their service via an upstream proxy be authenticated. As such, when you configure an upstream to Docker Hub, you will be required to provide credentials for authentication. |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a RPM Upstream
Create RedHat Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Weighting | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | (Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Distribution | The distribution version to index from the upstream, such as el/8r or fedora/32. |
| Source Packages | If selected, source packages will be available from the upstream. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| GPG Key | The source of a package signing key. When a signing key is provided, the Cloudsmith setup script will ensure this signing key is deployed to allow packages available on this upstream to be installed. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a NPM Upstream
Create NPM Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a NuGet Upstream
Create NuGet Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | (Default) Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a Python Upstream
Create Python Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a Dart Upstream
Create Dart Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a Ruby Upstream
Create Ruby Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a Swift Upstream
Create Swift Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Create a CRAN Upstream
Create CRAN Upstream Form
| Form Field | Description |
|---|---|
| Name | A descriptive name for this upstream source. A shortened version of this name will be used for tagging cached packages retrieved from this upstream. |
| Priority | The weighting of the Upstream source. Upstream sources are selected for resolving requests by sequential order (1..n), followed by creation date. |
| Upstream URL | The URL for this upstream source. This must be a fully qualified URL including any path elements required to reach the root of the repository. |
| Proxy Only | Proxy requests through to upstream sources in order to match assets that are not present in this repository. |
| Cache and Proxy | Proxy the initial request for an asset through to the upstream source and then store (cache) resolved assets in this repository for future requests. |
| Verify SSL Certificates | If enabled, SSL certificates are verified when requests are made to this upstream. We recommended leaving this enabled for all public sources to help mitigate Man-In-The-Middle (MITM) attacks. |
| Authentication (optional) | Optional credentials that can be provided if the upstream is not publicly accessible |
| Headers (optional) | Optional Key-Value headers that can be passed to upstreams with each request. |
Edit an Upstream Proxy
Click the blue "Edit Upstream" button to edit an upstream source:
Disable an Upstream Proxy
Click the orange "Disable Upstream" button to disable an upstream source:
Delete an Upstream Proxy
Click the red "Disable Upstream" button to disable an upstream source:
Updated 5 months ago
