Mirroring channels and packages#

Data Science & AI Workbench enables you to create local copies of a repositories so users can access packages from a centralized, on-premise location. This process is called mirroring. You can mirror the full content of a repository, or include only specific packages or types of packages from the repository in your mirror. You can also create mirrors in an air-gapped network improve performance and security.

You can mirror an online repository, or you can use a tarball containing package data to populate a channel in Workbench.

Prerequisites:

Note

It can take several hours to mirror an entire repository, depending on its size.

Creating a conda mirror#

The basic steps for creating a conda mirror are:

  1. Prepare your mirror configuration file.

  2. Log in to the Workbench CLI.

  3. If necessary, create a channel in the internal Workbench repository.

  4. Initiate the mirror by running the following command:

    anaconda-mirror-ae5 --file /path/to/<mirror.yaml>

    Note

    Append --dry-run to the command to see what actions would be taken by the mirror, without performing actual modifications.

Preparing your mirror configuration file#

Create a <mirror>.yaml file that details the configurations for the mirror.

Tip

You can name this file whatever you’d like. Anaconda recommends naming it the same as the channel you are mirroring to.

Basic configurations:

Define source channel locations, package platforms, and destination/storage location details. Manage package formats, clean up outdated packages, and test configurations without applying changes by including these configurations.

Parameter

Description

channels

List of URLs for channels you want to mirror from.

If a short channel name is supplied, Workbench uses its system-level .condarc file’s channel_alias: value to complete the channel URL.

platforms

List of platforms you want to mirror packages for. For example, win-64 or linux-64.

If no value is supplied, the mirror will include packages for all platforms available on the source channel.

dest_channel

The short name for the internal Workbench repository channel you are mirroring to. The rest of the channel URL is automatically completed by anaconda-enterprise-cli for you.

dest_site

The web address or path where you want to store your mirrored packages.

The specific formatting and necessity of this value depends on the type of destination repository. For more information, see repository-specific configurations.

format_policy

Determines how the mirror manages .conda and .tar.bz2 files:

  • prefer-conda or prefer-tarbz2 - Mirror one package format over the other if both are available for a package. If the preferred type is unavailable, the other file type is still downloaded.

  • only-conda or only-tarbz2 - Mirror packages that are available in the preferred file format only.

  • transmute-conda or transmute-tarbz2 - Convert packages to the preferred format as necessary. Requires the conda-package-handling package to function.

  • keep-both - Mirror both file types for all available packages.

Defaults to prefer-conda for repositories that support .conda formatting and only-tarbz2 for those that do not.

If your repository does not support .conda formatting, Anaconda recommends installing the conda-package-handling package and using the transmute-tarbz2 option.

clean

true / false - If true, removes packages from the destination channel that are not on the source channel when updating.

Default: false (to ensure packages are not inadvertently removed)

dry_run

true / false - If true, outputs what actions would be taken by the mirror, without performing actual modifications.

Default: false
Filtering configurations:

Fine-tune which packages are included in the mirror. Specify versions of Python or R packages that your packages should be compatible with, include only specific packages, or exclude packages by name and license family type.

Parameter

Description

python_versions

A comma-separated list of Python versions. Restricts all Python packages and packages that depend on Python to these versions.

r_versions

A comma-separated list of R versions. Restricts all R packages and packages that depend on R to these versions.

pkg_list

List of package names or valid MatchSpec strings. If supplied, only the specified packages will be mirrored, not their dependencies.

Cannot be paired with license_exclude, exclude, or include.

license_exclude

List of license families to exclude from the mirror. To see a list of valid license families, use the anaconda-mirror-ae5 --help command.

Cannot be paired with pkg_list.

exclude

List of package names or valid MatchSpec strings to exclude.

Cannot be paired with pkg_list.

include

List of package names or valid MatchSpec strings to override the mirror’s other filters and include these packages even if they would otherwise be filtered out.

Cannot be paired with pkg_list.

Note

For more information about MatchSpec, see package match specifications.

Advanced configurations:

Configure repository authentication, enforce platform restrictions, and manage SSL verification for secure connections.

Parameter

Description

username / password

Supplies credentials for repository authentication. For more information, see repository-specific configurations.

strict_platforms

true / false - If true, excludes noarch from the mirror.

Default: false (all platforms use noarch)

max_attempts

Number of retry attempts for failed connections.

Default: 5

max_failures

Number of failed transactions before stopping.

Default: 100

verify_ssl

true / false - Enables or disables SSL verification.

Default: true

Note

If Workbench is installed in a proxied environment, see Configuring conda in Workbench for information on setting the NO_PROXY variable.

Repository-specific configurations#

JFrog Artifactory#

For Artifactory destinations, the dest_site can be a repository hostname, or a full URL.

If you supply the hostname only, anaconda-mirror interprets the channel path as:

https://<dest_site>/artifactory/<dest_channel>

If you supply a URL, anaconda-mirror appends /artifactory/dest_channel to it to complete the channel path. For example, if you set the dest_site to https://example.site.com, the full path is interpreted as this:

https://example.site.com/artifactory/<dest_channel>

If your mirror is not at this location, further pathing can be included in the URL to reach the correct location. For example, if you set dest_site as https://example.site.com/artifactory/repo/subpath/, it will interpret the path literally and append the dest_channel value to the end:

https://example.site.com/artifactory/repo/subpath/<dest_channel>
To authenticate to a JFrog Artifactory repository:

  • Configure the username and password values in your .yaml file to contain your credentials. If both values are supplied, they are delivered using basic HTTP authentication. You can substitute an access token for your password if necessary.

  • Configure just the password value in your .yaml file. This is delivered as a bearer token using the Authorization: Bearer header. This must be an access token.

  • Configure your .netrc file to store your username and password for the repository. These values are delivered using basic HTTP authentication.

S3 bucket#

For Simple Storage Service (S3) buckets, the channel path is a concatenation of the dest_site and dest_channel values.

For example, if you were mirroring to an S3 bucket, your dest_site would be set to <bucket_name>/full/path/to/ and the full channel path is interpreted as:

<bucket_name>/full/path/to/<dest_channel>

Authentication to an S3 source is currently controlled entirely by the environment. For example, you can use the aws CLI tool to configure the target region and authenticate. You may wish to use the AWS_PROFILE environment variable to select among multiple configurations.

Local#

Much like the S3 bucket, the local repository channel path consists of a concatenation of the dest_site and dest_channel values.

No authentication is necessary for local repositories.

anaconda-enterprise-cli#

The dest_site value defaults to the <SITE_NAME> value established when you configure the workbench CLI. If you have only configured the CLI to be able to access one site (i.e. your Workbench instance), there is no need to specify this value.

Authentication is handled when you log in to the CLI.

Example conda and R mirrors#

Here are some example mirror .yaml files you can use to mirror some common repositories:

Anaconda’s main channel (full)
dest_channel: main
channels:
  - https://repo.anaconda.com/pkgs/main
platforms:
  - linux-64
Anaconda’s R channel (full)
dest_channel: r
channels:
  - https://repo.anaconda.com/pkgs/r
platforms:
  - linux-64
Air-gapped network mirror
dest_channel: anaconda
channels:
  - /file/path/to/unpacked/repository/packages
platforms:
  - linux-64

Mirroring a PyPI repository#

The full PyPI mirror size is currently close to 10TB, so ensure that your file storage location has sufficient disk space before proceeding.

Because anaconda-mirror does not handle .pip package formatting, mirrors for PyPI repositories containing such packages are managed by the anaconda-enterprise-cli tool.

The steps are identical to creating a conda mirror:

  1. Prepare your mirror configuration file.

  2. Log in to the Workbench CLI.

  3. If necessary, create a channel in the internal Workbench repository.

  4. Initiate the mirror by running the following command:

    anaconda-enterprise-cli mirror pypi --config pypi-mirror.yaml

This command loads the packages on https://pypi.org into the user’s account.

Mirrored packages can be viewed at https://<FQDN>/repository/pypi/pypi/simple/, replacing <FQDN> with the fully qualified domain name of your installation of Workbench. (The second pypi in the url should match the user configuration value described below.)

PyPI configurations:

PyPI mirror .yaml configuration values consist of the following:

Parameter

Description

user

The local user under which the PyPI packages are imported.

Default: pypi

pkg_list

List of package names to mirror. If supplied, only the specified packages will be mirrored, not their dependencies.

Cannot be paired with blocklist or allowlist.

allowlist

List of package names to mirror. If supplied, only the specified packages will be mirrored, not their dependencies.

Cannot be paired with pkg_list.

blocklist

List of package names to skip. Packages listed here are not mirrored.

Cannot be paired with pkg_list.

latest_only

If supplied, only the latest package versions are mirrored.

Default: false

remote_url

The URL of the PyPI mirror.

/pypi is appended to build the XML RPC API URL, /simple for the simple index and /pypi/{package}/{version}/json for the JSON API.

Default: https://pypi.python.org/

xml_rpc_api_url

A custom value for XML RPC URL. If this value is present, it takes precedence over the URL built using remote_url. Default: null.

simple_index_url

A custom value for the simple index URL. If this value is present, it takes precedence over the URL built using remote_url. Default: null.

use_xml_rpc

Whether to use the XML RPC API as specified by PEP381. If this is set to true, the XML RPC API is used to determine which packages to check. Otherwise, the script falls back to the simple index. If the XML RPC fails, the simple index is used.

Default: true

use_serial

If set to true, uses the serial number provided by the XML RPC API. Only packages updated since the last serial saved are checked. If this is set to false, all PyPI packages are checked for updates.

Default: true

create_org

Creates the mirror user as an organization instead of a regular user account. All superusers are added to the Owners group of the organization.

Default: false

Note

All mirrored PyPI-like channels are publicly available to pull packages from both inside and outside Workbench (no authentication is required).

Example PyPI mirror (partial)
allowlist:
  - requests
  - six
  - numpy
  - simplejson
latest_only: true
remote_url: https://pypi.org/
use_xml_rpc: true

Configuring pip#

To configure pip to use this new mirror, create pip.conf as follows:

# Replace <WORKBENCH_URL> with the actual URL to your Workbench instance
[global]
index-url=<WORKBENCH_URL>/repository/pypi/pypi/simple/

To configure Workbench sessions and deployments to automatically use the pip.conf, run the following command.

anaconda-enterprise-cli spark-config --config /etc/pip.conf pip.conf

For more specific information on configuring pip, see the official pip documentation.

On this page