Bash script to mirror some PyPi packages
The script requires 2 input files placed in the same directory as where the script is.
-
The first input file is called
pkgs_to_add.txt
and contains the packages to be added to the local mirror. Each package is one line. -
The second file is called
mirror.conf
and is a bandersnatch configuration file. As the packages and its dependencies will be appended to this configuration file at runtime to mirror PyPi, the file must end withpackages =
as part of[allowlist]
. Please seesample_mirror.conf
as an example.
-
The script creates a throwaway conda environment and install the packages in
pkgs_to_add.txt
individually to figure out the full set of dependencies required. -
The list of required packages and its dependencies is added to a file called
pkgs_in_mirror.txt
which serves as a master list of packages in the mirror. This file is sorted and have unique entries. Please see Note 1 for more details. -
The script activates a conda environment with bandersnatch installed.
-
pkgs_in_mirror.txt
is appended tomirror.conf
to form a complete bandersnatch configuration file. -
The script uses bandersnatch to mirror all editions of the whitelised packages from PyPi via the provided configuration file.
-
Finally, the script locates the newly added files and copies them to a folder called
to_transfer
to minimise transfer requirements.
- The list of required packages and its dependencies can be added to mirror.conf directly. However there may be duplicate packages as the maintainer of the local mirror add packages over time. Hence an intermediate file
pkgs_in_mirror.txt
was used.