Simple exporter tool that helps migrate repository files, data assets and issue tickets from projects on Pagure to GitLab
-
Ensure that you have
python3
andpython3-pip
installed.$ sudo dnf install python3 python3-pip --setopt=install_weak_deps=False
-
Create and activate a Python virtual environment in that directory.
$ python3 -m venv venv
(venv) $ source venv/bin/activate
-
Install
pagure-exporter
usingpip
in the activated virtual environment.(venv) $ pip install pagure-exporter
Sample output
Requirement already satisfied: pagure-exporter in ./venv/lib/python3.12/site-packages (0.1.0) Requirement already satisfied: GitPython<4.0.0,>=3.1.37 in ./venv/lib/python3.12/site-packages (from pagure-exporter) (3.1.40) Requirement already satisfied: click<9.0.0,>=8.1.3 in ./venv/lib/python3.12/site-packages (from pagure-exporter) (8.1.7) Requirement already satisfied: requests<3.0.0,>=2.31.0 in ./venv/lib/python3.12/site-packages (from pagure-exporter) (2.31.0) Requirement already satisfied: tqdm<5.0.0,>=4.64.1 in ./venv/lib/python3.12/site-packages (from pagure-exporter) (4.66.1) Requirement already satisfied: gitdb<5,>=4.0.1 in ./venv/lib/python3.12/site-packages (from GitPython<4.0.0,>=3.1.37->pagure-exporter) (4.0.10) Requirement already satisfied: charset-normalizer<4,>=2 in ./venv/lib64/python3.12/site-packages (from requests<3.0.0,>=2.31.0->pagure-exporter) (3.3.0) Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.12/site-packages (from requests<3.0.0,>=2.31.0->pagure-exporter) (3.4) Requirement already satisfied: urllib3<3,>=1.21.1 in ./venv/lib/python3.12/site-packages (from requests<3.0.0,>=2.31.0->pagure-exporter) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.12/site-packages (from requests<3.0.0,>=2.31.0->pagure-exporter) (2023.7.22) Requirement already satisfied: smmap<6,>=3.0.1 in ./venv/lib/python3.12/site-packages (from gitdb<5,>=4.0.1->GitPython<4.0.0,>=3.1.37->pagure-exporter) (5.0.1)
-
Ensure that you have
git
,python3
andpoetry
installed.$ sudo dnf install git python3 poetry --setopt=install_weak_deps=False
-
Clone the repository to the local storage and make it the present working directory.
$ git clone https://github.com/gridhead/pagure-exporter.git
$ cd pagure-exporter
Sample output
Cloning into 'pagure-exporter'... remote: Enumerating objects: 118, done. remote: Counting objects: 100% (118/118), done. remote: Compressing objects: 100% (78/78), done. remote: Total 118 (delta 48), reused 94 (delta 31), pack-reused 0 Receiving objects: 100% (118/118), 56.38 KiB | 4.34 MiB/s, done. Resolving deltas: 100% (48/48), done.
-
Create and activate a Python virtual environment in that directory.
$ python3 -m venv venv
(venv) $ source venv/bin/activate
-
Check the project configuration's validity and then install the project dependencies.
(venv) $ poetry check
(venv) $ poetry install
Sample output
All set!
Installing dependencies from lock file Package operations: 19 installs, 0 updates, 0 removals • Installing smmap (5.0.0) • Installing certifi (2022.12.7) • Installing charset-normalizer (3.0.1) • Installing click (8.1.3) • Installing gitdb (4.0.10) • Installing idna (3.4) • Installing mccabe (0.6.1) • Installing mypy-extensions (0.4.3) • Installing pathspec (0.11.0) • Installing platformdirs (2.6.2) • Installing pycodestyle (2.8.0) • Installing pyflakes (2.4.0) • Installing urllib3 (1.26.14) • Installing black (22.12.0) • Installing flake8 (4.0.1) • Installing gitpython (3.1.30) • Installing isort (5.12.0) • Installing requests (2.28.2) • Installing tqdm (4.64.1) Installing the current project: pagure-exporter (0.1.0)
-
Using an internet browser of your choice, open up Pagure and login to your account.
-
Click on your profile display picture and then, head over to the Account Settings page.
-
Under the API Keys section, click on the Create new API key button in the top right corner.
-
As this is the source namespace, check all the ACLs that are required to read the asset information associated with a repository, set a safe expiration date for the API token and write an appropriate description for its usage before clicking on the Create button.
-
Make note of the API token generated and ensure that they are not shared with others or used for a different purpose.
-
Head over to a repository that the currently logged-in username has at least a READ access to.
-
Make note of the source namespace in the format of
HOLDER/REPONAME
where theHOLDER
can be a group or a sole user. For example, in case of a repository located athttps://pagure.io/fedora-infra/ansible
- the source namespace would befedora-infra/ansible
. -
In another internet browser tab or window, open up GitLab and login to your account.
-
Click on the New project/repository option from the sidebar of the profile page and then, click on the Create blank project option.
-
Create a new empty repository which will act as the destination for the asset transfer. It is recommended to have the same name as the source namespace to avoid confusion, but it is not strictly required.
-
Head over to the created repository and make note of the Project ID. For example in this case, it is
42823949
for the destination repository namedgridhead/pagure-exporter-test
. -
Expand the sidebar to head over to the Repository section from the Settings section.
-
In the Protected Branches section, turn off all existing branch protection rules and allow force pushing to all branches temporarily to allow for the repository assets to be moved in here.
-
Expand the sidebar to head over to the Access Tokens section from the Settings section.
-
In the Project Access Token page, click on the Add new token button to begin creating a new access token.
-
As this is the destination namespace, check all the scopes that are required to write the asset information associated with a repository, pick an appropriate role, set a safe expiration date and write an appropriate description for its usage before clicking on the Create project access token button.
-
Make note of the API token generated and ensure that they are not shared with others or used for a different purpose.
-
Ensure that you have the following information handy before proceeding to the next steps.
- Username of an account that has at least the READ permissions in the source namespace on Pagure (Say
srceuser
) - Access token belonging to the aforementioned account with appropriate ACLs checked required for at least the READ permissions in the source namespace (Say
srcecode
) - Name of the source namespace in the format
HOLDER/REPONAME
where theHOLDER
can be a group or a sole user (Saysrcerepo
) - Username of an account that has at least the WRITE permissions in the destination namespace on GitLab (Say
destuser
) - Access token belonging to that aforementioned account appropriate roles and scopes required for at least the WRITE permissions in the destination namespace (Say
destcode
) - Name of the destination namespace in the format of uniquely identifiable
PROJECTID
string (Saydestrepo
)
- Username of an account that has at least the READ permissions in the source namespace on Pagure (Say
-
Check the current version of the installed project as well as the usage information.
(venv) $ pagure-exporter --version
(venv) $ pagure-exporter --help
Sample output
Pagure Exporter by Akashdeep Dhar <t0xic0der@fedoraproject.org>, version 0.1.0
Usage: pagure-exporter [OPTIONS] COMMAND [ARGS]... Options: -s, --srce TEXT Source namespace for importing assets from [required] -d, --dest TEXT Destination namespace for exporting assets to [required] -p, --pkey TEXT Pagure API key for accessing the source namespace [required] -g, --gkey TEXT GitLab API key for accessing the destination namespace [required] -f, --fusr TEXT Username of the account that owns the Pagure API key [required] -t, --tusr TEXT Username of the account that owns the GitLab API key [required] --version Show the version and exit. --help Show this message and exit. Commands: repo Initialize transfer of repository assets tkts Initiate transfer of issue tickets
-
Check the usage information of the available subcommands.
(venv) $ pagure-exporter -s a -d a -p a -g a -f a -t a repo --help
(venv) $ pagure-exporter -s a -d a -p a -g a -f a -t a tkts --help
Sample output
Usage: pagure-exporter repo [OPTIONS] Initialize transfer of repository assets Options: -b, --brcs TEXT List of branches to extract --help Show this message and exit.
Usage: pagure-exporter tkts [OPTIONS] Initiate transfer of issue tickets Options: -s, --status [OPEN|SHUT|FULL] Extract issue tickets of the mentioned status [default: OPEN] -r, --ranges TEXT... Extract issue tickets in the mentioned ranges -p, --select TEXT Extract issue tickets of the selected numbers -c, --comments Transfer all the associated comments -l, --labels Migrate all the associated labels -a, --commit Assert issue ticket states as they were -t, --secret Confirm issue ticket privacy as they were -o, --series Ensure issue tickets sequence as they were --help Show this message and exit.
-
Ensure that the location where the project repository was cloned is the present working directory and that the previously populated virtual environment is enabled.
$ cd pagure-exporter
$ source venv/bin/activate
-
Using an internet browser of your choice, visit the source namespace repository page on Pagure to pick the branches that you wish to transfer.
-
Execute the following command to begin migrating the repository assets from the source namespace on Pagure to the destination namespace on GitLab.
-
If only a set of branches are to be migrated
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ repo \ --brcs brca,brcb,brcc,brcd
For a set of branches available in the source namespace named
brca
,brcb
,brcc
andbrcd
to be migrated to the destination namespace. -
If all the available branches are to be migrated
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ repo
This is the default behaviour of the subcommand so if no branch names are provided, all the branches from the source namespace are migrated
-
-
If there are any contents available in the branches of the destination namespace, they will be overwritten by the contents from the branches migrated from the source namespace.
-
Ensure that the location where the project repository was cloned is the present working directory and that the previously populated virtual environment is enabled.
$ cd pagure-exporter
$ source venv/bin/activate
-
Execute the following command to begin extracting the issue tickets from the source namespace on Pagure to the destination namespace.
-
If the issue tickets of a certain status need to be transferred.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --status open
-
If the comments associated with the issue tickets need to be transferred.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --comments
-
If the labels associated with the issue tickets need to be transferred.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --labels
-
If the states associated with the issue tickets need to be transferred.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --commit
-
If the privacy associated with the issue tickets need to be transferred.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --secret
-
If the issue tickets need to be transferred in the sequence they were in.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --series
-
If the issue tickets from a range of issue identities need to be transferred.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --ranges STRT STOP
Issue tickets with identities
STRT
,STRT+1
...STOP-1
,STOP
would be considered here. -
If the issue tickets that need to be considered need to be cherry-picked.
(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --select NUM1,NUM2,NUM3 ...
Issue tickets with identities
NUM1
,NUM2
,NUM3
... would be considered here.
While these options can be mixed and matched to be used together, the options
--ranges
and--select
cannot be used at the same time as they perform identical functions.For example,
-
The following command will migrate all issue tickets, the identities of which fall between the range of
STRT
andSTOP
both included, with statusOPEN
along with the associated comments, labels, privacy and identifiers.(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --status open \ --comments \ --labels \ --secret \ --series \ --ranges STRT STOP
-
The following command will migrate all issue tickets with the identities
NUM1
,NUM2
,NUM3
... with statusSHUT
along with the associated labels, states, privacy and identifiers.(venv) $ pagure-exporter \ --fusr srceuser --pkey srcecode --srce srcerepo \ --tusr destuser --gkey destcode --dest destrepo \ tkts \ --status shut \ --labels \ --commit \ --secret \ --series \ --select NUM1,NUM2,NUM3 ...
-