MirrorUpdater.jl - Provides functionality for automatically mirroring Julia package repositories

MirrorUpdater.jl is a Julia application that provides functionality for automatically mirroring Julia package repositories.

MirrorUpdater.jl (and its sibling project, Snapshots.jl) are used to maintain the Julia package mirrors and snapshots hosted at:

You can host your own mirrors for free by following these instructions:

If you do not already have a personal GitHub account, create one. For the remainder of this README, MY_PERSONAL_GITHUB_USERNAME refers to the username of your personal GitHub account.

For example, for me, MY_PERSONAL_GITHUB_USERNAME is equal to DilumAluthge.

Log in to GitHub as MY_PERSONAL_GITHUB_USERNAME.

While logged in as MY_PERSONAL_GITHUB_USERNAME, enable two-factor authentication on the MY_PERSONAL_GITHUB_USERNAME account.

Make sure to store your two-factor recovery codes in a secure location!

While logged in as MY_PERSONAL_GITHUB_USERNAME, create a free GitHub organization that you will use only for hosting the mirrored repositories. For the remainder of this README, MY_GITHUB_ORG refers to the name of this organization. MY_PERSONAL_GITHUB_USERNAME should be an owner of the MY_GITHUB_ORG organization.

For example, for me, MY_GITHUB_ORG is equal to UnofficialJuliaMirror.

While logged in as MY_PERSONAL_GITHUB_USERNAME, go to the MY_GITHUB_ORG organization security settings page (https://github.com/organizations/MY_GITHUB_ORG/settings/security).

Next, make sure that the checkbox next to "Require two-factor authentication for everyone..." is CHECKED.

Finally, click the "Save" button.

Log out of the MY_PERSONAL_GITHUB_USERNAME account.

Create a new GitHub "bot" account that you will use ONLY for maintaining the mirror. For the remainder of this README, MY_GITHUB_BOT_USERNAME refers to the username of this account.

For example, for me, MY_GITHUB_BOT_USERNAME is equal to UnofficialJuliaMirrorBot.

Log in to GitHub as MY_GITHUB_BOT_USERNAME.

While logged in as MY_GITHUB_BOT_USERNAME, enable two-factor authentication on the MY_GITHUB_BOT_USERNAME account.

Make sure to store your two-factor recovery codes in a secure location!

While logged in as MY_GITHUB_BOT_USERNAME, create a personal access token for the MY_GITHUB_BOT_USERNAME account and store it in a secure place (such as a password manager). For the remainder of this README, my-github-bot-personal-access-token refers to this personal access token.

The personal access token should be treated as securely as a password. Do not share it with anyone. Do not save it in any unsecure location. Do not save it in a file. Do not commit it in a Git repository.

Log out of the MY_GITHUB_BOT_USERNAME account.

Log in to GitHub as MY_PERSONAL_GITHUB_USERNAME.

While logged in as MY_PERSONAL_GITHUB_USERNAME, go to the MY_GITHUB_ORG organization members page (https://github.com/orgs/UnofficialJuliaMirror/people).

Then, add MY_GITHUB_BOT_USERNAME as a member of the MY_GITHUB_ORG organization.

This will allow MY_GITHUB_BOT_USERNAME to create new repositories within the MY_GITHUB_ORG organization.

While logged in as MY_PERSONAL_GITHUB_USERNAME, fork the MirrorUpdater.jl repository to the MY_GITHUB_ORG organization.

Go to your fork of MirrorUpdater.jl: https://github.com/MY_GITHUB_ORG/MirrorUpdater.jl

In your fork, update lines 1 and 2 of config/github.jl to look like:

const GITHUB_ORGANIZATION = "MY_GITHUB_ORG"
const GITHUB_BOT_USERNAME = "MY_GITHUB_BOT_USERNAME"

Leave the rest of config/github.jl unchanged. Please do not stored your personal access token in the file.

In your fork, update line 1 of config/enabled-providers.jl to look like:

const GITHUB_ENABLED = true

If there are other registries of Julia packages that you would like to mirror, add them to the config/registries.jl file in your fork:

Update the other configuration files in the config/ folder of your fork as you see fit.

Congratulations, you have finished this section.

If you do not already have a personal GitLab account, create one. For the remainder of this README, MY_PERSONAL_GITLAB_USERNAME refers to the username of your personal GitLab account.

For example, for me, MY_PERSONAL_GITLAB_USERNAME is equal to DilumAluthge.

Log in to GitLab as MY_PERSONAL_GITLAB_USERNAME.

While logged in as MY_PERSONAL_GITLAB_USERNAME, enable two-factor authentication on the MY_PERSONAL_GITLAB_USERNAME account.

Make sure to store your two-factor recovery codes in a secure location!

While logged in as MY_PERSONAL_GITLAB_USERNAME, create a free GitLab group that you will use only for hosting the mirrored repositories. For the remainder of this README, MY_GITLAB_GROUP refers to the name of this group. MY_PERSONAL_GITLAB_USERNAME should be an owner of the MY_GITLAB_GROUP group.

For example, for me, MY_GITLAB_GROUP is equal to UnofficialJuliaMirror.

While logged in as MY_PERSONAL_GITLAB_USERNAME, go to the MY_GITLAB_GROUP group general settings page (https://gitlab.com/groups/MY_GITLAB_GROUP/-/edit).

Next, scroll down to the "Permissions, LFS, 2FA" section. Click the "Expand" button next to "Permissions, LFS, 2FA" to expand the section.

Then, make sure that the checkbox next to "Require all users in this group to setup Two-factor authentication" is CHECKED.

Finally, click the "Save changes" button.

Log out of the MY_PERSONAL_GITLAB_USERNAME account.

Create a new GitLab "bot" account that you will use ONLY for maintaining the mirror. For the remainder of this README, MY_GITLAB_BOT_USERNAME refers to the username of this account.

For example, for me, MY_GITLAB_BOT_USERNAME is equal to UnofficialJuliaMirrorBot.

Log in to GitLab as MY_GITLAB_BOT_USERNAME.

While logged in as MY_GITLAB_BOT_USERNAME, enable two-factor authentication on the MY_GITLAB_BOT_USERNAME account.

Make sure to store your two-factor recovery codes in a secure location!

While logged in as MY_GITLAB_BOT_USERNAME, create a personal access token for the MY_GITLAB_BOT_USERNAME account and store it in a secure place (such as a password manager). For the remainder of this README, my-gitlab-bot-personal-access-token refers to this personal access token.

The personal access token should be treated as securely as a password. Do not share it with anyone. Do not save it in any unsecure location. Do not save it in a file. Do not commit it in a Git repository.

Log out of the MY_GITLAB_BOT_USERNAME account.

Log in to GitLab as MY_PERSONAL_GITLAB_USERNAME.

While logged in as MY_PERSONAL_GITLAB_USERNAME, go to the MY_GITLAB_GROUP group members page (https://gitlab.com/groups/MY_GITLAB_GROUP/-/group_members).

Then, add MY_GITLAB_BOT_USERNAME as a member of the MY_GITLAB_GROUP group.

This will allow MY_GITLAB_BOT_USERNAME to create new repositories within the MY_GITLAB_GROUP group.

Go to your GitHub fork of MirrorUpdater.jl: https://github.com/MY_GITHUB_ORG/MirrorUpdater.jl

In your GitHub fork of MirrorUpdater.jl, update lines 1 and 2 of config/gitlab.jl to look like:

const GITLAB_GROUP = "MY_GITLAB_GROUP"
const GITLAB_BOT_USERNAME = "MY_GITLAB_BOT_USERNAME"

Leave the rest of config/gitlab.jl unchanged. Please do not stored your personal access token in the file.

In your GitHub fork of MirrorUpdater.jl, update line 2 of config/enabled-providers.jl to look like:

const GITLAB_ENABLED = true

Congratulations, you have finished this section.

If you do not already have a personal Bitbucket account, create one. For the remainder of this README, MY_PERSONAL_BITBUCKET_USERNAME refers to the username of your personal Bitbucket account.

For example, for me, MY_PERSONAL_BITBUCKET_USERNAME is equal to DilumAluthge.

Log in to Bitbucket as MY_PERSONAL_BITBUCKET_USERNAME.

While logged in as MY_PERSONAL_BITBUCKET_USERNAME, enable two-factor authentication on the MY_PERSONAL_BITBUCKET_USERNAME account.

Make sure to store your two-factor recovery codes in a secure location!

While logged in as MY_PERSONAL_BITBUCKET_USERNAME, create a free Bitbucket team that you will use only for hosting the mirrored repositories. For the remainder of this README, MY_BITBUCKET_TEAM refers to the name of this team. MY_PERSONAL_BITBUCKET_USERNAME should be an owner of the MY_BITBUCKET_TEAM team.

For example, for me, MY_BITBUCKET_TEAM is equal to UnofficialJuliaMirror.

While logged in as MY_PERSONAL_BITBUCKET_USERNAME, go to the MY_BITBUCKET_TEAM projects page (https://bitbucket.org/MY_BITBUCKET_TEAM/profile/projects).

Then, create a new project inside the MY_BITBUCKET_TEAM team. Make sure to UNCHECK the box next to "This is a private project." We want this project to be a public project.

For the remainder of this README, MY_BITBUCKET_PROJECT refers to the name of this project.

For example, for me, MY_BITBUCKET_PROJECT is equal to UnofficialJuliaMirrorProject.

Log out of the MY_PERSONAL_BITBUCKET_USERNAME account.

Create a new Bitbucket "bot" account that you will use ONLY for maintaining the mirror. For the remainder of this README, MY_BITBUCKET_BOT_USERNAME refers to the username of this account.

For example, for me, MY_BITBUCKET_BOT_USERNAME is equal to UnofficialJuliaMirrorBot.

Log in to Bitbucket as MY_BITBUCKET_BOT_USERNAME.

While logged in as MY_BITBUCKET_BOT_USERNAME, enable two-factor authentication on the MY_BITBUCKET_BOT_USERNAME account.

Make sure to store your two-factor recovery codes in a secure location!

While logged in as MY_BITBUCKET_BOT_USERNAME, create an app password for the MY_BITBUCKET_BOT_USERNAME account and store it in a secure place (such as a password manager). For the remainder of this README, my-bitbucket-bot-app-password refers to this app password.

The app password should be treated as securely as any other password. Do not share it with anyone. Do not save it in any unsecure location. Do not save it in a file. Do not commit it in a Git repository.

Log out of the MY_BITBUCKET_BOT_USERNAME account.

Log in to Bitbucket as MY_PERSONAL_BITBUCKET_USERNAME.

While logged in as MY_PERSONAL_BITBUCKET_USERNAME, go to the MY_BITBUCKET_TEAM team members page (https://bitbucket.org/MY_BITBUCKET_TEAM/profile/members).

Then, add MY_BITBUCKET_BOT_USERNAME as a member of the MY_BITBUCKET_TEAM team.

This will allow MY_BITBUCKET_BOT_USERNAME to create new repositories within the MY_BITBUCKET_TEAM team.

Go to your GitHub fork of MirrorUpdater.jl: https://github.com/MY_GITHUB_ORG/MirrorUpdater.jl

In your GitHub fork of MirrorUpdater.jl, update lines 1 and 2 of config/bitbucket.jl to look like:

const BITBUCKET_TEAM = "MY_BITBUCKET_TEAM"
const BITBUCKET_BOT_USERNAME = "MY_BITBUCKET_BOT_USERNAME"

Leave the rest of config/bitbucket.jl unchanged. Please do not stored your personal access token in the file.

In your GitHub fork of MirrorUpdater.jl, update line 3 of config/enabled-providers.jl to look like:

const BITBUCKET_ENABLED = true

Congratulations, you have finished this section.

Log in to GitHub as MY_PERSONAL_GITHUB_USERNAME.

Log in to Travis using the GitHub account MY_PERSONAL_GITHUB_USERNAME: https://travis-ci.com/

Enable Travis for your fork: https://travis-ci.com/profile/MY_GITHUB_ORG

Go to the Travis settings page for your fork: https://travis-ci.com/MY_GITHUB_ORG/MirrorUpdater.jl/settings

In the "General" section of the Travis settings page, turn ON the switch next to "Limit concurrent jobs". Then, enter 1 in the box to the right.

This step is important. You must limit the concurrent jobs to 1. If you do not, then you will probably trigger the API rate limits for GitHub, GitLab, and/or Bitbucket, which will cause your Travis jobs to fail.

In the "Environment Variables" section of the Travis settings page, add a new environment variable with name equal to GITHUB_BOT_PERSONAL_ACCESS_TOKEN and value equal to my-github-bot-personal-access-token. Make sure that the "Display value in build log" option is turned OFF.

In the "Cron Jobs" section of the Travis settings page, create a new cron job for your fork. For "Branch", select master. For "Interval", select weekly. For "Options", select Do not run if there has been a build in the last 24h.

Congratulations, you have finished this section.

Download the code from your fork:

git clone https://github.com/MY_GITHUB_ORG/MirrorUpdater.jl

cd into the MirrorUpdater.jl directory:

cd MirrorUpdater.jl

Install the dependencies of the project:

julia --project -e 'import Pkg; Pkg.resolve();'

Build the project:

julia --project -e 'import Pkg; Pkg.build("MirrorUpdater");'

Run the package tests:

julia --project -e 'import Pkg; Pkg.test("MirrorUpdater");'

Set the appropriate environment variables:

export GITHUB_BOT_PERSONAL_ACCESS_TOKEN="my-github-bot-personal-access-token"

export GITLAB_BOT_PERSONAL_ACCESS_TOKEN="my-gitlab-bot-personal-access-token"

export BITBUCKET_BOT_APP_PASSWORD="my-bitbucket-bot-app-password"

Run the updater:

julia --project run-github-mirror-updater.jl

Congratulations, you have finished this section.