StorageMirrorServer.jl

As I want it be available, fast, complete and persistent
Author johnnychen94
Popularity
6 Stars
Updated Last
1 Year Ago
Started In
April 2020

StorageMirrorServer

Unit Test PkgEval Codecov status

This package is used to set up a Julia Package Storage Server for mirror sites. The protocol details are described in https://github.com/JuliaLang/Pkg.jl/issues/1377.

TL;DR; A storage server contains all the static contents you need to download when you do ]add PackageName.

If you just want a cache layer service, PkgServer.jl is a better choice. This package is made to permanently keep the static contents.

To set up a storage server, you'll need to:

  1. get/update the static contents
  2. serve them as a HTTP(s) service using nginx or whatever you like

This package is written to make step 1 easy and stupid.

Basic Usage

  1. add this package ]add StorageMirrorServer
  2. modify the example script and save it as gen_static.jl
  3. pull/build data julia gen_static.jl

You can read the not-so-friendly docstrings for advanced usage, but here are something you may want:

  • Redirect output julia gen_static.jl > log.txt 2>&1
  • Utilize multiple threads, set environment variable JULIA_NUM_THREADS. For example, JULIA_NUM_THREADS=8 julia gen_static.jl would use 8 threads to pull data.

Environment Variables

There are some environment variables that you can use to help configure the download worker curl:

  • BIND_ADDRESS that passes to curl --interface $BIND_ADDRESS, this can be useful when multiple network cards are available (newly added in v0.2.1)

Examples

This package is used to power the Julia pkg mirror in the following mirror sites:

  • BFSU in Beijing Foreign Studies University
  • TUNA in Tsinghua University
  • SJTUG in Shanghai Jiao Tong University
  • USTC in University of Science and Technology of China

Acknowledgement

The first version of this package is modified from the original implementation gen_static.jl. During the development of this package, I get a lot of helps from Elliot Saba and Stefan Karpinski to understand the Pkg & Storage protocol designs. Chi Zhang from SJTUG has shown his great patience and passion in testing out the rolling versions and given me a lot of feedbacks and suggestions.

Used By Packages

No packages found.