This package was presented at JuliaCon, and the presentation is available under this link.
This package provides a rough interface to create Singularity containers from DrWatson Projects. It currently works best on Linux systems, as the build command currently not available on Mac.
Most of the code is still very WIP and based on my own processes and needs. If you use Singularity, or have a use case that I haven't considered yet, please reach out either by email or by opening an issue.
The Sylab documentation contains instructions to install Singularity, but appears to be slightly out of date. You can find additional information for
- Linux on this Github readme
- Mac on the download page for the beta release
- Windows, see this this issue stating that WSL 2 is required (instructions). After installing it, follow the instructions for Linux.
Alternatively, you can use the instructions for Vagrant on Windows and Mac.
This package uses a minimal debian-based container with Julia installed as a base. On the Sylab cloud you can find the Juliabase image, and an experimental container also including jupyter
If you would prefer to build them yourself, the def files are available in the
The package assumes that that the folder structure contains the following elements
├── scripts │ ├── run ├── src │ ├── module1 │ ├── module2 ├── container ├── <other folders> ├── Project.toml ├── Manifest.toml
and everything is under the control of a single git repository. This will be automatically the case if the project folder was created by DrWatson.jl.
The src and scripts folder will be copied into the container, and it is further assumed that the modules in it are registered as
deved in the project
buildsif will ask for root privileges, as the underlying
singularity build commands requires it. This is clearly a potential security risk, so if you are unsure, please inspect the
Singularity.pack file in the
The package provides the following functions. All these functions work from any folder as long as the correct project environment is loaded. They are also still WIP, so there is very little error checking being done.
generate_deffile(; excludepkgs = , commit = "master")
container folder if it does not exist yet, and generates the
Singularity.pack def file.
excludepkgsaccepts and array of package names. These packages will be removed from
Project.tomlinside the container. This is for packages that are needed locally, for example for visualization, but are not needed in the container and would only add bloat.
commitaccept any project commit hash, and will build the container using the
scriptfolder from that commit. Requires the git setting below.
buildsif(;verbose = false, force = true)
Builds the container image into the
container folder based on the existing def file.
verbosesends all the output of the build process to the REPL if set to
true, otherwise it will be written to file.
truecauses an existing image to be overwritten without asking for confirmation.
recreatedata(file::String; dir = )
Extracts the git commit hash and script name from a
DrWatson.@tagsaved file, and generated a def file. The resulting container, when
singularity run, should recreate the initial file directly.
dirallows the specification of a subdirectory of the
Transfers the image to the
host into a folder in the home directory of the same name as the project folder. This assumed that everything is configured such that
ssh host just works.
Currently, the commands build a single read-only image. This means, that after any change in the project the entire image needs to be rebuilt. This is partly as intended, as the result is a tamper-proof complete environment, that can be used at any point in the future the return the exact same results. However for projects that are still under more rapid development, I have possible ideas to make that initial phase not require frequent lengthy rebuilds.
- Generate different def files
- add interaction with singularity cloud and hub (pushing and pulling)
- add tests
- add various error handling and options
- (big) do some remote builder magic to make this work on windows/ mac
- Automate image build on repo push, as mentioned on singularity hub
- (bigger) add singularity binary ?