PkgEval.jl
Evaluate Julia packages.
Quick start
To use PkgEval.jl, you need to Docker and make sure you can start containers (typically,
you need to be a member of the docker
group):
$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
Start by installing the package:
git clone https://github.com/JuliaCI/PkgEval.jl.git
cd PkgEval.jl
julia --project -e 'import Pkg; Pkg.instantiate()'
Then start Julia with julia --project
and use the following commands to run the tests of a
list of packages on a selection of Julia versions:
julia> using PkgEval
julia> julia_versions = PkgEval.obtain_julia.(["1.3", "nightly"])
2-element Array{VersionNumber,1}:
v"1.3.0"
v"1.4.0-DEV-3c182bc5c2"
julia> PkgEval.run(julia_versions, ["Example"])
2×8 DataFrames.DataFrame. Omitted printing of 1 columns
│ Row │ julia │ registry │ name │ version │ status │ reason │ duration │
│ │ VersionNumber │ String │ String │ Version…⍰ │ Symbol │ Symbol⍰ │ Float64 │
├─────┼─────────────────────────┼──────────┼─────────┼───────────┼────────┼─────────┼──────────┤
│ 1 │ v"1.3.0" │ General │ Example │ v"0.5.3" │ ok │ missing │ 6.94 │
│ 2 │ v"1.4.0-DEV-3c182bc5c2" │ General │ Example │ v"0.5.3" │ ok │ missing │ 6.948 │
Test logs are part of this dataframe in the log
column. For example, in this case:
Resolving package versions...
Installed Example ─ v0.5.3
...
Testing Example tests passed
Other run
methods, that offer more options and control over the testing process, are
available as well. These methods however require you to first prepare the environment
yourself, by calling prepare_registry
to set-up the package registry, and prepare_julia
to download and unpack a binary version of Julia.
Why does my package fail?
If you want to debug why your package fails, it's probably easiest to use an interactive shell:
julia> using PkgEval
julia> julia_version = v"1.3.0" # use `obtain_julia` if you need a specific build
julia> julia_install = PkgEval.prepare_julia(julia_version)
julia> PkgEval.prepare_registry()
julia> PkgEval.run_sandboxed_julia(julia_install)
Now you can install, load end test your package. If that fails because of some missing
dependency, you can just install that using the apt
package manager within the container:
julia> # in the spawned container's Julia session, switch to REPL mode by pressing ;
shell> sudo apt update
shell> sudo apt install ...
Once you've found the missing dependency and verified that it fixes the tests of your package, make a pull request to include the dependency in the default image.
Analyzing results
Most of the time, you will want to compare the results that you obtained. For example:
julia> result = PkgEval.run([v"1.2.0", v"1.4.0-DEV-76ebc419f0"], ["AbstractNumbers"])
2×8 DataFrame. Omitted printing of 1 columns
│ Row │ julia │ registry │ name │ version │ status │ reason │ duration │
│ │ VersionNumber │ String │ String │ Version…⍰ │ Symbol │ Symbol⍰ │ Float64 │
├─────┼─────────────────────────┼──────────┼─────────────────┼───────────┼────────┼───────────────┼──────────┤
│ 1 │ v"1.2.0" │ General │ AbstractNumbers │ v"0.2.0" │ ok │ missing │ 24.768 │
│ 2 │ v"1.4.0-DEV-76ebc419f0" │ General │ AbstractNumbers │ v"0.2.0" │ fail │ test_failures │ 26.803 │
If you simply want to compare two Julia versions, use PkgEval.compare
:
julia> PkgEval.compare(result, v"1.2.0", v"1.4.0-DEV-76ebc419f0")
On v1.4.0-DEV-76ebc419f0, out of 1 packages 0 passed, 1 failed, 0 got killed and 0 were skipped.
Comparing against v1.2.0:
- AbstractNumbers status was ok, now fail (reason: test_failures)
In summary, 0 packages now succeed, while 1 have started to fail.
For more extensive evaluations, or when more versions are involved, use PkgEval.render
to generate a HTML site in the website/build
directory at the root of the repository:
julia> PkgEval.render(result)
Generating site at /home/tim/Julia/pkg/PkgEval/site/build
Choosing a different version of Julia
PkgEval ultimately needs a binary build of Julia to run tests with, but there's multiple
options to provide such a build. The easiest option is to use a version number that has
already been registered in the Versions.toml
database, together with an URL and hash to
download an verify the file. An error will be thrown if the specific version cannot be
found. This is done automatically when the prepare_julia
function is called (you will need
to call this method explicitly if you use a lower-level interface, i.e., anything but the
run
function from the quick start section above):
julia> PkgEval.prepare_julia(v"1.2.0-nonexistent")
ERROR: Requested Julia version not found
Alternatively, you can download a named release as listed in Releases.toml
. By calling
obtain_julia_release
with a release name, this release will be downloaded, hashed, and
added to the Versions.toml
database for later use. The method returns the version number
that corresponds with this added entry; you should use it when calling into other functions
of the package:
julia_version = PkgEval.obtain_julia_release("nightly")
PkgEval.run([julia_version], ...)
For even more control, you can build Julia by calling the perform_julia_build
function,
passing a string that identifies a branch, tag or commit in the Julia Git repository:
julia_version = PkgEval.perform_julia_build("master")
Similarly, this function returns a version number that corresponds with an entry added to
Versions.toml
:
["1.4.0-DEV-8f7855a7c3"]
file = "julia-1.4.0-DEV-8f7855a7c3.tar.gz"
sha = "dcd105b94906359cae52656129615a1446e7aee1e992ae9c06a15554d83a46f0"
If you get a permission error while building Julia, try to set the variable
BINARYBUILDER_RUNNER=privileged
, restart Julia and try the build again.
To facilitate all this, there's a higher-level function obtain_julia
that will try each of
the above methods until a valid version is found and returned. It is of course also possible
to build Julia yourself, in which case you will need to create a tarball, copy it to the
deps/downloads
directory, and add a correct version stanza to Versions.toml
.