Checkpoints.jl

A package for dynamically checkpointing program state
Author invenia
Popularity
12 Stars
Updated Last
1 Year Ago
Started In
December 2020

Checkpoints

stable build status coverage

Checkpoints.jl allows packages to register checkpoints which can serialize objects to disk during the execution of an application program, if the application program configures them.

A minimal working example consists of the package:

module MyPackage

using Checkpoints

MODULE = "MyPackage"

__init__() = Checkpoints.register(MODULE, ["foo", ])

function foo(x)
    with_checkpoint_tags(:foo1 => 1, :foo2 => 2) do
        checkpoint(MODULE, "foo", :data => 2x)
    end
    return 2x
end

end

and the application program:

using Checkpoints

Checkpoints.config("MyPackage.foo", "./path/to/checkpoints")

for i in 1:2
    with_checkpoint_tags(:iteration => i) do
        MyPackage.foo(1.0)
    end
end

which results in recorded checkpoints at

./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso
./path/to/checkpoints/iteration=2/foo1=1/foo2=2/MyPackage/foo.jlso

You can use index_checkpoint_files to get an index of the files, which is a Tables.jl table and so can e.g. be passed to DataFrame (and then you can do things like groupby etc):

julia> using DataFrames

julia> DataFrame(index_checkpoint_files("./path/to/checkpoints/"))
2×6 DataFrame
 Row │ prefixes        checkpoint_name  iteration   foo1        foo2        checkpoint_path
     │ Tuple          SubString       SubString  SubString  SubString  PosixPath
─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ ("MyPackage",)  foo              1           1           2           ./path/to/checkpoints/iteration=
   2 │ ("MyPackage",)  foo              2           1           2           ./path/to/checkpoints/iteration=

or worked with directly:

julia> [checkpoint_path(out) for out in index_checkpoint_files("./path/to/checkpoints/") if out.iteration=="1"]
1-element Array{FilePathsBase.PosixPath,1}:
 p"./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso"

Used By Packages

No packages found.