PerfChecker is a set of performance checking tools for Julia packages. The ultimate aim is to create an environment where the tool can run similarly to a test environment. By doing so, it would be possible to test the performance of a package
P in separate Julia instances. This would allow for each version of
- The use of the latest compatible versions of Julia and other dependencies of P
- Independence of compatibility requirements of
PerfChecker.jlfrom the environment used during performance checks.
Google Summer of Code (2023)
PerfChecker.jl, is participating in Google Summer of Code (GSoC) through the Julia language umbrella and is looking for contributors. Complete lists of projects:
This package consists of a set of tools designed to check the performance of packages over time and versions. The targeted audience is the whole community of packages' developers in Julia (not only JuliaConstraints).
This README provides a short demo on how PerfChecker can be used.
Basic features to implement (length ≈ 175 hours)
- PerfCheck environment similar to Test.jl and Pkg.jl
- Sugar syntax
@profilesimilar to Test.jl and Pkg.jl
- Interactive REPL interface
- Interactive GUI interface (using for instance Makie)
- Automatic Profiling ? (not sure how, there already is a bunch of super cool packages)
- Automatic plotting of previous features
Advanced features (length +≈ 175 hours)
- Smart semi-automatic analysis of performances
- Performances bottlenecks
- Allocations vs speed trade-off
- Descriptive plot captions
- Handle Julia and other packages versions
- Integrates with juliaup
- Automatically generate versions parametric space for both packages and Julia
Note that some features are interchangeable depending on the interest of the candidate. For candidates with a special interest in the JuliaConstraints ecosystem, checking the performances of some packages is an option.
175 hours – 350 hours (depending on features)
Recommended Skills (||)
- Familiarity with package development
- REPL and/or GUI interfaces
- Coverage, Benchmarks, and Profiling tools
Easy to Medium, depending on the features implemented
Although it is part of JuliaConstraints,
PerfChecker is a standalone project. As such a good start is to understand fully its features and workflow. For instance, one way is to write a small use case in the vein of the small tutorial below. Possible packages could be
- A JuliaConstraints package or dependency
- A package written by the GSoC candidate
- Another package from the Julia community
Please bear in mind that, ideally, writing performance checks for such a package should be simple.
Also, allocation checks generate memory files in the package local folder. Ideally the package should be
deved in a local environment.
To contribute, please fork the repo, create a new branch, make your changes, and submit a pull request. If you are unsure about anything or need any help, please don't hesitate to ask through issues, JuliaConstraints chat, or the
#juliaconstraints channel on Humans-of-Julia's Discord.
We encourage students and other possible GSoC contributors to participate in
PerfChecker's development as it would bring a tools for the Julia community as a whole. It would bring them experience and deep understanding of Julia packages development and, more generally, open source development along with performance testing.
We're looking forward for proposals submissions.
This tutorial is based on a beta version and is prone to change frequently. Please use it as a workflow example.
Let's write two small scripts to check allocations (
allocs.jl) and benchmarks (
bench.jl) for CompositionalNetworks.jl using
In the current state, we write and execute the scripts (and stores a local environment) in the
/perf folder of
CompositionalNetworks.jl. You can use
julia --project to activate that environment when running the script. For instance, I run the check for
CompositionalNetworks.jl with the following command,
julia -t 10 --project
To generate results from different versions of the targeted package, be sure to change the version in the local environment.
We will generate plots from the allocations check and the benchmark check from the
Please add to the environment the following packages (adapt to your use case):
Test.jl: for allocations
BenchmarkTools: for benchmarks
Remark on code compilation and
Depending on the nature of your code, it is important to be sure to trigger all compilation previous to the allocation check. This role is annotated in both scripts.
Note that in the case of
CompositionalNetworks.jl, it stochastically generates a great deal of methods to compile.
For deterministic code,
pre-alloc() can be minimal, and
@benchmark will handle triggering the necessary compilation before the checks.
The current state of
PerfChecker.jl requires the use of
Test.jl, but this requirement will disappear soon.
# Required to run the script using PerfChecker using Test # Target(s) using CompositionalNetworks # lastest release: 0.3.1 # Direct dependencies of this script using ConstraintDomains @testset "PerfChecker.jl" begin # Title of the alloc check (for logging purpose) title = "Explore, Learn, and Compose" # Dependencies needed to execute pre_alloc and alloc dependencies = [CompositionalNetworks, ConstraintDomains] # Target of the alloc check targets = [CompositionalNetworks] # Code specific to the package being checked domains = fill(domain([1, 2, 3]), 3) # Code to trigger precompilation before the alloc check pre_alloc() = foreach(_ -> explore_learn_compose(domains, allunique), 1:10) # Code being allocations check alloc() = explore_learn_compose(domains, allunique) # Actual call to PerfChecker alloc_check(title, dependencies, targets, pre_alloc, alloc; path=@__DIR__, threads=10) end
This script will output the table below (and store it as
mallocs/mallocs-0.3.1.csv). Note that the allocations are provided in decreasing order. The
.mem files generated by tracking allocations are automatically deleted (unless your code run into an error).
BenchmarkTools.jl provides already a great set of functionalities, we use it directly. In the future, it is likely that
PerfChecker.jl will provide synthetic sugar to wrap
@benchmark with similar behavior to make using
# Required to run the script using PerfChecker using BenchmarkTools # Target(s) using CompositionalNetworks # lastest release: 0.3.1 # Direct dependencies of this script using ConstraintDomains # Target of the benchmark target = CompositionalNetworks # Code specific to the package being checked domains = fill(domain([1, 2, 3, 4]), 4) # Code to trigger precompilation before the bench (optional) foreach(_ -> explore_learn_compose(domains, allunique), 1:10) # Code being benchmarked (be sure to enforce specific amounts of evals and samples for each version benchmarked) bench = @benchmark explore_learn_compose(domains, allunique) evals=1 samples=10 seconds=3600 # Store the bench results store_benchmark(bench, target; path=@__DIR__)
This script will output the results of
@benchmark as a table (and store it as
benchmarks/benchmark-0.3.1.csv). Note that it is recommended (but not necessary) to ensure that for each version of the package benchmarked, the output is of similar length.
We will generate some plots, in
perf/benchmarks. In the REPL (or a notebook), please run:
using PerfChecker using CompositionalNetworks alloc_plot([CompositionalNetworks]) bench_plot([COmpositionalNetworks])
Allocs (Pie Chart)
For each version checked with the previous scripts, we get a pie plot showing the distribution of the allocations (per line). Obviously, improving the allocations at the 5th line of
metrics.jl would improve allocations (and likely overall performances) in
CompositionalNetworks.jl. Let's try to spot issues through the evolution of allocations over time.
Allocs over time
Luckily, an overview of the evolution of the allocations within each file is also plotted. The allocations in
CompositionalNetworks.jl improve a lot from
v0.3.x. Interestingly, the changes also introduced an increase in allocations in the
metrics.jl file. Maybe there really is an issue (answer in future releases of
First, we should check how the performances are impacted by the changes in memory allocations.
Benchmarks (allocs and memory)
To confirm the improvement of allocations above, let's have a look at the evolution of allocations and memory use over time.
Both distribution of allocations and memory is very stable. This meet the improvement of the design of
CompositionalNetworks.jl that ensure allocations of one data structure at the start of each
explore_learn_compose call evaluated by both scripts.
Benchmarks (times and garbage collection times)
Garbage collection brings a lot of comfort for programmers, and it participates in the attractiveness of the Julia language. However, careless allocations can be a performance pitfall. Such was the case of
CompositionalNetworks.jl prior to
The changes introduce from that version clearly improved our GC issues. Does it reflect on the global time performance? (spoiler: yes, it does, cf next plot)
We can remark important deviations (beware the logarithmic scale ...) from the mean. As mentioned above,
CompositionalNetworks.jl uses a stochastic process, so it is not surprising. At least, memory (allocations) are stable.
Benchmarks (evolutions overview)
Well, we probably could get the gist of the previous 4 plots from the wrap-up plot below.
Note that the analysis on memory stability despite a stochastic process that reflect on the
gctimes is not possible here. But it looks much better if you only can show off one performance plot.
We appreciate contributions from users including reporting bugs, fixing issues, improving performance and adding new features.
To contribute, please fork the repo, create a new branch, make your changes, and submit a pull request. If you are unsure about anything or need any help, please don't hesitate to ask.
This package is part of the JuliaConstraints project. We thank the entire community for their contributions.