BenchmarkCI.jl

Author tkf
Popularity
16 Stars
Updated Last
4 Months Ago
Started In
January 2020

BenchmarkCI.jl

Lifecycle CI Status codecov.io

BenchmarkCI.jl provides an easy way to run benchmark suite via GitHub Actions. It only needs a minimal setup if there is a benchmark suite declared by BenchmarkTools.jl / PkgBenchmark.jl API.

Warning This package is still experimental. Make sure to fix the version number in your CI setup.

Setup

BenchmarkCI.jl requires PkgBenchmark.jl to work. See Defining a benchmark suite · PkgBenchmark.jl for more information. BenchmarkCI.jl also requires a Julia project benchmark/Project.toml that is used for running the benchmark.

Create a workflow file (required)

Create (say) .github/workflows/benchmark.yml with the following configuration:

name: Run benchmarks

on:
  pull_request:

jobs:
  Benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@latest
        with:
          version: 1.4
      - name: Install dependencies
        run: julia -e 'using Pkg; pkg"add PkgBenchmark BenchmarkCI@0.1"'
      - name: Run benchmarks
        run: julia -e 'using BenchmarkCI; BenchmarkCI.judge()'
      - name: Post results
        run: julia -e 'using BenchmarkCI; BenchmarkCI.postjudge()'
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Setup with benchmark/Manifest.toml

If benchmark/Manifest.toml is checked into the repository, benchmark/Project.toml must include parent project as well. Run dev .. in benchmark/ directory to add it:

shell> cd ~/.julia/dev/MyProject/

shell> cd benchmark/

(@v1.x) pkg> activate .
Activating environment at `~/.julia/dev/MyProject/benchmark/Project.toml`

(benchmark) pkg> dev ..

Additional setup (recommended)

It is recommended to add following two lines in .gitignore:

/.benchmarkci
/benchmark/*.json

This is useful for running BenchmarkCI locally (see below).

Printing benchmark result (optional)

Posting the benchmark result as a comment for every push for each PR may be too noisy. In such case, using BenchmarkCI.displayjudgement() instead of BenchmarkCI.postjudge() may be useful.

      - name: Print judgement
        run: julia -e 'using BenchmarkCI; BenchmarkCI.displayjudgement()'

Running BenchmarkCI interactively

shell> cd ~/.julia/dev/MyProject/

julia> using BenchmarkCI

julia> BenchmarkCI.judge()
...

julia> BenchmarkCI.displayjudgement()
...

Used By Packages

No packages found.