TestTools

Documentation
Build Status

TestTools is a collection of CLI tools and APIs that simplifies code testing, coverage analysis, and style checking for the Julia programming language. Our goal is to make it a joy to do software testing (or at least save effort and keystrokes).

• Easy-to-use (and fast) CLI tools for testing

• Compatible with Pkg.test()

• Enhanced test set functionality – diffs for failed comparisons and fail-fast support

• Noninvasive – introduces no package-level dependencies

• Start Julia in the default (global) environment.

  • Note. Installation in the default environment makes the CLI tools available from within all projects.

• Install the TestTools package.

  pkg> add TestTools  # Press ']' to enter the Pkg REPL mode.

• Install the CLI tools (to ~/.julia/bin).

  julia> using TestTools; TestTools.install()

Run unit tests in a single file.

$ jltest test/tests.jl

Run all unit tests contained in a directory.

$ jltest test/

Run unit tests with fail-fast enabled (i.e., halt testing after the first failing test).

$ jltest -x test/tests.jl

Generate a coverage report (after running unit tests while collecting coverage data).

$ julia -e 'import Pkg; Pkg.test("TestTools"; coverage=true)'  # run unit tests

$ jlcoverage  # generate coverage report
-------------------------------------------------------------------------------
File                                  Lines of Code     Missed   Coverage
-------------------------------------------------------------------------------
src/TestTools.jl                                  0          0        N/A
src/jlcodestyle/cli/cli.jl                       34          0     100.0%
...
src/pkg.jl                                       42          3      92.9%
-------------------------------------------------------------------------------
TOTAL                                           289          7      97.6%

Run basic code style check (reformatting of source file disabled).

$ jlcodestyle src/TestTools.jl

$ jlcodestyle --verbose src/TestTools.jl
[ Info: Style = BlueStyle
[ Info: Overwrite = false
Formatting src/TestTools.jl

No style errors found.

$ jlcodestyle examples/jlcodestyle/not-blue-style.jl
Style errors found. Files not modified.

Run code style check with reformatting of source file enabled.

$ jlcodestyle --overwrite examples/jlcodestyle/not-blue-style.jl
Style errors found. Files modified to correct errors.

• TestTools leverages several excellent Julia packages to support its core capabilities.

  • Coverage

  • CoverageTools

  • JuliaFormatter

• TestTools borrows ideas (and some code) from the following great Julia packages.

  • TestSetExtensions

    • The base code for EnhancedTestSet (which implements diffs for comparisons and progress dots) comes directly from TestsetExtensions.ExtendedTestSet.

    • The run_tests() and find_tests() methods are essentially a re-implementation and refactoring of the TestsetExtensions.@includetests macro as methods.

  • SafeTestsets

    • The strategy for isolating tests came from the SafeTestsets.@safetestset macro.

  • jlpkg

    • The strategy for installing CLI executables came from jlpkg.install().

• TestTools was inspired by analogous code testing packages in the Python ecosystem:

  • pytest

  • coverage

  • pycodestyle