ReferenceTests

ReferenceTests.jl is a Julia package that adds a couple of additional macros to your testing toolbox. In particular, it focuses on functionality for testing values against reference files, which in turn the package can help create and update if need be. ReferenceTests.jl is build on top of FileIO.jl and designed to be used alongside Base.Test.

Package Status	Package Evaluator	Build Status

Introduction

It is very common for Julia packages to test the functionality of their exported functions against known input-to-output combinations. We will refer to such kind of tests as reference tests. In most cases these will be quite simple; something along the line of @test f(x) == y, where f is a function of the user package and x is some interesting input value for which the desired output y is known.

For testing the output of more complex functions, for which the expected output is more complicated (e.g. anything image processing related), using @test can be a little cumbersome to work with. To that end this package provides the @test_reference macro, which expects a filename (relative to the file that invokes the macro) and an expression that evalutes to the value of interest.

using ReferenceTests
@test_reference "stringtest1.txt" collect(1:20)

If you put the above code into your test/runtests.jl and execute the file in an interactive julia session (i.e. with include), then it will trigger an interactive dialog if the results don't match. This dialog allows the user to update the reference files. If you do not want to be prompted, just delete the reference data before running the tests.

The given file stringtest1.txt is assumed to be the relative path to the file that contains the macro invocation. This likely means that the path is relative to the test/ folder of your package.

The file-extension of the filename (here txt), as well as the type of the result of evaluating the expression (here String), determine how the actual value is compared to the reference value. The default implementation will do a simple equality check with the result of FileIO.load. This means that it is the user's responsibility to have the required IO package installed.

Colorant arrays (i.e.) receive special treatment. If the extension of the filename is txt then the package ImageInTerminal.jl will be used to create a string-based crude approximation of the image. This will have low storage requirements and also allows to view the reference file in a simple terminal using cat.

using ReferenceTests, TestImages
@test_reference "imagetest1.txt" testimage("cameraman")

Note that while a text-based storage of reference images can be convenient, proper image formats (e.g. png) are also supported by the package. Those, however, will require the proper FileIO backends to be installed.

Another special file extension is sha256 which will cause the hash of the result of the given expression to be stored and compared as plain text. This is useful for a convenient low-storage way of making sure that the return value doesn't change for selected test cases.

Updating References within Package Tests

Reference tests are typically used within a package's test/runtests.jl test suite. These tests are easy to run via pkg> test but the child process used within pkg> test is non-interactive, so the update prompt will not show if there are mismatches.

To update references within a package test suite, there are three options:

• Set the environment variable JULIA_REFERENCETESTS_UPDATE to "true" and run pkg> test, which will force update any non-matches. You can then check changes to any git-tracked reference images before commit.
• Delete any reference images you wish to update and run pkg> test, given that missing references are created automatically.
• Run the test/runtests.jl interactively. This may be easier using the TestEnv.jl package, given that the test environment used by pkg> test is a merge of the src/Project.toml and test/Project.toml environments.

Documentation

Check out the stable documentation or dev documentation.

Additionally, you can make use of Julia's native docsystem. The following example shows how to get additional information on @test_reference within Julia's REPL:

?@test_reference

License

This code is free to use under the terms of the MIT license.