Performance evaluation tools for multiclass, multirater classification models
Author beacon-biosignals
16 Stars
Updated Last
1 Year Ago
Started In
December 2020


CI codecov Docs: stable Docs: development

Lighthouse.jl is a Julia package that standardizes and automates performance evaluation for multiclass, multirater classification models. By implementing a minimal interface, your classifier automagically gains a thoroughly instrumented training/testing harness (Lighthouse.learn!) that computes and logs tons of meaningful performance metrics to TensorBoard in real-time, including:

  • test set loss
  • inter-rater agreement (e.g. Cohen's Kappa)
  • PR curves
  • ROC curves
  • calibration curves

Lighthouse itself is framework-agnostic; end-users should use whichever extension package matches their desired framework (e.g.

This package follows the YASGuide.


To install Lighthouse for development, run:

julia -e 'using Pkg; Pkg.develop(PackageSpec(url=""))'

This will install Lighthouse to the default package development directory, ~/.julia/dev/Lighthouse.


Note that Lighthouse's LearnLogger logs metrics to a user-specified path in TensorBoard's logdir format. TensorBoard can be installed via python3 -m pip install tensorboard (note: if you have tensorflow>=1.14, you should already have tensorboard). Once TensorBoard is installed, you can view Lighthouse-generated metrics via tensorboard --logdir path where path is the path specified by Lighthouse.LearnLogger. From there, TensorBoard itself can be used/configured however you like; see for more information.

You can use alternative loggers, as long as they comply with the logging interface.