What were my hyper-parameter settings again?
Author mkschleg
23 Stars
Updated Last
1 Year Ago
Started In
December 2018


Build Status codecov

How did I get these results?

A framework for managing hyper-parameter settings, and running experiments. Lightly based on litetracer from Borealis.

What is this?

This repository is for giving you the tools you need to make experiments reproducible. This repository is mostly built around machine learning and reinforcement learning projects, but there is no reason it is restricted to these types of projects. I've developed this around my own tastes and needs, but should be generally usable for any style of experiment which needs to do massively parallel parameter sweeps of a set of functions.

How To use

The best way to see how to use this repository is in the examples folder. parallel.jl shows all the parallel and experiment utilities in action, while running an experiment in experiment.jl which shows off the data management, and arg parsing.


To run the config example:

julia --project examples/parallel_config.jl examples/test_config.toml 5 test_config --numworkers 6


Temporarily, I've updated a local version of the SLURM cluster managers as the ClusterManagers.jl package updates to 1.x.

What's next.

The goal is to make a convenient framework for data analysis for experiments in ML/RL. The goal is to create something like mlflow for Julia. It might be worthwhile to actually take advantage of the mlflow framework, or even a similar framework for the future.

Plan for further integration w/ config files

  • Define arg_iterators in terms of config files
  • Define compatibility w/ arg settings and arg parse setting.


  • Allow for TOML/JSON specification -> Implemented through Config.jl
  • Data searching (Mostly done. Could probably use more testing...)
  • Plotting utilities (Up Next!)
  • Frontend for visualizing data.
  • Testing with slurm backend. (Make sure your experiments run BEFORE a job is scheduled.)
  • Move away from JLD2 for settings files. CSV? TOML?
  • More examples. (maybeeee.....)
  • Docs

HELP WANTED! Contact me via issues panel, or setup a PR.

Known Bugs: