Adaptive stress testing of black-box systems within POMDPs.jl
Author sisl
12 Stars
Updated Last
1 Year Ago
Started In
February 2020


Documentation Build Status codecov

Adaptive stress testing of black-box systems, implemented within the POMDPs.jl ecosystem.

See the documentation for more details.


If you use this package for research purposes, please cite the following:


  title = {{POMDPStressTesting.jl}: Adaptive Stress Testing for Black-Box Systems},
  author = {Robert J. Moss},
  journal = {Journal of Open Source Software},
  year = {2021},
  volume = {6},
  number = {60},
  pages = {2749},
  doi = {10.21105/joss.02749}


To stress test a new system, the user has to define the GrayBox and BlackBox interface outlined in src/GrayBox.jl and src/BlackBox.jl.

GrayBox Interface

The GrayBox simulator and environment interface includes:

  • GrayBox.Simulation type to hold simulation variables
  • GrayBox.environment(sim::Simulation) to return the collection of environment distributions
  • GrayBox.transition!(sim::Simulation) to transition the simulator, returning the log-likelihood

BlackBox Interface

The BlackBox system interface includes:

  • BlackBox.initialize!(sim::Simulation) to initialize/reset the system under test
  • BlackBox.evaluate!(sim::Simulation) to evaluate/execute the system under test
  • BlackBox.distance(sim::Simulation) to return how close we are to an event
  • BlackBox.isevent(sim::Simulation) to indicate if a failure event occurred
  • BlackBox.isterminal(sim::Simulation) to indicate the simulation is in a terminal state

Functions ending with ! may modify the Simulation object in place.


Several solvers are implemented.

Reinforcement learning solvers

Deep reinforcement learning solvers1

Stochastic optimization solvers

Baseline solvers


Example Notebook

An example implementation of the AST interface is provided for the Walk1D problem:


Install the POMDPStressTesting.jl package via:

] add POMDPStressTesting


To run the test suite, you can use the Julia package manager.

] test POMDPStressTesting


We welcome contributions! Please fork the repository and submit a new Pull Request.

Package maintained by Robert Moss: mossr@cs.stanford.edu

1 TRPO and PPO thanks to Shreyas Kowshik's initial implementation.