ReinforcementLearningTrajectories
Design
The relationship of several concepts provided in this package:
┌───────────────────────────────────┐
│ Trajectory │
│ ┌───────────────────────────────┐ │
│ │ AbstractTraces │ │
│ │ ┌───────────────┐ │ │
│ │ :trace_A => │ AbstractTrace │ │ │
│ │ └───────────────┘ │ │
│ │ │ │
│ │ ┌───────────────┐ │ │
│ │ :trace_B => │ AbstractTrace │ │ │
│ │ └───────────────┘ │ │
│ │ ... ... │ │
│ └───────────────────────────────┘ │
│ ┌───────────┐ │
│ │ Sampler │ │
│ └───────────┘ │
│ ┌────────────┐ │
│ │ Controller │ │
│ └────────────┘ │
└───────────────────────────────────┘
Trajectory
A Trajectory contains 3 parts:
- A
containerto store data. (Usually anAbstractTraces) - A
samplerto determine how to sample a batch fromcontainer - A
controllerto decide when to sample a new batch from thecontainer
Typical usage:
julia> t = Trajectory(Traces(a=Int[], b=Bool[]), BatchSampler(3), InsertSampleRatioControler(1.0, 3));
julia> for i in 1:5
push!(t, (a=i, b=iseven(i)))
end
julia> for batch in t
println(batch)
end
(a = [4, 5, 1], b = Bool[1, 0, 0])
(a = [3, 2, 4], b = Bool[0, 1, 1])
(a = [4, 1, 2], b = Bool[1, 0, 1])Traces
TracesMultiplexTracesCircularSARTTracesEpisodeEpisodes
Samplers
BatchSamplerMetaSamplerMultiBatchSampler
Controllers
InsertSampleRatioControllerAsyncInsertSampleRatioController
Please refer tests for common usage. (TODO: generate docs and add links to above data structures)
Acknowledgement
This async version is mainly inspired by deepmind/reverb.