DistributedReinforcementLearning.jl

If it works, it works everywhere!
Author JuliaReinforcementLearning
Popularity
2 Stars
Updated Last
3 Months Ago
Started In
October 2020

If it works, it works everywhere!


Design

Components

  • 👷 Worker, a worker creates a task to run an experiment in background. It periodically sends out transitions between agent and environment, and fetches latest parameter.
  • 📢 WorkerProxy, a worker proxy collects messages from/to workers on the same node so that some message data (model params) can be shared accross different workers.
  • 💿 TrajectoryManager, a trajectory manager is a wrapper around an AbstractTrajectory. It takes in a bulk of transitions and samples a batch of training data in respond to rquests.
  • 💡 Trainer, a trainer is a wrapper around an AbstractPolicy, it does nothing but to update its internal parameters when received a batch of training data and periodically broadcast its latest parameters.
  • ⏱️ Orchestrator, an orchestrator is in charge of controlling the start, stop and the speed of communications between the above components.

Note that:

  1. We adopt the actor model here. Each instance of the above components is an actor. Only messages are passing between them.
  2. A node is a process in Julia. Different nodes can be on one machine or accross different machines.
  3. Tasks in different workers are initiated with Threads.@spawn. There's no direct communication between them by design.
  4. In single node environment (WorkerNode and MainNode are the same one), the WorkerProxy can be removed and workers communicate with Orchestrator directly.

Messages

  • 1️⃣ (👷📢) InsertTransitionMsg, contains the local transitions between agent and environment in an experiment.
  • 2️⃣ (📢⏱️) InsertTransitionMsg from different workers.
  • 3️⃣ (⏱️💿) InsertTransitionMsg and SampleBatchMsg (which contains the address of Trainer).
  • 4️⃣ (💿💡) BatchTrainingDataMsg
  • 5️⃣ (💡💿) UpdatePriorityMsg, only necessary in prioritized experience replay related algorithms.
  • 6️⃣ (💡⏱️) LoadParamsMsg, contains the latest parameters of the policy.
  • 7️⃣ (⏱️📢) LoadParamsMsg
  • 8️⃣ (📢👷) LoadParamsMsg

Used By Packages

No packages found.