GridWorlds.jl

Help! I'm lost in the flatland!
Author JuliaReinforcementLearning
Popularity
19 Stars
Updated Last
1 Year Ago
Started In
August 2020

GridWorlds

A package for creating grid world environments for reinforcement learning in Julia. The focus of this package is on being lightweight and efficient.

This package is inspired by gym-minigrid. In order to cite this package, please refer to the file CITATION.bib. Starring the repository on GitHub is also appreciated. For benchmarks, refer to benchmark/benchmark.md.

Table of contents:

Getting Started

import GridWorlds

env = GridWorlds.EmptyRoomDirected()

env(GridWorlds.MOVE_FORWARD)
env(GridWorlds.TURN_LEFT)
env(GridWorlds.TURN_RIGHT)

import ReinforcementLearningBase

ReinforcementLearningBase.state(env)
ReinforcementLearningBase.action_space(env)
ReinforcementLearningBase.reward(env)
ReinforcementLearningBase.is_terminated(env)
ReinforcementLearningBase.reset!(env)

# play interactively using Makie.jl
# you need to first manually install Makie.jl with the following command
# ] add Makie
# first time plot may be slow
import Makie

GridWorlds.play(env, file_name = "example.gif", frame_rate = 24)

Design

Reinforcement Learing API for the environments

This package uses the API provided in ReinforcementLearningBase.jl so that it can seamlessly work with the rest of the JuliaReinforcementLearning ecosystem.

Representation of an environment

An environment instance (often named env) contains within it an instance of GridWorldBase (often named world). A world contains a 3-D boolean array (which is a BitArray{3} and is often named grid) of size (num_objects, height, width). Each tile of the grid can have multiple objects in it, as represented by a multi-hot encoding along the first dimension of the grid. The objects contained in the world do not contain any fields. Any information related to such objects that needs to be stored is cached separately as fields of env.

Customizing an existing environment

The behaviour of environments is easily customizable. Here are some of the things that one may typically want to customize:

  1. Keyword arguments allow for enough flexibility in most environments. For example, most environments allow creation of rectangular worlds.
  2. Of course, one can also override the ReinforcementLearningBase (RLBase) API methods directly for a greater degree of customization. For example, the default implementation of the RLBase.reset! method for an environment is appropriately randomized (like the goal position and agent start position in the EmptyRoom environment). In case one needs some other behaviour, one can do so by simply overriding this particular method, and reuse the rest of the behaviour as it is (like the effects of actions in this environment).
  3. Most environments offer multiple state representations. One can modify the RLBase.StateStyle(env) method to choose to partially observe (RLBase.StateStyle(env) returns RLBase.Observation{Any}()) or fully observe (RLBase.StateStyle(env) returns RLBase.InternalState{Any}()) the current state of an environment. During rendering, some environments display a gray shaded area surrounding the agent's character. The shaded area corresponds to the region of the grid that is observed via RLBase.state(env) in partially observable settings (when RLBase.StateStyle(env) is set to return RLBase.Observation{Any}()). In the case of fully observable environments (RLBase.StateStyle(env) returns RLBase.InternalState{Any}()), the entire grid is returned as part of RLBase.state(env). For Directed environments, the direction of the agent is not encoded inside the grid. So when fully observing the environment using RLBase.InternalState{Any}(), RLBase.state(env) would return the direction of the agent separately.

Here is the EmptyRoomUndirected environment with RLBase.Observation{Any}():

Here is the EmptyRoomUndirected environment with RLBase.InternalState{Any}():

For more details, it is highly recommended that you take a look at the source code of the particular environment that you are working with.

Rendering

GridWorlds.jl offers two modes of rendering:

  1. Textual Rendering

    This mode can be used directly in the terminal to render an environment. In this mode, we can display only one character per tile. If multiple objects are present in the same tile, then the object with the least index in the objects field of world is chosen. For example, if world.objects is (GridWorlds.Agent(), GridWorlds.Wall(), GridWorlds.Goal()) and if both Agent and Goal are present on a tile, then the character for Agent will be rendered for that particular tile.

    Here is an example of a textual rendering of the SequentialRoomsDirected environment:

  2. Graphical Rendering

    If available, one can optionally use Makie.jl in order to render an environment graphically. It is also possible to play with an environment interactively, and save animations of the same. See the examples given below in List of Environments.

List of Environments

  1. EmptyRoomDirected

  2. EmptyRoomUndirected

  3. GridRoomsDirected

  4. GridRoomsUndirected

  5. SequentialRoomsDirected

  6. SequentialRoomsUndirected

  7. MazeDirected

  8. MazeUndirected

  9. GoToTargetDirected

  10. GoToTargetUndirected

  11. DoorKeyDirected

  12. DoorKeyUndirected

  13. CollectGemsDirected

  14. CollectGemsUndirected

  15. DynamicObstaclesDirected

  16. DynamicObstaclesUndirected

  17. SokobanDirected

  18. SokobanUndirected

  19. Snake

  20. Catcher

  21. TransportDirected

  22. TransportUndirected

Used By Packages

No packages found.