Flatland Data and Simulation Model#
Overview#
The following digram shows the RL control loop,
obs, rewards, dones, info = env.step(action_dict)
showing the flow of inputs and outputs of step:
flowchart LR
subgraph PolicyRunner
end
subgraph RailEnv
agent((agent))
Rewards
ObservationBuilder
EffectsGenerator
TransitionMap
ResourceMap
end
subgraph Policy
agent_policy((agent))
end
RailEnv -->|observations| Policy
Policy -->|actions| RailEnv
agent o--o agent_policy
RailEnv -->|info| PolicyRunner
RailEnv -->|rewards| PolicyRunner
Adapted from RLlib.
Glossary#
Flatland Environment#
Concept |
Description |
|---|---|
Agent |
Each train is a single agent in Flatland. |
(MA)RL Loop |
(Multi-Agent) Reinforcement Learning Loop is a control loop: a policy receives observations from the env and outputs actions. See above. |
Policy |
Recevies an observation for each agent and outputs an action for each agent. The policy may consist of a single sub-policy applied to each agent unilaterally or it may have different sub-policies for each agent individually or even consider all observations jointly. |
Policy Runner |
Takes an env and a policy and runs the MARL loop for evaluation by calling |
Callbacks |
Callbacks at the start and end of an episode ( |
Trajectory |
The output of a policy run: actions, positions, rewards. |
Actions |
An action for each agent. |
Rewards |
A reward (number or dict) for each agent. Also used for the component building the rewards during |
Effects Generator |
Called by the env at the start of an episode ( |
Observation Builder |
Called by env to build the observation from the env’s state. |
Configuration |
Each agent is in a configuration, e.g. its position on the grid or in the graph. Updated by env during a |
Transition Map |
Defines the topology the agents live in, i.e. the configurations and transitions between configuration as well as actions required for the transition. |
Resource Map |
Defines which resources are used by the topology elements, e.g. rails can be allocated only mutually exclusively. Currently, each topology element requires exactly one resource. |
classDiagram
namespace Core {
class Environment {
<<interface>>
}
class TransitionMap {
<<interface>>
}
class ResourceMap {
<<interface>>
}
class GridTransitionMap {
}
}
namespace Rail {
class AbstractRailEnv {
}
class RailEnv {
}
class GraphTransitionMap {
}
}
AbstractRailEnv --|> Environment
RailEnv --|> AbstractRailEnv
GridTransitionMap --|> TransitionMap
GraphTransitionMap --|> TransitionMap
AbstractRailEnv --> TransitionMap
AbstractRailEnv --> ResourceMap
Flatland Transition Maps#
Concept |
Grid |
Graph |
|---|---|---|
Coordinate |
Row, column, direction how cells can be entered: |
A node ( |
Transition Map |
For each coordinate, which are possible successor coordinates. |
A pair of nodes defines a directed edge. |
Configuration |
The coordinate |
A node. |
Resource Map |
Each |
Each node is mapped to a mutually exclusively allocatable resource ( |
In Flatland 4.3.0, configurations are coordinates: ((r,c),d) in the grid and nodes in the graph.
Flatland Rail#
Concept |
Description |
|---|---|
Malfunction Generator |
An instance of effects generator, changing agent’s state. |
Rail (Generator) |
Outputs a transition map and stopping points, and, optionally, stations and links. |
Line (Generator) |
Outputs a sequence of stops, which are sets of configurations. The first is a singleton called initial configuration; the last is called target; the others are called intermediate stops. |
Timetable (Generator) |
Adds a time window (earliest, latest) for each stop. Earliest of the initial configuration is enforced by the simulation (putting the agent’s state to ready to depart in the env); the remaining are used in the rewards for penalizing late arrival or early departure. |
Station |
Collection of stopping points and gates. |
City |
In sparse rail generator, internally used alias for station. |
Stopping Point |
“Bahnhofsmitte” configurations where agents can stop. |
Gate |
A collection of pins. |
Link |
Connects two gates. Has fibres. Not all pairs of pins in the gate need be connected. |
Fibre |
A sequence of transitions linking two pins of a link. |
Pin |
A single configuration attached to a gate. |
Chain |
A sequence of links terminating/starting at the same station at different reachable gates. |