Using RailEnv#
In this notebook, we will see how to create, interact with and render railway systems with RailEnv the flatland env class.
You can run this document as an interactive notebook in one click:
Setup#
Install Flatland
!pip install -U flatland-rl
Requirement already satisfied: flatland-rl in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (4.1.0)
Requirement already satisfied: attrs in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (25.3.0)
Requirement already satisfied: boto3 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (1.38.8)
Requirement already satisfied: click in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (8.1.8)
Requirement already satisfied: crowdai_api in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.1.22)
Requirement already satisfied: dataclasses in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.6)
Requirement already satisfied: graphviz in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.20.3)
Requirement already satisfied: importlib_resources<2.0.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (1.5.0)
Requirement already satisfied: ipycanvas in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.13.3)
Requirement already satisfied: ipyevents in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (2.0.2)
Requirement already satisfied: ipython in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (8.36.0)
Requirement already satisfied: ipywidgets in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (8.1.6)
Requirement already satisfied: matplotlib in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (3.10.1)
Requirement already satisfied: msgpack_numpy in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.4.8)
Requirement already satisfied: msgpack in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (1.1.0)
Requirement already satisfied: networkx in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (3.4.2)
Requirement already satisfied: numpy<2 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (1.26.4)
Requirement already satisfied: pandas in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (2.2.3)
Requirement already satisfied: Pillow in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (11.2.1)
Requirement already satisfied: pyglet in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (2.1.6)
Requirement already satisfied: pydantic in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (2.11.4)
Requirement already satisfied: PyYAML in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (6.0.2)
Requirement already satisfied: recordtype in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (1.4)
Requirement already satisfied: redis in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (6.0.0)
Requirement already satisfied: seaborn in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.13.2)
Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (65.5.0)
Requirement already satisfied: tqdm in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (4.67.1)
Requirement already satisfied: svgutils in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.3.4)
Requirement already satisfied: timeout_decorator in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from flatland-rl) (0.5.0)
Requirement already satisfied: botocore<1.39.0,>=1.38.8 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from boto3->flatland-rl) (1.38.8)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from boto3->flatland-rl) (1.0.1)
Requirement already satisfied: s3transfer<0.13.0,>=0.12.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from boto3->flatland-rl) (0.12.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from botocore<1.39.0,>=1.38.8->boto3->flatland-rl) (2.9.0.post0)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from botocore<1.39.0,>=1.38.8->boto3->flatland-rl) (2.4.0)
Requirement already satisfied: six>=1.5 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.39.0,>=1.38.8->boto3->flatland-rl) (1.17.0)
Requirement already satisfied: requests>=2.18.4 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from crowdai_api->flatland-rl) (2.32.3)
Requirement already satisfied: python-gitlab>=1.3.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from crowdai_api->flatland-rl) (5.6.0)
Requirement already satisfied: requests-toolbelt>=1.0.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from python-gitlab>=1.3.0->crowdai_api->flatland-rl) (1.0.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from requests>=2.18.4->crowdai_api->flatland-rl) (3.4.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from requests>=2.18.4->crowdai_api->flatland-rl) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from requests>=2.18.4->crowdai_api->flatland-rl) (2025.4.26)
Requirement already satisfied: comm>=0.1.3 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipywidgets->flatland-rl) (0.2.2)
Requirement already satisfied: traitlets>=4.3.1 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipywidgets->flatland-rl) (5.14.3)
Requirement already satisfied: widgetsnbextension~=4.0.14 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipywidgets->flatland-rl) (4.0.14)
Requirement already satisfied: jupyterlab_widgets~=3.0.14 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipywidgets->flatland-rl) (3.0.14)
Requirement already satisfied: decorator in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (5.2.1)
Requirement already satisfied: exceptiongroup in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (1.2.2)
Requirement already satisfied: jedi>=0.16 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (0.19.2)
Requirement already satisfied: matplotlib-inline in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (0.1.7)
Requirement already satisfied: pexpect>4.3 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (4.9.0)
Requirement already satisfied: prompt_toolkit<3.1.0,>=3.0.41 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (3.0.51)
Requirement already satisfied: pygments>=2.4.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (2.19.1)
Requirement already satisfied: stack_data in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (0.6.3)
Requirement already satisfied: typing_extensions>=4.6 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from ipython->flatland-rl) (4.13.2)
Requirement already satisfied: wcwidth in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from prompt_toolkit<3.1.0,>=3.0.41->ipython->flatland-rl) (0.2.13)
Requirement already satisfied: parso<0.9.0,>=0.8.4 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from jedi>=0.16->ipython->flatland-rl) (0.8.4)
Requirement already satisfied: ptyprocess>=0.5 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from pexpect>4.3->ipython->flatland-rl) (0.7.0)
Requirement already satisfied: contourpy>=1.0.1 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from matplotlib->flatland-rl) (1.3.2)
Requirement already satisfied: cycler>=0.10 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from matplotlib->flatland-rl) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from matplotlib->flatland-rl) (4.57.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from matplotlib->flatland-rl) (1.4.8)
Requirement already satisfied: packaging>=20.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from matplotlib->flatland-rl) (25.0)
Requirement already satisfied: pyparsing>=2.3.1 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from matplotlib->flatland-rl) (3.2.3)
Requirement already satisfied: pytz>=2020.1 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from pandas->flatland-rl) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from pandas->flatland-rl) (2025.2)
Requirement already satisfied: annotated-types>=0.6.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from pydantic->flatland-rl) (0.7.0)
Requirement already satisfied: pydantic-core==2.33.2 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from pydantic->flatland-rl) (2.33.2)
Requirement already satisfied: typing-inspection>=0.4.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from pydantic->flatland-rl) (0.4.0)
Requirement already satisfied: async-timeout>=4.0.3 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from redis->flatland-rl) (5.0.1)
Requirement already satisfied: executing>=1.2.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from stack_data->ipython->flatland-rl) (2.2.0)
Requirement already satisfied: asttokens>=2.1.0 in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from stack_data->ipython->flatland-rl) (3.0.0)
Requirement already satisfied: pure-eval in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from stack_data->ipython->flatland-rl) (0.2.3)
Requirement already satisfied: lxml in /opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages (from svgutils->flatland-rl) (5.4.0)
[notice] A new release of pip is available: 25.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
The Environment#
Let’s first go over the main use cases of RailEnv, the Flatland environment.
The basic usage of the RailEnv environment consists in creating a RailEnv object endowed with
a rail generator, that generates new rail networks on each reset,
a line generator, that generates start and end points for each agent on reset,
an observation builder, that provides a suitable observation vector to the agents.
For now, let’s see how we can create rail networks and use them to train agents.
from flatland.envs.rail_env import RailEnv
from flatland.envs.rail_generators import sparse_rail_generator
from flatland.envs.line_generators import sparse_line_generator
from flatland.envs.observations import GlobalObsForRailEnv
rail_generator = sparse_rail_generator(max_num_cities=2)
# Initialize the properties of the environment
random_env = RailEnv(
width=24,
height=24,
number_of_agents=1,
rail_generator=rail_generator,
line_generator=sparse_line_generator(),
obs_builder_object=GlobalObsForRailEnv()
)
# Call reset() to initialize the environment
observation, info = random_env.reset()
/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/flatland/envs/rail_generators.py:321: UserWarning: Could not set all required cities! Created 1/2
warnings.warn(city_warning)
/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/site-packages/flatland/envs/rail_generators.py:217: UserWarning: [WARNING] Changing to Grid mode to place at least 2 cities.
warnings.warn("[WARNING] Changing to Grid mode to place at least 2 cities.")
Visualising the env#
You can use the method RenderTool.render_env() to render the env in realtime.
For the sake of this tutorial we define a helper function for inline visualization in the notebook
import PIL
from flatland.utils.rendertools import RenderTool
from IPython.display import clear_output
def render_env(env,wait=True, env_renderer = None):
if env_renderer is None:
env_renderer = RenderTool(env)
env_renderer.render_env()
image = env_renderer.get_image()
pil_image = PIL.Image.fromarray(image)
clear_output(wait=True)
display(pil_image)
import PIL
from flatland.utils.rendertools import RenderTool
from IPython.display import clear_output
render_env(random_env)
Observations#
The environment provides very complete observations by default. You typically won’t use this object as-is. One of the main objectives of the Flatland challenge is to find suitable observations to solve the task at hand.
By default, the environment provides global observations.
import numpy as np
for agent_handle in random_env.get_agent_handles():
print('Observations for agent {}:'.format(agent_handle))
agent_obs = observation[agent_handle]
print('- Transition map\n{}\n'.format(np.transpose(agent_obs[0], (2, 0, 1))))
print('- Agent position\n{}\n'.format(np.transpose(agent_obs[1], (2, 0, 1))))
print('- Agent target \n{}\n'.format(np.transpose(agent_obs[2], (2, 0, 1))))
Observations for agent 0:
- Transition map
[[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 1. 0. ... 0. 0. 0.]
[0. 1. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]
[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]
[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]
...
[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]
[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]
[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 1. ... 0. 0. 0.]]]
- Agent position
[[[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
...
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]]
[[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
...
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]]
[[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
...
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]]
[[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
...
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]
[-1. -1. -1. ... -1. -1. -1.]]
[[ 0. 0. 0. ... 0. 0. 0.]
[ 0. 0. 0. ... 0. 0. 0.]
[ 0. 0. 0. ... 0. 0. 0.]
...
[ 0. 0. 0. ... 0. 0. 0.]
[ 0. 0. 0. ... 0. 0. 0.]
[ 0. 0. 0. ... 0. 0. 0.]]]
- Agent target
[[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]
[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]]
The Agent#
RailEnv is targeted at multi-agents experiments. For this purpose, it is derived from RLLib’s MultiAgentEnv class. You can read more details about it here.
The environment is run by supplying the step function with a dictionary of actions, whose keys are agents’ handles and the corresponding values are the selected actions. This dictionary is passed to the environment which checks the validity of all actions and update the environment state.
The environment returns an array of new observations, a reward dictionary for all the agents as well as a flags indicating which agents are done. This information can be used to update the policy of your agent and if done[‘__all__’] == True the episode terminates.
Let us implement a simple agent that takes a valid random action at each step.
class RandomController:
def __init__(self, action_size):
self.action_size = action_size
def act(self, observations):
actions = dict()
for agent_handle, observation in enumerate(observations):
action = np.random.randint(self.action_size)
actions.update({agent_handle: action})
return actions
The environment provides a openai gym-like interface.
The env simulation moves forward with the step() method which takes a dictionary of valid actions and returns the following
observations represeting the state of the env and the observation generator
rewards - the score that rates the agent’s performance
status of compeletion of tasks for each agent
additional information regarding the status of the env
In the example below, we use env.get_agent_handles() to enumerate through the handles, and RailEnvActions.to_char to get a symbol representing the agent’s direction: Backward, Forward, Left, Right or Stop.
from flatland.envs.rail_env import RailEnvActions
controller = RandomController(random_env.action_space[0])
observations, info = random_env.reset()
actions = controller.act(observations)
# Perform a single action per agent
for (handle, action) in actions.items():
print('Agent {} will perform action {} ({})'.format(handle, action, RailEnvActions.to_char(action)))
next_obs, all_rewards, dones, info = random_env.step({handle: action})
print('Rewards for each agent: {}'.format(all_rewards))
print('Done for each agent: {}'.format(dones))
print('Misc info: {}'.format(info))
Agent 0 will perform action 0 (B)
Rewards for each agent: {0: 0}
Done for each agent: {0: False, '__all__': False}
Misc info: {'action_required': {0: True}, 'malfunction': {0: 0}, 'speed': {0: 1.0}, 'state': {0: <TrainState.READY_TO_DEPART: 1>}}
def run_episode(env):
env_renderer = RenderTool(env)
controller = RandomController(env.action_space[0])
observations, info = env.reset()
score = 0
actions = dict()
for step in range(50):
actions = controller.act(observations)
next_observations, all_rewards, dones, info = env.step(actions)
for agent_handle in env.get_agent_handles():
score += all_rewards[agent_handle]
, env_renderer=env_rendererrender_env(env)
print('Timestep {}, total score = {}'.format(step, score))
if dones['__all__']:
print('All done!')
return
print("Episode didn't finish after 50 timesteps.")
Run an episode in the random environment#
run_episode(random_env)
Timestep 0, total score = 0
Timestep 1, total score = 0
Timestep 2, total score = 0
Timestep 3, total score = 0
Timestep 4, total score = 0
Timestep 5, total score = 0
Timestep 6, total score = 0
Timestep 7, total score = 0
Timestep 8, total score = 0
Timestep 9, total score = 0
Timestep 10, total score = 0
Timestep 11, total score = 0
Timestep 12, total score = 0
Timestep 13, total score = 0
Timestep 14, total score = 0
Timestep 15, total score = 0
Timestep 16, total score = 0
Timestep 17, total score = 0
Timestep 18, total score = 0
Timestep 19, total score = 0
Timestep 20, total score = 0
Timestep 21, total score = 0
Timestep 22, total score = 0
Timestep 23, total score = 0
Timestep 24, total score = 0
Timestep 25, total score = -31
All done!