Evaluation Metrics#
The ECML 2026 challenge is the newest competition around the Flatland environment.
In this edition, we are encouraging participants to develop innovative solutions that leverage reinforcement learning. The scenario setup and the evaluation metrics are designed accordingly. However, we are still open for other solutions as well, e.g. operations research, and encourage participants to benchmark their state-of-the art algorithms
⚖ Evaluation metrics#
The competition was evaluated with flatland-rl 4.2.6, as pinned in the starterkit. flatland-rl 4.3.0 changed the reward semantics described below: no collision penalty for stops issued by the controller (#452), intermediate stops served at any halting cell of the station (#453) and the normalization (#457). Scores computed with 4.3.0 can therefore differ from the published ones.
Normalized Episode Rewards#
The primary metrics uses the normalized return from your agents - the higher the better.
What is the normalized return?
The returns are the sum of Flatland’s default rewards your agents accumulate during each episode as described in rewards.md
To normalize these return, we scale them so that they stays in the range \([0.0, 1.0]\). To guarantee this, the maximum penalty per agent can be at most
max_episode_steps. This normalized rewards allows to compare results between environments of different dimensions and different number of agents.
In code:
normalized_reward = sum([max(cumulative_rewards[agent.handle], - self.env.max_episode_steps) for agent in agents]) / (
self.env.max_episode_steps * self.env.get_num_agents()) + 1
The total reward is then
where \(m_{l,s}\) and \(a_{l,s}\) are the maximum episode steps and number of agents for scenario \(s\) of level \(l\) respectively. The (unnormalized) total reward for agent \(a\) in level \(l\) and scenario \(s\) is the sum over the different categories of rewards/penalties
For comparing the contributions of different reward categories to the scenario score we look at the normalized category score
for level \(l\), scenario \(s\) and category \(c\). Note that this score allows only for a comparison of the composition of the reward and not of total rewards, since
regardless of the total reward.
The episodes finish when all the trains have reached their target, or when the maximum number of time steps is reached. Therefore:
The minimum possible value (i.e. worst possible) is 0.0, which occurs if every agent accumulates a total penalty of at least
max_episode_steps(penalties beyondmax_episode_stepsper agent are not counted, see the code above). Agents not reaching their goal do not necessarily lead to the minimum: e.g. the penalty for an agent that never departs is proportional to its travel time on the shortest path, which is usually less thanmax_episode_steps.The maximum possible value (i.e. best possible) is 1.0, which would occur if all the agents would reach their targets and intermediate stops on time, i.e. not receive any penalty.
Submission Score#
The submission score is the sum of the normalized scenario rewards.
Evaluation is stopped when a submission does not reach the threshold of 25% completed agents within a level (5 scenarios).
Factors in reward function#
The factors for the reward function in this competition are:
factor |
value |
|---|---|
journey not started (cancellation factor) |
5 |
cancellation time buffer |
0 |
delay at target |
1 |
target not reached minimum penalty |
100 |
intermediate stop not served |
50 |
intermediate late arrival |
0.5 |
intermediate early departure |
0.5 |
collision (forced stop, incl. invalid action) |
250 |
This configuration is implemented using --rewards flatland.envs.rewards.ECML2026Rewards.
⛽ Time and Resource limits#
The agents have to act within time limits:
You are allowed up to 30 minutes per scenario.
The full evaluation must finish in 5 hours.
The agents are evaluated in a container with resource limits
4 CPU cores
15 GiB of main memory.
We do not provide GPUs.
Detailed overview over resource limits#
Limit[1] |
Value |
Submission outcome |
Details |
|---|---|---|---|
|
|
Not created |
Error in frontend as error |
|
|
Failure |
per scenario, submission pod should be listed by now, i.e. pulling has started by now. |
|
|
Failure |
per scenario, submission pod should have reached running state by now, i.e. pulling should be done by now . |
|
|
Success with termination cause |
per scenario, excluding technical overhead for starting pods and running offline trajectory evaluation; results do not include the overlong scenario |
|
|
Failure/cleanup |
per scenario, including technical overhead for starting pods for submission |
|
|
Failure/cleanup |
all scenarios, including technical overhead for starting pods for orchestration and evaluation |
|
|
Success with termination cause |
all scenarios, excluding technical overhead for starting pods and running offline trajectory evaluation; results do not include the overlong scenario |
|
|
Success with termination cause |
|
|
|
Failure |
resource limits for the orchestration job pod, one per submission, which runs the submission’s scenario pods and evaluates their trajectories |
|
|
Failure |
resource limits for the submission pod, one per scenario |
To move from one level to the next, a submission has to achieve a mean success rate of 25% over the 5 scenarios of that level, i.e., at least 25% of the trains have to reach their destination. Constraints to ensure a level playing field:
30 minutes per scenario limit
5 hours per submission limit
2 daily submissions limit
computation resources for all submissions: 4 CPU cores with 15 GiB of memory
Failing to satisfy the success rate or time constraint requirement results in the termination of the submission, leaving the summed score as up to this point.
gantt
dateFormat HH:mm
axisFormat %H:%M
title Per-scenario pod lifecycle — nested time limits
section Waiting phase (from job creation, t=0)
wait_for_pod_to_start_limit — pod not yet listed (max 2m): crit, start1, 00:00, 2m
wait_for_pod_to_run_limit — Pending/Unknown incl. image pull (max 30m, measured from t=0): active, run1, 00:00, 30m
section Running phase
running_time_limit — pod Running, per scenario (max 30m, starts once Running): done, running1, 00:30, 30m
section Hard k8s ceiling (per scenario pod)
active_deadline_seconds — on pod spec, pull+run together (1h = wait_for_pod_to_run_limit + running_time_limit, no slack): crit, adl1, 00:00, 60m
gantt
dateFormat HH:mm
axisFormat %H:%M
title Submission-level — across ALL scenarios
section Orchestration job (whole submission, hard k8s ceiling)
orchestration_job_active_deadline_seconds — ALL scenarios + technical overhead (8h): crit, orch, 00:00, 480m
section Scenario pods (run sequentially, one at a time)
Scenario 1 pod (start+pull+run+eval): done, s1, 00:00, 15m
Scenario 2 pod: done, s2, after s1, 45m
Scenario 3 pod: active, s3, after s2, 30m
"… remaining scenarios (up to 1h each, per active_deadline_seconds) …": crit, sN, after s3, 90m
section Cumulative running-time budget (Running-phase seconds only, excl. pull/eval overhead — NOT wall clock)
total_running_time_limit exhausted — per eval.md (5h): milestone, tot_doc, 05:00, 0m
📪 Daily Submission Limits and Submission Closure.#
You can submit up to 2 times per day.