Backed by Y CombinatorIndustry
Timestep-Ops. Wind farms, offshore wells, refineries and factories, with months of operating records.
From Start to End. Notes on the chart: Fault, First sign, Warning time.
Scores
The best model averages 43%.
Mean score, with its 95% interval
Version 17, updated 4 November 2026. Read the methodology.
How they work
- Keys checked against the data
- Each answer key is checked against the records.
- Two judges
- Numeric checks need no judge. Two judges from different labs grade the rest.
- Controls
- Each task's reference answer scores 100% under each judge family. Wrong answers, including answers that take a shortcut, score 10% or less.
- Ready to run
- Each system is a Harbor task: the agent and the data in one container, the grader in another. Bring any model.
- Runs in parallel
- Our stress test ran 64 sandboxes at once, with no failures.
- Proof of learning
- Models that train on our environments improve.
Run it
Each sample is a Harbor task. Unpack it and run any model.
# replay the reference solution: it scores 100%
harbor run -p . -a oracle
# run your model
harbor run -p . -a <agent> -m <provider>/<model>Questions
What are these environments, and how do they work?
Each environment is a real system with months of operating records. The agent works through the records with code, finds what happened and why, and submits its answer with the evidence.
Which model scores highest?
Claude Opus 5.5 leads, with a mean score of 43%. The scores above show every model we ran.
Who builds the environments?
Reific's research team builds each environment from real records and checks every answer key against them.
How are models evaluated?
Every model runs in the same harness, with the same tools and budget. Numeric checks need no judge, and two judges from different labs grade the rest against keys checked against the records.
What does the score measure?
The share of the possible reward that a model earns, from 0% to 100%. Each bar is the model's mean over the environments, and the line under it shows its 95% interval.
Can I reproduce the results myself?
Yes. Each environment is a Harbor task with its own grader. Ask for samples, run any model and compare.
Can I license the environments for training?
Yes. We license environments for RL training, and models that train on them improve. We also build new environments to order. Samples are available the same day.