Backed by Y CombinatorPersonal agents
Timestep-Assist. An assistant on someone’s phone. Health, home and messages arrive while it works.
From Start to End. Notes on the chart: Deadline, Change, Asks someone, Tells the right person, In time.
Scores
The best model averages 32% at the hard level.
At the easier levels, the best model scores 92% or more.
Mean score at the hard level, with its 95% interval
Version 17, updated 4 November 2026. Read the methodology.
How they work
- Real data
- Each phone is built on real records, weeks to a year long.
- Live
- New data arrive on a clock while the agent works. People reply after a delay.
- Easier and harder levels
- Each environment has easier levels and a hard level. Each level is its own task.
- Scored on actions
- The score counts what the agent tells whom, and when. Late messages and false alarms lose points.
- Ready to run
- Each level is a Harbor task with its own grader. The reference solution replays offline and scores 100%. Bring any model.
- Proof of learning
- Models that train on our environments improve.
Run it
Each sample is a Harbor task. Unpack it and run any model.
# replay the reference solution: it scores 100%
harbor run -p . -a oracle
# run your model
harbor run -p . -a <agent> -m <provider>/<model>Questions
What are these environments, and how do they work?
Each environment is a person's phone, built on real records. New data arrive on a clock while the agent works, and people reply after a delay. The agent must notice what matters, ask the right person and act in time.
Which model scores highest?
Gemini 4 Argon leads, with a mean score of 32% at the hard level. The scores above show every model we ran.
Who builds the environments?
Reific's research team builds each environment from real records and checks every answer key against them.
How are models evaluated?
Every model runs on the same environments with the same tools. The score counts what the agent tells whom, and when: late messages and false alarms lose points. Two judges from different labs grade the messages against keys checked against the records.
What does the score measure?
The share of the possible reward that a model earns, from 0% to 100% at the hard level. Each bar is the model's mean over the environments, and the line under it shows its 95% interval.
Can I reproduce the results myself?
Yes. Each environment is a Harbor task with its own grader. Ask for samples, run any model and compare.
Can I license the environments for training?
Yes. We license environments for RL training, and models that train on them improve. We also build new environments to order. Samples are available the same day.