Backed by Y CombinatorA curriculum for reasoning over time
Each of our environments has easier levels and a hard level. The best models clear the easier levels and still fail the hard one. That gradient makes an environment useful for training, not only for evaluation.
In this article
An eval measures. A curriculum teaches.
An evaluation needs tasks that separate models. Training needs more than that. If no model ever earns a reward, there is nothing to learn from. If every model earns the full reward, there is nothing left to learn. Reinforcement learning works best on tasks at the edge of what a model can do.
The idea is older than language models. People learn faster when the examples start easy and grow harder, and so do neural networks: a curriculum that orders training from simple to complex can speed up learning and lead to better results. 1 In reinforcement learning the same insight has a sharper form. Methods that keep a pool of tasks and replay the ones with the most to teach, the tasks where the agent is neither sure to succeed nor sure to fail, learn faster and generalise better than methods that sample tasks at random. 2
The same environment can serve both purposes. A team can evaluate a model on the hard level, train it across all the levels, and evaluate it again on the hard level to see what changed. The reward that drives the training and the score that measures the result come from the same grader.
A whole field now studies how to choose the next task for a learning agent. 3 Its first requirement is simple: the tasks must span a range of difficulty, from tasks the agent can already solve to tasks it cannot yet touch.
Easier levels and a hard level
Each environment has easier levels and a hard level, and each level is its own task with its own grader. At the easier levels of our personal agents family, the best model scores 92% or more. At the hard level, the best model averages 32%.
Each level is its own task with its own grader, so a team can train on any mix of levels and measure progress on each one separately.
That gap is the useful part. A model can earn reward early, on the easier levels, and keep earning it as it improves, up to the hard level. The levels share one purpose: each one asks for the same kind of judgement about a world that keeps changing, at its own difficulty.
| Level | Best model | For training |
|---|---|---|
| Easier levels | 92% or more | Reward comes often, so a model can learn from its first runs. |
| Hard level | 32% on average | Most of the reward is still to earn. |
Why reasoning over time needs the gradient
The hard level asks for several things at once. The agent must notice a change inside a long history, decide when the evidence is enough, find the person who holds the missing fact, and act before the moment passes. A model that cannot yet do any of these well almost never earns a reward on the hard level, and a reward it never earns teaches it nothing.
Easier levels give the same model a chance to earn reward while it learns the habits that the hard level needs. Each success is a signal that training can use. As the model gets better, the hard level comes within reach, and it keeps a clear target in front of the model until the end.
Time adds its own difficulty to learning. When a task runs for a long time, the step that earned the reward may come hours before the reward itself, and a model must learn which of its early steps mattered. Rewards that come often, as they do on the easier levels, give a model more signal while it learns which steps matter.
The gradient also helps the team that runs the training. A model that improves on the easier levels and stalls on the hard one shows where its limit is. A model that improves on both shows that the training transfers.
What to look for in training environments
A few questions help a team choose environments for training. Does the best current model still fail the hardest level? If not, the environment has nothing left to teach the models that matter. Do the easier levels give reward often enough for training to start? Does the grader check answers against the records, so that a model cannot earn reward with a shortcut? And can the environment run many copies at once without failures?
Our environments are built to answer yes to each of these questions. The best model still fails the hard level, the easier levels give reward often, the keys are checked against the records, and the environments run in parallel.
Built to run at scale
Training needs many runs at once, without failures. Each level is a Harbor task: the agent and the data in one container, the grader in another. The reference solution replays offline and earns the full score, so a team can check its setup before it spends anything on training. Our stress test ran 64 sandboxes at once, with no failures. Because each level is its own task, a team can spread its runs over many machines and mix the levels in every batch.
Reliability matters more for training than for evaluation. An evaluation that fails once can run again. A training run that loses rollouts to broken environments learns from a biased sample, and the team may never see why.
Models improve
Models that train on our environments improve. We license environments for RL training, and we build new ones to order. Samples are available the same day, so a team can run a model on them before it decides anything.