Timing is part of the answer

We are measuring the wrong thing

Every major evaluation of AI agents works the same way. It waits until every piece of data has arrived, then asks its question, then grades the answer. That design tests one skill well: making sense of events once they are over. That skill matters. Investigations, audits and post-mortems depend on it.

But a growing share of the work we want from agents happens while events are still unfolding: watching a machine, managing a schedule, running a process. In that work, the answer has a deadline. A diagnosis that arrives after the moment to act has passed is not a slightly worse answer. It is a wrong one.

An agent that is right too late is still wrong. Static benchmarks cannot see this failure. Worse, the models trained on them learn to commit it.

The skill we are not measuring

Think about the people we trust most in any field: the operator who can tell a machine is about to fail, the engineer who sees trouble coming in the grid load, the assistant who mentions a problem before it becomes one.

What makes them valuable is not that they know more. It is that they know when. They act on partial evidence. They know how much evidence is enough. They know which question to ask and whom to ask. And they know that waiting is itself a decision, often the most expensive one.

That is judgement under time. It is the core of nearly every job that matters. Our benchmarks do not test it at all.

Objection one: “Just make the agent act faster.”

If waiting is the problem, the obvious fix is speed. Lower the latency, check more often, alert sooner.

That answer is wrong, and it is wrong in a way that shows the whole problem.

An agent that acts at every flicker is not vigilant. It is noise. False alarms cost attention, and attention runs out. The first false alarm gets checked. The third gets ignored. And the true alarm, when it finally comes, gets ignored too.

So there are three ways to fail, not one. Too late, and the moment is gone. Too early, and trust is gone. To the wrong person, and both are gone. The skill is not speed. It is the narrow path between them: enough evidence, in time, to the person who can act. No dial labelled “faster” finds that path. Only judgement does.

Objection two: “Timing is an engineering problem, not a model problem.”

Put the model in a loop, the argument goes, poll it every few minutes, and timing takes care of itself.

But a loop only decides how often the model is asked. It does not decide what the model should do when asked. Every time it wakes up, the model faces the same question: is what I see now enough to act on, or should I wait? Is this a pattern or a blip? Who needs to know?

No scheduler answers those questions. They are reasoning. And a model trained only on finished questions has never had to answer them.

Objection three: “Static benchmarks are cheap, reproducible, and comparable.”

They are. That is their real value, and it is why they have lasted.

But it does not matter how cheaply you can repeat a measurement if it measures the wrong thing. Being able to repeat a test does not make it valid. And there is no trade-off to accept here: an environment that releases its data on a clock, the same way every time, is just as repeatable as a static test. The world can be replayed. Most evaluations simply choose not to replay it.

Why it gets worse, not better

This is not a gap that scale will close on its own. It is a gap that training widens.

A reward that checks only the final answer sends a clear signal: gather everything, commit to nothing, speak at the end. Every training run built on that signal pushes models further toward waiting. The industry is not failing to teach timing. It is teaching the opposite.

Reinforcement learning has long had a name for the deeper difficulty: credit assignment. When the reward arrives long after the action that earned it, the learner has to work out which earlier choices mattered. ¹ In tasks where data keeps arriving, that is the whole game: was the decision on day four what saved the outcome on day eleven? A static benchmark never asks.

The evidence

We built environments to test this. Data arrives on a clock. People reply after a delay. The agent knows only what has reached it so far, and the score depends on what it says, to whom, and when.

On easier tasks, the best frontier model scores 92% or more.

On hard tasks, it scores 32%.

The models are not short of knowledge. They are short of timing.

What follows

If timing is part of the answer, three things follow.

Evaluations must let time run. Tasks should release data as the world would, and keep separate when something happened, when the agent could have known, and when the outcome became clear.

Rewards must count the moment. Not just whether the agent was right, but whether it was right while being right still mattered.

Progress must be measured honestly. A model that scores near-perfectly on frozen tasks and collapses on live ones has not mastered the task. It has mastered the test.

Agents worth trusting in homes, plants, hospitals and power grids will not come from grading them on how well they explain what already happened. They will come from grading them the way the world grades all of us:

Not only on whether we were right.

On whether we were right in time.