Frontier evals for LLM reasoning over time series

Simple questions are close to solved

Ask a frontier model with a Python sandbox when a sensor first stayed above a limit, which unit changed most between two weeks, or whether a reading is missing or zero, and it will usually get it right.

That is real progress, and it changes what an evaluation has to do. A benchmark built from single-step questions now measures arithmetic and tool use. It cannot separate one frontier model from the next, and it says little about whether a model can carry out an investigation that an engineer would trust. Anthropic's guidance on evaluating agents stresses realistic tasks and clear, repeatable grading. ¹ Single-step questions meet the second standard and no longer meet the first.

Time series add their own difficulty. Public question-answering sets for operational data, such as ARFBench for software incident response, already ask questions about when something happened and how long it lasted. ² In our experience the difficulty grows quickly as the history gets longer and the question gets more specific.

Where frontier models still fail

The hard part starts when the answer depends on reasoning across a long, real history rather than a single calculation. A model has to decide which windows matter, keep track of what was known at the moment a decision was made, and resist the explanation a quick look suggests. Each of those steps is easy to describe and easy to skip.

• Long histories, where the relevant minutes sit inside weeks of ordinary data and the model must choose where to look.
• Decision cutoffs, where a record that arrived later must not leak into an earlier judgement.
• Questions with a trap: a pattern that looks like the answer and is not, such as a daily cycle mistaken for a change.
• Multi-step investigations, where each step depends on getting the last one right and an early slip compounds.

Leakage is the quiet failure

Historical cases are attractive because the outcome is already known. They also make it easy to grade a model on information it could not have had. A fault recorded at 09:12 may only reach the investigation system at 09:24; an evaluation of a decision taken at 09:20 must exclude it. Feature stores solve the same problem with point-in-time joins, which attach to each row only the values that were available at that row's timestamp. ³

Evaluations of time-series reasoning need the same discipline. Every question should carry its decision time, every file should be cut at that time, and the grader should check that the answer does not depend on anything later. Forecasting research has long used rolling-origin evaluation for the same reason: a method is scored only on data that lay in its future. ⁴

What a useful eval needs

Every question needs an answer that can be checked against its data, not graded by opinion. The question has to be fair: answerable from the information available at the stated time, and hard for a reason that matters in practice rather than a trick of wording. If a careful analyst with the same files could not reach the answer, the question is broken, not the model.

Scoring should separate what a model observed from what it concluded, so a correct reading with a wrong diagnosis is not counted as a success, and a cautious answer that names its uncertainty is not punished as a miss. Teams running agents in production build infrastructure to replay real incidents and evaluate the agent's behaviour on them, not only on synthetic prompts. ⁵ The grading should look at the evidence and the reasoning, not only the final string.

The questions also have to come from histories that behave like real operations, with gaps, noise, regime changes, delayed records and events. Clean synthetic series are useful for unit tests of a method. They are not where frontier reasoning breaks.

From evals to environments

An evaluation asks its question once. Our RL environments go one step further: the history keeps arriving while the agent works. New records reach it on a clock, people reply after a delay, and the agent must decide when the evidence is enough to act. The rules above still apply. Every answer key is checked against the records, nothing that arrives later may decide an earlier judgement, and the histories are real, with their gaps, noise and late corrections.

That turns a question into a task with a reward. On industrial and scientific systems, the agent works through months of records with code and submits its answer with the evidence. Numeric checks need no judge, and two judges from different labs grade the rest against keys checked against the records. On a person's phone, the score counts what the agent tells whom, and when: late messages and false alarms lose points.

The results show the gap that single questions hide. On our leaderboard, no frontier model earns half of the possible reward in any family. Our personal agents environments also come with easier levels and a hard level, so the same environment can measure a model and train one. Each environment is a Harbor task with its own grader, and the reference answer for every task earns the full score, so anyone can run a model on it and compare.

Work with us

We build evaluations and training environments for LLM reasoning over time series, from operational histories, with answers verified against the data and decision times enforced. They can be scoped to a domain, a data shape or a capability you want to measure.

For frontier evals of LLM reasoning over time series data, reach out to us.