Backed by Y CombinatorWhat counts as a correct answer?
An operational explanation must survive checks on the readings, the conclusion and the proposed action. Here is how to evaluate those separately, including when an agent should say it does not know.
In this article
Begin with a claim someone can check
Consider an illustrative machine investigation. Vibration increases, then temperature rises. An agent describes the sequence accurately and concludes that a bearing has failed. It recommends replacing the bearing. The paragraph is clear, uses the right engineering vocabulary and cites a chart. How much of it should pass an evaluation?
The measurements may be correct while the diagnosis remains unconfirmed. The proposed repair requires another judgment. A single score for whether the paragraph sounds like an engineer’s answer conceals these differences. It also makes improvement difficult: the team cannot tell whether to fix the data selection, the interpretation or the recommendation.
For an operational task, we would begin by agreeing what the answer must establish. Identify the machine, the decision and the information available to the person making it. Then evaluate each consequential claim against that contract. The example here illustrates this method; it is not a reported diagnosis or a Reific performance result.
Separate the parts of a useful answer
We propose five checks: observation, interpretation, action, uncertainty and review effort. Observation covers the entity, interval, units, values and ordering. Interpretation asks whether the proposed condition is established, merely consistent with the readings or contradicted by another source. Action asks whether the next step follows from the evidence and the permitted task. Uncertainty records what remains unresolved. Review effort measures what a person must still verify or correct.
Each check needs its own evidence. A chart showing a vibration increase supports that observation. It does not independently verify a bearing defect. A reference explanation containing the same technical terms cannot settle the question either: two answers can be linguistically similar while naming different components or reversing the order of events.
Anthropic’s evaluation guidance distinguishes code, model and human graders. 1 Applied here, arithmetic and source references can often be checked programmatically; a domain expert should establish the interpretation and action criteria. An LLM judge can help apply a rubric, but agreement with that judge is not a substitute for checking what happened.
| Claim in the answer | Evidence needed | Review result |
|---|---|---|
| Vibration increased | Source values, units, comparison period | Check the observation |
| A bearing is failing | Corroborating inspection or diagnostic evidence | Cause remains unconfirmed |
| Inspect the assembly | Proportionate next check and responsible owner | Judge the action separately |
| The cause is unresolved | Known gaps and a useful next question | Credit justified uncertainty |
A recommendation needs its own evidence
In the machine example, a reviewer might accept both signal changes and reject the bearing diagnosis. That need not make the answer useless. It could still justify a narrower next step: compare operating conditions, inspect relevant maintenance history or request the measurement that distinguishes competing explanations. The task owner should decide which of those steps is appropriate before the evaluation begins.
Grade recommendations at the level of authority the agent actually has. An assistant asked to prepare an investigation should not receive extra credit for recommending an intervention it cannot justify. Conversely, an evaluation should not require a definitive cause when the available evidence can only support referral to an engineer.
Keep consequential errors visible. Correct units and polished writing should not average away a recommendation aimed at the wrong asset. Define which mistakes invalidate the answer, then use the remaining checks to explain why it failed and what useful work it nevertheless completed.
Abstention has two failure modes
An agent can fail by asserting too much. It can also fail by declining a question the supplied evidence answers. If the task asks which signal changed first, refusing because the root cause is unknown is an unnecessary abstention. If it asks which component must be replaced, the same missing cause may be decisive.
This is related to the risk–coverage tradeoff studied in selective classification: rejecting some cases changes both the error rate on answered cases and the fraction of cases answered. 2 An operational evaluation should therefore report accepted answers, unsupported conclusions, justified abstentions and avoidable abstentions separately. A lower error rate achieved by answering almost nothing may leave the whole workload with the engineer.
Make an abstention useful enough to review. It should identify the unresolved claim, the missing evidence and, where possible, a specific next check. Include cases where the record is sufficient as well as cases with missing, conflicting or irrelevant context. Otherwise the evaluation rewards one habitual response.
Preserve what was knowable at the time
Historical cases are convenient because a later outcome may already be known. They also create an easy route for leakage. In a second illustrative example, a fault event occurs at 09:12, but its record reaches the investigation system at 09:24. An evaluation of a 09:20 decision must exclude that record. Filtering only on the event timestamp would incorrectly admit it.
Forecast evaluation has an established temporal discipline: the forecast must be constructed without future observations. 3 For an operational investigation, we extend that discipline to availability time, document revisions and later corrections. A repair finding recorded the next day can help grade the diagnosis; it cannot be supplied as evidence the agent supposedly had the previous morning.
Retain the input snapshot, the decision cutoff and the basis for every outcome label. Where arrival times or historical document versions are unavailable, state that the case cannot establish performance under the original information constraints. A retrospective explanation can still be useful, but it answers a different question.
Only information already available enters the task.
- Fault eventEvent time does not tell us when its record arrived.
- Delayed record arrivesExcluded from the 09:20 input.
- Repair findingMay establish the outcome for scoring.
Compare against a capable baseline
The comparison should give an ordinary agent appropriate query and code tools. For a question about a maximum, a join or a before-and-after interval, those tools may solve the task directly. Comparing an evidence layer only against an agent forced to read a large raw table would leave its practical contribution unresolved.
Hold the questions, eligible information, tool permissions and grading criteria fixed. Record the model and configuration used by each approach. Separate cases used to refine the system from the cases used to report its performance, and keep related windows from the same incident together when assigning those groups.
Research benchmarks can help isolate specific capabilities. ARFBench, for example, evaluates questions about time-series anomalies in software incident data. 4 Our application of that distinction is straightforward: a component score should be described at the task it measures. It does not, by itself, establish that an agent can choose an appropriate intervention or reduce an engineer’s workload.
Report the work left to the reviewer
An evaluation report should let someone reconstruct a result: attempted cases, accepted cases, failure types, abstentions, source checks and expert corrections. Show the difficult examples alongside the successful ones. If reviewers disagree, retain that disagreement and resolve the rubric before interpreting small score differences.
This is the standard we want an evaluation of Reific to meet. The output should help a person reach a justified conclusion with less repeated investigation. Establishing that requires both a correct answer and a record of the work still needed to trust it.