Backed by Y CombinatorWhen does an evidence layer earn its place?
A fair pilot compares direct tools, the platform you already use and an agent with Reific on the same work. Define the comparison before spending engineering time on integration.
In this article
Start with the investigation that keeps recurring
A team evaluating an evidence layer is already paying for an investigation process. Someone chooses a time window, finds the relevant entities, queries their histories, compares events and checks the conclusion. Some of that work may be automated. The useful starting point is to identify which part still requires repeated engineering effort and whether it matters enough to change.
Take an illustrative task: at 09:20, which machines need an engineer’s review, and what observations support each referral? An answer needs a bounded set of candidates, source readings and a defensible reason for inclusion. The task is more specific than asking an agent to find insights. It also gives the evaluator a way to recognize a good answer from any implementation.
A pilot should test the proposed improvement to that job. It should establish whether the additional component changes answer quality, review effort or cost enough to justify its continuing maintenance. A successful demonstration alone does not answer that question.
Give each approach a fair chance
Begin with direct tools. An agent can query a database, calculate changes in code and retrieve specific events. Prometheus, for example, exposes expression queries over a specified time range and resolution through its HTTP API. 1 If an adequately scoped query supplies everything needed, adding another service may bring little benefit. That is a useful result to discover early.
The second approach is the platform already responsible for the workflow, including its current investigation features and the team’s own extensions. Established platforms can be substantial alternatives. Datadog describes both an incident-investigation agent and infrastructure for replaying and evaluating its behavior. 2 Evaluate the actual configured system your engineers would use, with the access and context that make it effective.
The third approach is the same agent workflow with Reific supplying evidence. This is a comparison to run, not an assumed ranking. Its case for adoption is whatever additional value the evaluation establishes after preparation, retrieval, interpretation and review are all counted. Keep the direct-tool and platform approaches in the comparison even when they answer some cases easily; those cases are part of the workload too.
Agent with suitable query and code access
The workflow the team already relies on
Agent with the proposed evidence layer
Write the acceptance contract before integration
The person responsible for accepting the result should define the question and rubric. For the machine-review example, that means agreeing which observations warrant referral, which sources support them, what counts as a missed case and when uncertainty is acceptable. If the historical record cannot support a reliable answer, mark that case accordingly. Do not quietly turn uncertain maintenance notes into confirmed labels.
Choose cases that represent the intended workload: routine questions, relevant changes, apparent changes explained by operating conditions and incomplete records. Preserve the information available at the original decision time. Keep a development set for fixing integration and a separate assessment set for the comparison. Group related cases so nearly identical windows from one incident do not appear on both sides.
Agree the meaningful improvement before opening assessment results. That might be fewer consequential errors without more missed cases, or less reviewer time at an unchanged acceptance standard. Define the allowed latency, spending and preparation effort too. The thresholds should come from the job’s economics and error costs, with an explicit owner who can decide what the result permits next.
Make the comparison paired and inspectable
Run each approach on every eligible assessment case. Use the same question, decision cutoff, underlying information and outcome rubric. For the two agent configurations, hold the downstream model, prompt and tool permissions constant wherever possible. The changed input is the evidence treatment being evaluated. Let the direct-tool agent select and aggregate data rather than forcing it to consume a full export.
An existing platform may use models or internal context you cannot hold constant. Record that difference. Its result is a comparison of complete workflows; it cannot isolate the effect of one evidence component. Both comparisons are useful, provided the report says which question each answers.
Fix the number of attempts in advance and retain every attempt. Anthropic notes that agent behavior can vary across trials. 3 Report both the paired case results and variation between attempts, without selecting the best answer from one approach and the first from another. Treat repeated trials and related windows as related observations, not additional independent incidents. Have reviewers inspect outputs without implementation names where practical, and preserve the grading rationale.
Account for the work around the model
The cost ledger begins before the first answer. Record the engineering time spent mapping entities, preparing data, configuring permissions and repairing integration problems. During each run, retain requests, tool results, model usage, elapsed time, failures and retries. Include evidence preparation and service charges alongside the agent’s own consumption.
Keep one-time setup separate from recurring effort. A reusable mapping may be a reasonable investment; a mapping rebuilt manually for every case is part of the continuing operating cost. Treat reviewer time the same way. Track how long it takes to verify sources, correct a conclusion and produce an acceptable final answer, rather than timing only the generated response.
Inspect the differences case by case. Fewer model tokens may coexist with slower retrieval. A more accurate first answer may still require more expert review. A cheap workflow may omit hard cases. Report these tradeoffs with the workload denominator intact, then apply the acceptance contract instead of choosing whichever metric makes a preferred system look strongest.
Require a result that can be used again
Before expanding the integration, repeat the agreed procedure on another reserved batch representing the intended work. This checks whether the setup and review process can be reused, within the limits of the sample. It also exposes dependencies a carefully prepared demonstration can hide: who notices a missing source, who updates a changed field and who investigates a failed run.
The decision may be to keep the direct tools, use the existing platform more fully or continue with Reific on a narrower class of tasks. An inconclusive result should identify the missing evidence and whether obtaining it is worth the effort. A promising offline result can justify a bounded trial in the real workflow; it does not yet establish recurring operational value.
For Reific to earn its place, the comparison should show a useful difference that survives those conditions. The team should be able to inspect where the difference came from, understand the remaining failures and estimate the work required to keep it. That is a more durable basis for adoption than the breadth of a product description.