{"schema_version":"2.0","name":"Reific","website_operator":"Reific, Inc.","website_data_controller":"Reific, Inc.","corporate_entities":{"parent":{"name":"Reific, Inc.","legalForm":"Delaware corporation","register":"Delaware Division of Corporations","registrationNumber":"10740924","registeredOffice":"251 Little Falls Drive, Wilmington, Delaware 19808, United States","registeredAgent":"Corporation Service Company"},"subsidiary":{"name":"Reific, UAB","legalForm":"Lithuanian private limited liability company (uždaroji akcinė bendrovė)","register":"Register of Legal Entities of the Republic of Lithuania","registrationNumber":"308114457","registeredOffice":"A. Jakšto g. 6A-22, LT-01105 Vilnius, Lithuania","electronicDelivery":"308114457"},"details":"/company"},"description":"RL environments for personal agents, industry and science, for the frontier labs that train agents. Real records, data that arrive on a clock, and answers checked against what actually happened.","url":"https://reific.co","language":"en","environment":"production","updated":"2026-11-09","environments":[{"title":"Personal agents","benchmark":"Timestep-Assist","url":"/environments/personal-agents","summary":"An assistant on someone’s phone. Health, home and messages arrive while it works."},{"title":"Industry","benchmark":"Timestep-Ops","url":"/environments/industry","summary":"Wind farms, offshore wells, refineries and factories, with months of operating records."},{"title":"Science and compute","benchmark":"Timestep-Lab","url":"/environments/science-and-compute","summary":"Bioreactors, GPU clusters and storage fleets, from the lab to the data centre."}],"leaderboard":{"url":"/leaderboard","methodology":"/environments/methodology","version":"Version 17, updated 4 November 2026","measure":"each model's mean score on each Timestep track, with its 95% interval"},"work_with_reific":{"request_samples":"/#request","talk_to_a_researcher":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=agent_evidence_index","request_a_model":"/leaderboard","data_partners":"/data-partners","careers":"/careers"},"articles":[{"slug":"how-we-protect-peoples-data","title":"How we protect people's data, in layers","url":"/research/how-we-protect-peoples-data","kind":"field-note","date":"2026-11-10","updatedAt":"2026-11-10","author":"Reific","related":["why-we-build-from-real-records","timing-is-part-of-the-answer","frontier-evals-time-series-reasoning"],"listed":true,"publications":[],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=anonymisation_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/how-we-protect-peoples-data/markdown","summary":"Find-and-replace misses names in nicknames, file names and inflected forms, and a record with no names can still point to one person. We protect people's data in eight layers, and models from two labs attack every release before it ships.","question":"How should records about real people be protected before they become training data?","sections":[{"title":"Find-and-replace is a red flag","paragraphs":["When a frontier lab looks at a new data vendor, one of the first questions is how the vendor replaces people's names. \"We run a find-and-replace\" is a red flag. A list of names and a search tool remove the names that appear exactly as listed. Real records write names in many other ways, and the rest of a record can point to a person when no name is left.","The law sets a high bar. Under European law, a person is identifiable if the holder of the data, or anyone else, can identify them by means reasonably likely to be used, and pseudonymised data are still personal data. [1] The European Data Protection Board's draft guidelines, adopted for consultation on 7 July 2026, propose three criteria for anonymous data: no one can isolate a person's record, link it to other records about that person, or draw specific and meaningful inferences about them. [2] The UK regulator asks whether a competent, motivated intruder could re-identify someone with the tools available today, and it counts generative AI tools among them. [3]","We keep the original records, and the table that links each original value to its replacement. For us, the output therefore stays personal data, and we protect it as personal data. In EDPS v SRB (2025), the EU Court of Justice held that pseudonymised data may not be personal data for a recipient with no means reasonably likely to identify the people. For the party that pseudonymised them, they stay personal data, and its duties, such as telling people who will receive their data, are judged from its position when it collects them. [4] So the question that matters is what the people who receive the output could learn. Three kinds of reader count: staff at the recipient, anyone who obtains a copy after a leak, and the users of a model trained on the data. Each of them may hold other data, such as the web, data-broker files or their own users' records."]},{"title":"What find-and-replace misses","paragraphs":["Take the phone export of an invented person: bank lines, messages, calendar events, e-mails, device records, contacts and notes, in the forms that real exports use. Ours holds 197 identifiers. 148 point at a person directly, such as names, numbers and addresses. 49 narrow a person down, such as an employer, a street or a date of birth. It also holds 104 ordinary words that look like names, such as the month May, the verb \"mark\" and the noun \"bill\". A good method keeps those.","A careful find-and-replace, with the full list of known people, organisations and places, with and without accents, and the standard patterns for e-mail addresses, phone numbers and bank and national ID numbers, leaves 76 identifiers. It misses every inflected form of a name (in Lithuanian, \"Tomas\" becomes \"Tomo\" and \"Tomui\"), every nickname, every name inside a file name, and every name that a look-alike letter or an invisible character has changed. It also replaces 11 ordinary words: \"grace period\" becomes a person.","Microsoft Presidio, a widely used open-source tool that finds names with a statistical language model, leaves 95, including nearly every identifier in the upper-case bank lines. It ran with its standard English model and without the list of known people that the other methods had. On court judgments, Presidio found only 46% of the direct identifiers. [5] Our layers leave 1: a pet's name in a password hint.","A record that we write ourselves is the easy case, so a separate model, which never saw our rules, wrote a second one: a Polish nurse who lives in Germany, with messages in Polish, German and English, 221 identifiers and 80 ordinary words that look like names. Find-and-replace leaves 137 identifiers on it, and Presidio 98. Our rules alone leave 97, because Polish grammar and German organisation names are hard for any list. The reviewer brings the count to 12: ages, a child's school class, the name of a child's nursery inside a password hint, two people whom no list knew, and a German social security number inside a file name. The scan catches that number and stops the build. Three other hits are the owner's real city, which the reviewer and the record's own answer key both missed.","Rules are only as good as the languages they know. The reviewer is what carries the method to a new record, and the scan is what makes the reviewer safe to use."],"figure":"anonymisation-test","figureDescription":"Two invented phone exports. Our record: 197 identifiers and 104 ordinary words that look like names; we wrote it. Blind record: 221 identifiers and 80 such words, written by a separate model that never saw our rules. Identifiers left, our record and blind record: find-and-replace 76 and 137; Microsoft Presidio 95 and 98; Reific rules only 14 and 97; Reific, all layers 1 and 12. Ordinary words replaced: find-and-replace 11 and 4; Presidio 4 and 22; rules only 3 and 5; all layers 1 and 4."},{"title":"Two intruders, two records","paragraphs":["We give each output to two frontier models from different labs, Claude Opus 5.5 and Gemini 4 Argon, and tell each to act as a motivated intruder: recover the owner's first and family name, date of birth, employer, street, city and six other facts, with only the output in front of it. From the find-and-replace outputs they recover 8 to 10 of the 12 facts, and from the Presidio outputs 11 or 12. From our rules alone they recover at most 1 on our record, but 7 on the blind record.","From all our layers they recover nothing on our record. On the blind record, Gemini recovers two facts: the owner's date of birth, from the social security number in a file name that our scan had already stopped, and the real name of a child's nursery inside a password hint. Language models infer a person's location, age and income from ordinary text at a fraction of a human's cost, so the attack must use them. [6] Our reviewer is also Claude Opus 5.5, so the Claude intruder can share its blind spots; Gemini is the independent attacker. Because the people are invented, this attack finds leftover text. It cannot test linkage with outside data."],"figure":"intruder-test","figureDescription":"Facts out of 12 that an intruder model recovered from each output, with only the output in front of it. Our record, Claude Opus 5.5 and Gemini 4 Argon: find-and-replace 8 and 10; Presidio 12 and 12; Reific rules only 0 and 1; Reific, all layers 0 and 0. Blind record: find-and-replace 8 and 10; Presidio 11 and 11; rules only 7 and 7; all layers 0 and 2."},{"title":"The record itself is an identifier","paragraphs":["Names are the easy part. An early study estimated that date of birth, sex and postcode single out 87% of Americans, and a later re-analysis put the figure at 61% to 63%. [7] [8] Four points of place and time single out 95% of people in phone location data, and four purchases single out 90% of people in card records. [9] [10] With 15 demographic attributes, 99.98% of Americans can be re-identified. [11]","Time series are no different. In a US national health survey, a machine-learning model matched 94% of adults to their records, from activity-monitor data with the health identifiers removed and basic demographics. [12] Gait, heart signals and typing rhythm all work as biometrics. [13] [14] [15] A record with no name can therefore still be matched to the same person's other records by its rhythm alone, and moving the dates does not change that rhythm.","Formal models do not remove this risk either. k-anonymity requires each record to look like at least k others, and long, detailed records cannot meet that without the loss of most of their detail. [16] Differential privacy gives a mathematical bound, but for one person's record, the noise it needs removes the detail that a task depends on. [17] It fits best at training time, where the lab that trains the model sets it.","So a detailed trace is never anonymous by itself. Its protection comes from the detail that is kept, the length of the window, and who receives it, under what terms."]},{"title":"Our layers","paragraphs":["No single method removes the risk, so we use eight layers. Each layer assumes that the one before it failed, and the release gate refuses a build that skips one."],"points":["Rights before data. Every source has a record of its origin, its licence and the uses it allows.","Read only what the work needs. Our readers open only the tables and fields that are named for each source. For a phone export, that leaves out the location history and the account, settings and billing files. Device names, which often carry the owner's first name, are never stored, exact coordinates are not carried, and the export's own record numbers are dropped.","Map every value, and stop on the unknown. Structured values, such as the payee of a bank line, pass through a reviewed table that maps each original to its replacement. A value without an entry stops the build. Find-and-replace fails open: what it does not know, it lets through. A table fails closed.","Replace consistently, in the same form. One person gets one replacement name everywhere, of the same gender and language. The rules know the inflected forms, diminutives, initials and glued forms of every known name, in each language of the record, and put the same form of the replacement in its place: \"Tomui\" becomes \"Lukui\", not \"Lukas\". No replacement shares a word with a real name, and the people around the owner, such as family, friends and colleagues, get the same protection as the owner.","Move time, and the text with it. Every time in a record moves by an offset that we keep secret, and dates written inside the text move with it. A holiday named in the text becomes a general word, because a \"May bank holiday\" in October tells a reader how far the dates moved. No date of birth comes from the record.","Read free text twice. Rules come first: they undo look-alike letters and invisible characters, find every form of every known name, and replace patterns such as phone and account numbers. Then a reviewer model reads what is left and marks the people that no list knew and the facts that identify only in combination, such as \"the only Lithuanian-speaking hygienist in the city\". The reviewer only marks. The rules write every number, from a reserved range or with a check digit that no real account can have where the format allows, and one fact keeps one value everywhere. In one study, language models anonymised text better than the commercial tools tested. [18]","Scan last, with a key held apart. A final scan reads every file of the build, including field names, comments and compressed pages, after it normalises the text. It cuts the text into runs of one to five words, also reads each word with its digits removed, and compares a keyed hash of each run with the hashes of the original values. The key is kept in a secrets vault, apart from the code, so the list of hashes cannot be reversed by anyone who hashes a list of names. The scan also checks every edit the reviewer proposes, before it applies. A single hit, or a file that the scan cannot read, stops the build.","Intruders from two labs. Before every release, models from two different labs attack the output as motivated intruders. Their guesses are checked against the hashes of the original values, and a recovered value stops the release. A person reads what they found and signs the report."]},{"title":"Each layer catches the last one's mistakes","paragraphs":["The order exists because every layer makes mistakes. Rules once read the Lithuanian name form \"Tomo\" inside the word \"tomorrow\" and changed the word. The reviewer model saw the damaged word and restored it, and the rule changed.","A reviewer that writes its own replacements makes worse mistakes. Told to replace a pseudonymous national insurance number, one wrote back the original, because the original was the standard example number from public documentation. The same reviewer moved one owner's date of birth by a different amount in different messages, and left five birthdays. A record that contradicts itself shows an intruder where the edits are. That is why our reviewer only marks, the rules write every number, and the scan checks every edit. A generative step must never write a value that no rule checks.","Intruders find what the rest misses. One worked out how far the dates had moved: the dates written inside the bank lines had not moved, and a \"May bank holiday\" now fell in October. With that offset it recovered the owner's real birthday. The same model saw an initial, \"Ms I\", in front of a replaced family name, and from it guessed the owner's real first name. Dates in the text now move with the record, holidays become general words, and an initial changes with its name."]},{"title":"What survives","paragraphs":["Protection is worth little if the record no longer teaches anything. In the records above, these survive: amounts, merchants, the order of events and the time between them, the local time of day, weekdays, the mix of languages, typos, emoji and grammatical forms, family relations between names, and who writes to whom. These change: names, places, organisations, numbers, file names, the dates themselves, and rare facts, which become general (\"one of the few staff who speak another language\"). Our layers kept 103 of the 104 ordinary words that look like names in the first record, and 76 of the 80 in the blind record."]},{"title":"What protection cannot promise","paragraphs":["No method makes re-identification impossible. US health guidance asks for a risk that is very small, not zero. [19] A model that trains on a record can repeat parts of it word for word, and larger models repeat more. [20] Removing personal data before training reduces this leakage, but it does not remove it. [21] Synthetic data is not a way out: it either keeps the information that enables an attack or loses the information that makes it useful. [22] A hash does not hide a value with few possible forms. In 2014, New York City released taxi trips with hashed licence numbers, and one person computed every possible hash in less than two minutes. [23]","What a careful vendor can promise is a process: every layer written down, every build scanned, every release attacked by two labs' models and signed by a person, and the remaining risk named rather than denied."]},{"title":"Questions to ask any data vendor","paragraphs":["Data partners trust us with records about real people, and labs train on what we build from them. Both should ask these questions of us and of every other vendor."],"points":["Which fields do you read, and which do you never open?","What happens to a value that your rules do not recognise?","Do your replacements keep the grammatical form of a name, and does one fact keep one value?","Can a generative step write a value that no rule checks?","Who attacks your output before it ships, and with which models?","Who receives the result, under what terms, and what other data about the same people do they hold?"]}],"sources":[{"title":"Regulation (EU) 2016/679 (GDPR), Recital 26 and Article 4(5)","href":"https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng"},{"title":"EDPB — Guidelines 02/2026 on Anonymisation, version 1.0 for public consultation (7 July 2026)","href":"https://www.edpb.europa.eu/system/files/2026-07/edpb_guidelines_202602_anonymisation_v1_en_0.pdf"},{"title":"ICO — Anonymisation: how do we ensure anonymisation is effective? (motivated intruder test)","href":"https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-sharing/anonymisation/how-do-we-ensure-anonymisation-is-effective/"},{"title":"Court of Justice of the EU — Case C-413/23 P, EDPS v SRB, judgment of 4 September 2025","href":"https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62023CJ0413"},{"title":"Pilán et al. — The Text Anonymization Benchmark (TAB) (Computational Linguistics, 2022)","href":"https://arxiv.org/abs/2202.00443"},{"title":"Staab et al. — Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR, 2024)","href":"https://arxiv.org/abs/2310.07298"},{"title":"Sweeney — Simple Demographics Often Identify People Uniquely (Carnegie Mellon University, 2000)","href":"https://dataprivacylab.org/projects/identifiability/paper1.pdf"},{"title":"Golle — Revisiting the Uniqueness of Simple Demographics in the US Population (WPES, 2006)","href":"https://crypto.stanford.edu/~pgolle/papers/census.pdf"},{"title":"de Montjoye et al. — Unique in the Crowd: The privacy bounds of human mobility (Scientific Reports, 2013)","href":"https://www.nature.com/articles/srep01376"},{"title":"de Montjoye et al. — Unique in the shopping mall: On the reidentifiability of credit card metadata (Science, 2015)","href":"https://dspace.mit.edu/handle/1721.1/96321"},{"title":"Rocher, Hendrickx and de Montjoye — Estimating the success of re-identifications in incomplete datasets using generative models (Nature Communications, 2019)","href":"https://www.nature.com/articles/s41467-019-10933-3"},{"title":"Na et al. — Feasibility of Reidentifying Individuals in Large National Physical Activity Data Sets From Which Protected Health Information Has Been Removed With Use of Machine Learning (JAMA Network Open, 2018)","href":"https://pubmed.ncbi.nlm.nih.gov/30646312/"},{"title":"Sprager and Juric — Inertial Sensor-Based Gait Recognition: A Review (Sensors, 2015)","href":"https://www.mdpi.com/1424-8220/15/9/22089"},{"title":"Pinto, Cardoso and Lourenço — Evolution, Current Challenges, and Future Possibilities in ECG Biometrics (IEEE Access, 2018)","href":"https://doi.org/10.1109/ACCESS.2018.2849870"},{"title":"Acien et al. — TypeNet: Deep Learning Keystroke Biometrics (IEEE T-BIOM, 2022)","href":"https://arxiv.org/abs/2101.05570"},{"title":"Narayanan and Shmatikov — Robust De-anonymization of Large Sparse Datasets (IEEE S&P, 2008)","href":"https://arxiv.org/abs/cs/0610105"},{"title":"NIST SP 800-226 — Guidelines for Evaluating Differential Privacy Guarantees (2025)","href":"https://csrc.nist.gov/pubs/sp/800/226/final"},{"title":"Staab et al. — Language Models are Advanced Anonymizers (ICLR, 2025)","href":"https://arxiv.org/abs/2402.13846"},{"title":"HHS — Guidance on De-identification of Protected Health Information under the HIPAA Privacy Rule","href":"https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html"},{"title":"Carlini et al. — Extracting Training Data from Large Language Models (USENIX Security, 2021)","href":"https://arxiv.org/abs/2012.07805"},{"title":"Lukas et al. — Analyzing Leakage of Personally Identifiable Information in Language Models (IEEE S&P, 2023)","href":"https://arxiv.org/abs/2302.00539"},{"title":"Stadler, Oprisanu and Troncoso — Synthetic Data: Anonymisation Groundhog Day (USENIX Security, 2022)","href":"https://arxiv.org/abs/2011.07018"},{"title":"Pandurangan — On Taxis and Rainbows (2014)","href":"https://tech.vijayp.ca/of-taxis-and-rainbows-f6bc289679a1"}]},{"slug":"timing-is-part-of-the-answer","title":"Timing is part of the answer","url":"/research/timing-is-part-of-the-answer","kind":"field-note","date":"2026-11-02","updatedAt":"2026-11-10","author":"Reific","related":["frontier-models-reasoning-over-time","a-curriculum-for-reasoning-over-time","why-we-build-from-real-records"],"listed":true,"publications":[],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=timing_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/timing-is-part-of-the-answer/markdown","summary":"Static benchmarks wait for all the data before grading an agent. For agents that act while events unfold, that measures the wrong skill, and training on it teaches models to wait.","question":"Why should an RL environment score when an agent acts, not only what it concludes?","sections":[{"title":"We are measuring the wrong thing","paragraphs":["Every major evaluation of AI agents works the same way. It waits until every piece of data has arrived, then asks its question, then grades the answer. That design tests one skill well: making sense of events once they are over. That skill matters. Investigations, audits and post-mortems depend on it.","But a growing share of the work we want from agents happens while events are still unfolding: watching a machine, managing a schedule, running a process. In that work, the answer has a deadline. A diagnosis that arrives after the moment to act has passed is not a slightly worse answer. It is a wrong one.","An agent that is right too late is still wrong. Static benchmarks cannot see this failure. Worse, the models trained on them learn to commit it."]},{"title":"The skill we are not measuring","paragraphs":["Think about the people we trust most in any field: the operator who can tell a machine is about to fail, the engineer who sees trouble coming in the grid load, the assistant who mentions a problem before it becomes one.","What makes them valuable is not that they know more. It is that they know when. They act on partial evidence. They know how much evidence is enough. They know which question to ask and whom to ask. And they know that waiting is itself a decision, often the most expensive one.","That is judgement under time. It is the core of nearly every job that matters. Our benchmarks do not test it at all."]},{"title":"Objection one: “Just make the agent act faster.”","paragraphs":["If waiting is the problem, the obvious fix is speed. Lower the latency, check more often, alert sooner.","That answer is wrong, and it is wrong in a way that shows the whole problem.","An agent that acts at every flicker is not vigilant. It is noise. False alarms cost attention, and attention runs out. The first false alarm gets checked. The third gets ignored. And the true alarm, when it finally comes, gets ignored too.","So there are three ways to fail, not one. Too late, and the moment is gone. Too early, and trust is gone. To the wrong person, and both are gone. The skill is not speed. It is the narrow path between them: enough evidence, in time, to the person who can act. No dial labelled “faster” finds that path. Only judgement does."]},{"title":"Objection two: “Timing is an engineering problem, not a model problem.”","paragraphs":["Put the model in a loop, the argument goes, poll it every few minutes, and timing takes care of itself.","But a loop only decides how often the model is asked. It does not decide what the model should do when asked. Every time it wakes up, the model faces the same question: is what I see now enough to act on, or should I wait? Is this a pattern or a blip? Who needs to know?","No scheduler answers those questions. They are reasoning. And a model trained only on finished questions has never had to answer them."]},{"title":"Objection three: “Static benchmarks are cheap, reproducible, and comparable.”","paragraphs":["They are. That is their real value, and it is why they have lasted.","But it does not matter how cheaply you can repeat a measurement if it measures the wrong thing. Being able to repeat a test does not make it valid. And there is no trade-off to accept here: an environment that releases its data on a clock, the same way every time, is just as repeatable as a static test. The world can be replayed. Most evaluations simply choose not to replay it."]},{"title":"Why it gets worse, not better","paragraphs":["This is not a gap that scale will close on its own. It is a gap that training widens.","A reward that checks only the final answer sends a clear signal: gather everything, commit to nothing, speak at the end. Every training run built on that signal pushes models further toward waiting. The industry is not failing to teach timing. It is teaching the opposite.","Reinforcement learning has long had a name for the deeper difficulty: credit assignment. When the reward arrives long after the action that earned it, the learner has to work out which earlier choices mattered. [1] In tasks where data keeps arriving, that is the whole game: was the decision on day four what saved the outcome on day eleven? A static benchmark never asks."]},{"title":"The evidence","paragraphs":["We built environments to test this. Data arrives on a clock. People reply after a delay. The agent knows only what has reached it so far, and the score depends on what it says, to whom, and when.","On easier tasks, the best frontier model scores 92% or more.","On hard tasks, it scores 32%.","The models are not short of knowledge. They are short of timing."],"figure":"levels","figureDescription":"Personal agents, 2 November 2026. Easier levels: the best model scores 92% or more, so reward comes often. Hard level: the best model averages 32%, so most of the reward is still to earn. Each level is its own task with its own grader."},{"title":"What follows","paragraphs":["If timing is part of the answer, three things follow.","Evaluations must let time run. Tasks should release data as the world would, and keep separate when something happened, when the agent could have known, and when the outcome became clear.","Rewards must count the moment. Not just whether the agent was right, but whether it was right while being right still mattered.","Progress must be measured honestly. A model that scores near-perfectly on frozen tasks and collapses on live ones has not mastered the task. It has mastered the test.","Agents worth trusting in homes, plants, hospitals and power grids will not come from grading them on how well they explain what already happened. They will come from grading them the way the world grades all of us:","Not only on whether we were right.","On whether we were right in time."]}],"sources":[{"title":"Sutton and Barto — Reinforcement Learning: An Introduction (2nd edition)","href":"http://incompleteideas.net/book/the-book-2nd.html"}]},{"slug":"frontier-models-reasoning-over-time","title":"What frontier models show about reasoning over time","url":"/research/frontier-models-reasoning-over-time","kind":"field-note","date":"2026-11-02","updatedAt":"2026-11-02","author":"Reific","related":["timing-is-part-of-the-answer","a-curriculum-for-reasoning-over-time","why-we-build-from-real-records"],"listed":true,"publications":[],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=frontier_models_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/frontier-models-reasoning-over-time/markdown","summary":"Version 7 of our leaderboard puts 15 models on industrial and scientific systems and 14 on personal agents. No model earns half of the possible reward in any family, and different labs lead different families.","question":"How well do frontier models reason over time on real records?","sections":[{"title":"The leaders","paragraphs":["Each score is the share of the possible reward that a model earns, averaged over the environments of a family, with its 95% interval. These are the leaders in version 7 of the leaderboard, from 2 November 2026.","The three families test different kinds of work. On personal agents, an assistant runs on a person's phone while new data arrive and people reply late; the score is for the hard level. On industry and on science and compute, the agent works through months of records from a real system with code, finds what happened and why, and submits its answer with the evidence."],"points":["Industry: Claude Opus 5.5 leads with 43.1% (±5.4). Gemini 4 Argon follows with 35.0%, and Claude Sonnet 5.5 with 34.1%.","Science and compute: Claude Opus 5.5 leads with 29.7% (±14.6). GPT-6 Astra follows with 22.2%, and GPT-6.1 Sol with 16.7%.","Personal agents, at the hard level: Gemini 4 Argon leads with 31.6% (±5.4). Claude Opus 5.5 follows with 27.0%, and Claude Sonnet 5.5 with 17.4%."],"figure":"family-leaders","figureDescription":"Leaderboard version 7, 2 November 2026; each model's mean share of the possible reward with its 95% interval. Personal agents, hard level: Gemini 4 Argon 31.6% (±5.4), Claude Opus 5.5 27.0% (±4.0), Claude Sonnet 5.5 17.4% (±4.0). Industry: Claude Opus 5.5 43.1% (±5.4), Gemini 4 Argon 35.0% (±5.6), Claude Sonnet 5.5 34.1% (±5.8). Science and compute: Claude Opus 5.5 29.7% (±14.6), GPT-6 Astra 22.2% (±11.5), GPT-6.1 Sol 16.7% (±6.1)."},{"title":"What a score means","paragraphs":["A score is not the share of tasks that a model solves. It is the share of the possible reward that the model earns. A model can earn part of the reward on a task that it does not finish, and on a person's phone it can lose reward on a task that it answers, if it answers late or raises a false alarm on the way.","That makes the scores comparable across families, and it makes them useful for training. A model that moves from 30% to 40% has learned something that the board can see, even if it still finishes few tasks in full."]},{"title":"No family is close to solved","paragraphs":["The best model earns less than half of the possible reward in every family. These are not trick questions. Each answer key is checked against the records, and every task has a reference answer that earns the full score.","The tasks are hard because the work is hard. The histories are long, the data keep arriving, and the evidence is spread over many sources. A model must choose where to look, keep track of what it knew at each moment, and resist the first explanation that fits.","That leaves room at the top. A benchmark that the best model almost passes stops telling models apart, and it stops being useful for training too. On every family, the leader is still further from the full score than it is from the models below it."]},{"title":"The same conditions for every model","paragraphs":["Every model runs on the same environments with the same tools. On industry and on science and compute, every model also gets the same harness and the same budget.","The board also changes as new models arrive. Versions 3 to 7 added Claude Haiku 5.5, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, Qwen3.8-Flash-Next, MAI-Thinking-1 and Nemotron 3 Ultra. None of them entered the top three of any family, so the leaders above are the same as they were in version 3."]},{"title":"Different labs lead different families","paragraphs":["Gemini 4 Argon leads on a person's phone, where data arrive while the agent works and people reply late. Claude Opus 5.5 leads on industrial and scientific systems, where the agent works through months of records with code.","A single benchmark number would hide this. The two kinds of work ask for different strengths. On a phone, the agent must decide when the evidence is enough and whom to tell. On a recorded history, it must search a long record with code and build an answer that the evidence supports. A model that is strong on one kind of time is not always strong on the other.","For a team that chooses a model, the lesson is practical. Measure the model on the kind of time your product has. A model that tops a general board may not top yours."]},{"title":"Read the intervals","paragraphs":["Every mean on the board comes with its 95% interval, because a single number hides how much a score moves between runs. Evaluations of language models are statistical experiments, and their results need error bars before anyone compares them. [2]","Some gaps are larger than their intervals and some are not. On industry, Gemini 4 Argon at 35.0% and Claude Sonnet 5.5 at 34.1% sit well inside each other's intervals, so the board does not separate them. The gap from Claude Opus 5.5 at 43.1% to the next model is larger than the interval of either one. On science and compute, the intervals are wider than the gaps, so even the order at the top is less settled than on the other two families."]},{"title":"A high score is hard to fake","paragraphs":["On industrial and scientific systems, wrong answers, including answers that take a shortcut, score 10% or less. Numeric checks need no judge. Two judges from different labs grade the rest against keys checked against the records, because a language model that grades answers can favour answers that look like its own. [3]","On a person's phone, late messages and false alarms lose points. A model earns its score by finding what happened and saying it in time, not by sounding confident."]},{"title":"Run it yourself","paragraphs":["Each environment is a Harbor task with its own grader, so you can run your own model under the same conditions and compare it with the board. The methodology page explains how we run and score each model. [1] To get started, request samples or talk to a researcher.","The board shows averages with their intervals, not scores for single environments. That keeps the measure honest: no one can tune a model to one task and climb the board with it."]}],"sources":[{"title":"Reific — Leaderboard methodology","href":"https://reific.co/environments/methodology"},{"title":"Miller — Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations","href":"https://arxiv.org/abs/2411.00640"},{"title":"Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena","href":"https://arxiv.org/abs/2306.05685"}]},{"slug":"why-we-build-from-real-records","title":"Why we build from real records","url":"/research/why-we-build-from-real-records","kind":"field-note","date":"2026-11-02","updatedAt":"2026-11-02","author":"Reific","related":["timing-is-part-of-the-answer","frontier-models-reasoning-over-time","a-curriculum-for-reasoning-over-time"],"listed":true,"publications":[],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=real_records_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/why-we-build-from-real-records/markdown","summary":"Real operating records have gaps, drift, late corrections and sources that disagree. Synthetic data rarely has them. An environment built from real records teaches the work as it really is.","question":"Why build RL environments from real records instead of synthetic data?","sections":[{"title":"Clean data teaches the wrong lesson","paragraphs":["Take a water pumping station that reports pressure, flow and the level of its tank every minute. Over three weeks the pressure falls a little, inside the normal daily swing. The level sensor stops reporting for two days. An operator writes about a stop in the log four days after it happened, and the alarm list shows the same alarm forty times in one hour.","A synthetic version of this station would be smooth, complete and labelled. Every reading would arrive on time, every sensor would tell the truth, and a label would say what went wrong. An agent that learns there learns that data are complete and that labels tell the story. Neither is true in a real plant.","The wrong lesson shows up as soon as the agent meets a real record. It adds up a week of flow, treats the two silent days as zero, and reports a drop that never happened. It reads the forty alarms as forty events. It trusts the log entry for the date the operator wrote it, not the date of the stop. Each mistake is small, and together they produce a confident answer about a problem that is not there."],"figure":"missing-data","figureDescription":"Controlled public-data example, 12 January 2016, source timestamp labels 00:00–01:50. Original: 12/12 readings, 1,860 Wh total, mean 155 Wh per reading. Constructed missing version: 8/12 available readings (66.7% coverage), observed sum 420 Wh and observed-only mean 52.5 Wh; full-window total and mean unknown from this incomplete record. Constructed zero version: all 12 values present, including four synthetic zeroes, total 420 Wh and mean 35 Wh. Zero-filling the gaps invents four observations and changes the denominator. No physical fault, agent accuracy, or product advantage is established."},{"title":"What real records contain","paragraphs":["Real records hold the difficulties that an agent meets in real work, in the proportions that real work has them."],"points":["Gaps. A sensor goes quiet, and a missing reading is not a zero.","Drift. A slow change hides inside the normal noise for weeks.","Late corrections. A log entry or a reading changes after the fact.","Disagreement. Two sources tell different stories, and the agent must find out which one is right.","People. Notes and messages that people write in their own words, at their own time."]},{"title":"The difficulties carry information","paragraphs":["None of these is noise to clean away before the real work starts. Each one tells the agent something. A gap may mean that a sensor failed, that a site lost power, or that a link went down, and those three causes lead to different actions. A late correction shows who noticed a problem, and when. Two sources that disagree point at the one that broke.","Drift is the hardest of them. The statistics of a real system change over time, so a pattern that meant one thing last month can mean another thing now. [1] An agent that has seen only stable synthetic series has no reason to ask whether the ground moved. An agent trained on real histories learns to ask that question first."]},{"title":"Real records make honest answer keys","paragraphs":["Each answer key is checked against the records. The key says what happened, not what a generator decided. The records stay messy and the key stays true, so the agent must reach the answer the way an engineer would: from the evidence.","When a generator writes the data, it also writes the answer, and a task built that way tests whether an agent can reverse the generator. A model can learn the generator's habits instead of the world's: the fault always starts on a round hour, the label always follows the spike. Real records have no such habits to learn. The only way to the answer is through the evidence."]},{"title":"Where synthetic data fits","paragraphs":["Synthetic data still has a place. It is the right tool for a unit test of a method, for a rare case that a team wants to show on purpose, and for privacy when a record cannot leave its owner.","But a model that learns mostly from generated data drifts away from the real distribution, and the rare cases in its tails disappear first. [2] A generator can make a sensor fail, but only in the ways its author imagined. Real failures are stranger: a sensor that repeats its last good value for a week, a clock that jumps an hour twice a year, a technician who logs a repair under the wrong unit. For reasoning over time, the rare cases are the point. The slow leak, the sensor that fails once a year and the correction that arrives a week late are the events that an assistant exists to catch."]},{"title":"What an agent learns from real records","paragraphs":["An agent that trains on real records learns habits that clean data never asks for. It checks how much of a period it can actually see before it adds anything up. It looks for a second source before it trusts the first. It reads the date on a log entry with care, because the date a person wrote is not always the date of the event. And when part of the picture is missing, it says so, and it asks someone who can fill the gap.","These habits are what separate an assistant that people trust from one they check behind. They are hard to teach with rules, because every system breaks in its own way. They are easier to learn from many real histories, each with its own gaps and its own mistakes."]},{"title":"Where the records come from","paragraphs":["We buy and license months or years of time-stamped records from operations, sensors, labs and compute, and we build RL environments from them. The most useful records are long, come from more than one source for the same system, and keep the notes that people wrote at the time. If you hold records like these, partner with us.","Records like these are rarer than they look. Most organisations collect them to run their own operations, and few think of them as material for training. A history of years, with its notes and its mistakes, is exactly what a model needs to learn the work."]}],"sources":[{"title":"Lu et al. — Learning under Concept Drift: A Review","href":"https://arxiv.org/abs/2004.05785"},{"title":"Shumailov et al. — AI models collapse when trained on recursively generated data (Nature, 2024)","href":"https://www.nature.com/articles/s41586-024-07566-y"}]},{"slug":"how-experts-shape-our-environments","title":"How experts shape our environments","url":"/research/how-experts-shape-our-environments","kind":"field-note","date":"2026-11-10","updatedAt":"2026-11-10","author":"Reific","related":["why-we-build-from-real-records","timing-is-part-of-the-answer","a-curriculum-for-reasoning-over-time"],"listed":true,"publications":[],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=experts_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/how-experts-shape-our-environments/markdown","summary":"Every environment we ship is checked by people who do the work it is about: clinicians for health tasks, engineers and operators for industrial ones, scientists and site reliability engineers for the lab and the data centre. Their decisions are already in the records we build from, their rules set the thresholds, they review how we grade, and every lesson from their reviews goes into the next environment we build.","question":"Where do experts come into Reific's RL environments?","sections":[{"title":"Experts are already in the record","paragraphs":["Every record we build from holds more than sensor readings. Somewhere in it, a person who knew the system made a decision."],"points":["A GP notes that a cough has become a chest infection.","A wind technician writes that a gearbox bearing was worn, not the sensor.","A control-room operator logs when a compressor tripped and why.","A scientist marks a bioreactor batch as contaminated on day six.","An on-call engineer closes an incident on a GPU cluster with the real cause."],"after":["Those entries matter more to us than any label we could add later. They were written at the time, by someone with the authority and the context to make the call. An agent working on the record should give them the weight a colleague would. We build our answer keys on decisions like these, and check them against what the record shows later."]},{"title":"The threshold comes from whoever owns it","paragraphs":["We never invent a threshold. If a task needs one, the person or organisation responsible for the system has already set it, in a care plan, a maintenance manual, an alarm list, a batch specification or a runbook. The agent has to find that rule and apply it to the data exactly as written.","At home. A woman with asthma checks her peak flow every morning. Her personal best is 420 litres a minute, and the action plan she completed with her doctor splits that into three zones. At 336 or more (green) she carries on. From 210 to 335 (yellow) she takes her reliever and checks again an hour later. Below 210 (red) she takes it and calls her doctor at once. [1] From Monday her readings fall: 390, 360, 340, then 320 on Thursday. On Friday there is no reading.","On a wind farm. The operator's alarm list sets a warning at 80 °C for a turbine's gearbox bearing and a shutdown at 90 °C. Over a week the daily peak climbs: 72, 76, 79, then 83 °C. Then the turbine's data link drops for two days.","The questions are the same in both cases, and both can be checked. Was the limit crossed, and on which day? What does the plan or the alarm list say to do next? The answer comes from the data and the rule, so there is no opinion of ours in the key. The gap matters too: nothing in either record says things are back to normal."],"figure":"expert-roles","figureDescription":"Who decides each part of a health task. What happened, and when: the record, with the clinician's own notes. What the threshold is: the patient's own plan. What advice is safe: safety rules, checked by clinicians. Whether the task matches real practice: clinicians who review the tasks. Whether the grading is fair: people who check the judges' grades."},{"title":"Some mistakes fail the task outright","paragraphs":["An agent never loses marks for escalating to the right person. But an answer with correct figures still fails if it crosses a line that an expert in that field would never cross. For example:"],"points":["Personal agents (Timestep-Assist): telling someone not to contact their care team; sending someone to the emergency department on a wearable's reading alone; stating a diagnosis as fact when the record only shows a change; suggesting a treatment change their own plan does not give.","Industry (Timestep-Ops): advising a restart before the cause of a trip is known; recommending that a safety alarm be bypassed or its limit raised; stating a root cause as fact when the data only shows a correlation.","Science and compute (Timestep-Lab): releasing a batch that is outside its specification; draining or rebooting nodes that are serving production work without the step the runbook requires."],"after":["The experts who review each family write and check these rules."]},{"title":"Expert help takes time","paragraphs":["Help from a professional is rarely instant. Many nurse lines open only in office hours. A field crew cannot reach an offshore turbine in a storm. A vendor's support desk answers the next working day.","The nurses, engineers and support teams that an agent meets in our environments behave the same way. The agent can ask them, and on some tasks it has to, but the reply comes when a real reply would come and holds only what that person would know. So the agent has to decide whether a question is worth the wait, and what to do in the meantime."]},{"title":"Every environment is reviewed before it ships","paragraphs":["A high score can hide problems that only an expert will see. In one study, a model set a new best score on medical multiple-choice questions, yet clinicians who read its full answers still found key gaps. [2] The same holds outside medicine: an answer can pass a check and still be one no engineer would act on.","That is why every environment we ship is reviewed by people from the field it covers. Clinicians review our health tasks. Engineers and operators review our industrial tasks. Scientists and infrastructure engineers review our lab and data-centre tasks. They read the questions, the safety rules and the answers we give credit for, and they tell us where a task does not match their real work: a question no patient or operator would ask in those words, or an action no clinic or control room would take."]},{"title":"Experts check the judges too","paragraphs":["Many answers are graded by language models acting as judges, and judges make mistakes. So the judging system goes through the same expert review as the tasks."],"points":["Experts from the field read the grading rules and the rubrics the judges apply, and flag any rule that would reward an answer they would never accept, or punish one they would.","People grade a sample of answers before they see the judges' grades, and say how a real user, clinician or operator would feel about each answer.","Where the judges and the experts disagree, we find out why, then fix the rule or the judge."],"after":["We treat an unfair or unclear grading rule as seriously as a wrong grade. A judge that is right on average but wrong on the cases that matter teaches a model the wrong lesson."]},{"title":"Every lesson goes into the next environment","paragraphs":["Each review teaches us something, and we write it down as a rule for how we build. Every new environment starts from the full set of those rules, in every family, so a lesson learned on a health task also improves the next industrial or lab task.","Some examples of rules that came from review:"],"points":["One reviewer noticed that some questions read like an exam, not like a message to an assistant or an operator. We rewrote them, and every environment since is written the way real people ask.","Reviewers found tasks where the answer depended on a reading the agent could not yet have seen. Every environment now checks that answers use only the data available at the decision time.","Judges sometimes gave credit for a confident answer that ignored a gap in the data. Every grading rule now asks whether the answer notices missing data."],"after":["The result is that our environments get better with each release. The newest ones carry every lesson from the ones before them."]},{"title":"What experts decide, and what the record decides","paragraphs":["The usual way to bring experts into AI evaluation is to have them write a grading rubric for each case. One public health benchmark did this with 262 physicians across 5,000 conversations. [3] The approach has limits. Two experts can read the same case differently. A rubric written afterwards cannot see what happened next. And every new task needs more expert hours.","We divide the work another way. The people responsible for each system set the thresholds, in their own plans, manuals and runbooks. The experts who review our work check the safety rules, the judging system, and that each task matches real practice. The record decides what happened and when, and we grade each answer against it. What the experts teach us once, every later environment keeps.","Every new record adds to this. A few years of one patient's notes, one wind farm's maintenance log or one cluster's incident history can hold dozens of decisions, each made by someone who knew the system. An agent that trains on those records learns from all of them."]}],"sources":[{"title":"NHLBI — Asthma Action Plan (NIH Publication No. 20-HL-5251)","href":"https://www.nhlbi.nih.gov/health-topics/all-publications-and-resources/asthma-action-plan-0"},{"title":"Singhal et al. — Large language models encode clinical knowledge (Nature, 2023)","href":"https://www.nature.com/articles/s41586-023-06291-2"},{"title":"Arora et al. — HealthBench: Evaluating Large Language Models Towards Improved Human Health","href":"https://arxiv.org/abs/2505.08775"}]},{"slug":"a-curriculum-for-reasoning-over-time","title":"A curriculum for reasoning over time","url":"/research/a-curriculum-for-reasoning-over-time","kind":"field-note","date":"2026-11-02","updatedAt":"2026-11-02","author":"Reific","related":["timing-is-part-of-the-answer","frontier-models-reasoning-over-time","why-we-build-from-real-records"],"listed":true,"publications":[],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=curriculum_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/a-curriculum-for-reasoning-over-time/markdown","summary":"Each of our environments has easier levels and a hard level. The best models clear the easier levels and still fail the hard one. That gradient makes an environment useful for training, not only for evaluation.","question":"What makes an RL environment useful for training, and not only for evaluation?","sections":[{"title":"An eval measures. A curriculum teaches.","paragraphs":["An evaluation needs tasks that separate models. Training needs more than that. If no model ever earns a reward, there is nothing to learn from. If every model earns the full reward, there is nothing left to learn. Reinforcement learning works best on tasks at the edge of what a model can do.","The idea is older than language models. People learn faster when the examples start easy and grow harder, and so do neural networks: a curriculum that orders training from simple to complex can speed up learning and lead to better results. [1] In reinforcement learning the same insight has a sharper form. Methods that keep a pool of tasks and replay the ones with the most to teach, the tasks where the agent is neither sure to succeed nor sure to fail, learn faster and generalise better than methods that sample tasks at random. [2]","The same environment can serve both purposes. A team can evaluate a model on the hard level, train it across all the levels, and evaluate it again on the hard level to see what changed. The reward that drives the training and the score that measures the result come from the same grader.","A whole field now studies how to choose the next task for a learning agent. [3] Its first requirement is simple: the tasks must span a range of difficulty, from tasks the agent can already solve to tasks it cannot yet touch."]},{"title":"Easier levels and a hard level","paragraphs":["Each environment has easier levels and a hard level, and each level is its own task with its own grader. At the easier levels of our personal agents family, the best model scores 92% or more. At the hard level, the best model averages 32%.","Each level is its own task with its own grader, so a team can train on any mix of levels and measure progress on each one separately.","That gap is the useful part. A model can earn reward early, on the easier levels, and keep earning it as it improves, up to the hard level. The levels share one purpose: each one asks for the same kind of judgement about a world that keeps changing, at its own difficulty."],"figure":"levels","figureDescription":"Personal agents, 2 November 2026. Easier levels: the best model scores 92% or more, so reward comes often. Hard level: the best model averages 32%, so most of the reward is still to earn. Each level is its own task with its own grader."},{"title":"Why reasoning over time needs the gradient","paragraphs":["The hard level asks for several things at once. The agent must notice a change inside a long history, decide when the evidence is enough, find the person who holds the missing fact, and act before the moment passes. A model that cannot yet do any of these well almost never earns a reward on the hard level, and a reward it never earns teaches it nothing.","Easier levels give the same model a chance to earn reward while it learns the habits that the hard level needs. Each success is a signal that training can use. As the model gets better, the hard level comes within reach, and it keeps a clear target in front of the model until the end.","Time adds its own difficulty to learning. When a task runs for a long time, the step that earned the reward may come hours before the reward itself, and a model must learn which of its early steps mattered. Rewards that come often, as they do on the easier levels, give a model more signal while it learns which steps matter.","The gradient also helps the team that runs the training. A model that improves on the easier levels and stalls on the hard one shows where its limit is. A model that improves on both shows that the training transfers."]},{"title":"What to look for in training environments","paragraphs":["A few questions help a team choose environments for training. Does the best current model still fail the hardest level? If not, the environment has nothing left to teach the models that matter. Do the easier levels give reward often enough for training to start? Does the grader check answers against the records, so that a model cannot earn reward with a shortcut? And can the environment run many copies at once without failures?","Our environments are built to answer yes to each of these questions. The best model still fails the hard level, the easier levels give reward often, the keys are checked against the records, and the environments run in parallel."]},{"title":"Built to run at scale","paragraphs":["Training needs many runs at once, without failures. Each level is a Harbor task: the agent and the data in one container, the grader in another. The reference solution replays offline and earns the full score, so a team can check its setup before it spends anything on training. Our stress test ran 64 sandboxes at once, with no failures. Because each level is its own task, a team can spread its runs over many machines and mix the levels in every batch.","Reliability matters more for training than for evaluation. An evaluation that fails once can run again. A training run that loses rollouts to broken environments learns from a biased sample, and the team may never see why."]},{"title":"Models improve","paragraphs":["Models that train on our environments improve. We license environments for RL training, and we build new ones to order. Samples are available the same day, so a team can run a model on them before it decides anything."]}],"sources":[{"title":"Bengio et al. — Curriculum Learning (ICML 2009)","href":"https://ronan.collobert.com/pub/matos/2009_curriculum_icml.pdf"},{"title":"Jiang, Grefenstette and Rocktäschel — Prioritized Level Replay (ICML 2021)","href":"https://arxiv.org/abs/2010.03934"},{"title":"Portelas et al. — Automatic Curriculum Learning For Deep RL: A Short Survey","href":"https://arxiv.org/abs/2003.04664"}]},{"slug":"frontier-evals-time-series-reasoning","title":"Frontier evals for LLM reasoning over time series","url":"/research/frontier-evals-time-series-reasoning","kind":"field-note","date":"2026-10-08","updatedAt":"2026-11-02","author":"Reific","related":["everest-rare-event-forecasting"],"listed":true,"publications":[],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=evals_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/frontier-evals-time-series-reasoning/markdown","summary":"Frontier models handle simple questions about time series well. Long histories, decision cutoffs and questions with a trap in them still break them. For frontier evals of LLM reasoning over time series data, reach out to us.","question":"How do you measure whether a frontier model can reason over long time-series histories?","sections":[{"title":"Simple questions are close to solved","paragraphs":["Ask a frontier model with a Python sandbox when a sensor first stayed above a limit, which unit changed most between two weeks, or whether a reading is missing or zero, and it will usually get it right.","That is real progress, and it changes what an evaluation has to do. A benchmark built from single-step questions now measures arithmetic and tool use. It cannot separate one frontier model from the next, and it says little about whether a model can carry out an investigation that an engineer would trust. Anthropic's guidance on evaluating agents stresses realistic tasks and clear, repeatable grading. [1] Single-step questions meet the second standard and no longer meet the first.","Time series add their own difficulty. Public question-answering sets for operational data, such as ARFBench for software incident response, already ask questions about when something happened and how long it lasted. [2] In our experience the difficulty grows quickly as the history gets longer and the question gets more specific."]},{"title":"Where frontier models still fail","paragraphs":["The hard part starts when the answer depends on reasoning across a long, real history rather than a single calculation. A model has to decide which windows matter, keep track of what was known at the moment a decision was made, and resist the explanation a quick look suggests. Each of those steps is easy to describe and easy to skip."],"points":["Long histories, where the relevant minutes sit inside weeks of ordinary data and the model must choose where to look.","Decision cutoffs, where a record that arrived later must not leak into an earlier judgement.","Questions with a trap: a pattern that looks like the answer and is not, such as a daily cycle mistaken for a change.","Multi-step investigations, where each step depends on getting the last one right and an early slip compounds."]},{"title":"Leakage is the quiet failure","paragraphs":["Historical cases are attractive because the outcome is already known. They also make it easy to grade a model on information it could not have had. A fault recorded at 09:12 may only reach the investigation system at 09:24; an evaluation of a decision taken at 09:20 must exclude it. Feature stores solve the same problem with point-in-time joins, which attach to each row only the values that were available at that row's timestamp. [3]","Evaluations of time-series reasoning need the same discipline. Every question should carry its decision time, every file should be cut at that time, and the grader should check that the answer does not depend on anything later. Forecasting research has long used rolling-origin evaluation for the same reason: a method is scored only on data that lay in its future. [4]"],"figure":"decision-time","figureDescription":"Illustrative point-in-time evaluation: a fault occurs at 09:12, a decision is made at 09:20, a delayed record arrives at 09:24, and a repair finding is recorded the next day. The late record and repair finding are excluded from the 09:20 input; later evidence can support retrospective scoring. Event time alone does not establish availability."},{"title":"What a useful eval needs","paragraphs":["Every question needs an answer that can be checked against its data, not graded by opinion. The question has to be fair: answerable from the information available at the stated time, and hard for a reason that matters in practice rather than a trick of wording. If a careful analyst with the same files could not reach the answer, the question is broken, not the model.","Scoring should separate what a model observed from what it concluded, so a correct reading with a wrong diagnosis is not counted as a success, and a cautious answer that names its uncertainty is not punished as a miss. Teams running agents in production build infrastructure to replay real incidents and evaluate the agent's behaviour on them, not only on synthetic prompts. [5] The grading should look at the evidence and the reasoning, not only the final string.","The questions also have to come from histories that behave like real operations, with gaps, noise, regime changes, delayed records and events. Clean synthetic series are useful for unit tests of a method. They are not where frontier reasoning breaks."]},{"title":"From evals to environments","paragraphs":["An evaluation asks its question once. Our RL environments go one step further: the history keeps arriving while the agent works. New records reach it on a clock, people reply after a delay, and the agent must decide when the evidence is enough to act. The rules above still apply. Every answer key is checked against the records, nothing that arrives later may decide an earlier judgement, and the histories are real, with their gaps, noise and late corrections.","That turns a question into a task with a reward. On industrial and scientific systems, the agent works through months of records with code and submits its answer with the evidence. Numeric checks need no judge, and two judges from different labs grade the rest against keys checked against the records. On a person's phone, the score counts what the agent tells whom, and when: late messages and false alarms lose points.","The results show the gap that single questions hide. On our leaderboard, no frontier model earns half of the possible reward in any family. Our personal agents environments also come with easier levels and a hard level, so the same environment can measure a model and train one. Each environment is a Harbor task with its own grader, and the reference answer for every task earns the full score, so anyone can run a model on it and compare."]},{"title":"Work with us","paragraphs":["We build evaluations and training environments for LLM reasoning over time series, from operational histories, with answers verified against the data and decision times enforced. They can be scoped to a domain, a data shape or a capability you want to measure.","For frontier evals of LLM reasoning over time series data, reach out to us."]}],"sources":[{"title":"Anthropic — Demystifying evals for AI agents","href":"https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents"},{"title":"Xie et al. — ARFBench: Time Series Question Answering for Software Incident Response","href":"https://arxiv.org/abs/2604.21199"},{"title":"Feast — Point-in-time joins and data availability","href":"https://docs.feast.dev/getting-started/concepts/point-in-time-joins"},{"title":"Hyndman and Athanasopoulos — Time series cross-validation","href":"https://otexts.com/fpp3/tscv.html"},{"title":"Datadog — Building a real-world evaluation platform for autonomous SRE agents","href":"https://www.datadoghq.com/blog/engineering/bits-ai-eval-platform/"}]},{"slug":"everest-rare-event-forecasting","title":"EVEREST: when the rare event is the one that matters","url":"/research/everest-rare-event-forecasting","kind":"research","date":"2026-09-17","updatedAt":"2026-11-02","author":"Reific","related":["frontier-evals-time-series-reasoning"],"listed":true,"publications":[{"title":"EVEREST: A Transformer for Probabilistic Rare-Event Anomaly Detection with Evidential and Tail-Aware Uncertainty","venue":"ICLR 2026 · Main conference","href":"https://proceedings.iclr.cc/paper_files/paper/2026/hash/0b6566f471cd601cbfc895962623124e-Abstract-Conference.html"},{"title":"Forecasting Solar Flares with the EVEREST Transformer","venue":"TSALM · ICLR 2026 workshop","href":"https://openreview.net/forum?id=KgjNR9TgCB"}],"resources":[],"demo":{"url":"https://calendly.com/reific/demo?utm_source=reific&utm_medium=website&utm_campaign=agent_discovery&utm_content=research_article","prompt":"Request samples, or talk to a researcher about the environments."},"markdown":"/research/everest-rare-event-forecasting/markdown","summary":"A closer look at rare-event forecasting: what a probability means, how an alert changes a decision, and what the evidence supports.","question":"What makes a rare-event forecast useful enough to act on?","sections":[{"title":"The quiet answer can look excellent","paragraphs":["Imagine reviewing a system that almost always says everything is fine. Most of the time, it is right. Its aggregate accuracy looks reassuring. Then ask a different question: how many of the events that required attention did it catch? A model can perform well on the first question and fail completely on the second.","EVEREST, first-authored by Reific founder Antanas Žilinskas with Robert N. Shorten and Jakub Mareček, was published at the ICLR 2026 main conference. It studies rare-event forecasting from numerical time series. [1]","A related paper by the same authors, “Forecasting Solar Flares with the EVEREST Transformer,” was accepted at TSALM, the ICLR 2026 workshop on Time Series in the Age of Large Models. [7]","The illustration below applies an always-quiet rule to a published evaluation population. It makes the cost of an uninformative answer visible. This rule is an explanatory example, not the research model. The useful question is what we would need to measure before trusting a better answer."],"figure":"rare-events","figureDescription":"M5–72h: 71,729 windows; 71,625 negative, 104 positive. Always-quiet illustration: 99.86% accuracy, 104/104 positive windows missed. Not EVEREST performance; windows are not independent flares. Appendix B, Table 5."},{"title":"What the model is trying to learn","paragraphs":["EVEREST uses a transformer to combine measurements across time and estimate an event probability within a specified horizon. Its training includes objectives for rare cases, uncertainty, and extreme outcomes; inference uses one classification output. Solar flares provide the main evaluation. [2]","The distinction between training and use matters. During training, we can ask more of a model than simply choosing the correct label. We can shape which mistakes receive attention, then examine whether those choices improve the output we actually need. The test remains the observed behavior of that output.","One relevant idea comes from focal loss, developed for object detection: many easy examples can overwhelm the contribution of difficult ones. Reducing their weight gives learning more reason to resolve the remaining errors. That principle concerns the training signal; it does not establish that every unusual observation deserves an alert. [3]","For an operator, the event definition comes first. A brief excursion above a limit and a sustained loss of service are different targets. A prediction about the next hour and a prediction about next week are different commitments. Without those definitions, an apparently precise probability is difficult to test or use."]},{"title":"A probability should earn its meaning","paragraphs":["Calibration asks whether predicted probabilities agree with observed frequencies. Consider a hypothetical collection of comparable forecasts, each assigning an event an 80% probability. If the probabilities are calibrated for that collection, events should occur about four times in five over enough observations. A lone successful prediction cannot establish this property. Guo and colleagues showed why classification performance and probability calibration need separate evaluation. [4]","The practical consequence is that a model's confidence should be checked where it will influence a decision. An average across ordinary conditions may answer a different question from the one posed by the subset that triggers intervention. Sample size also matters: a narrow slice can look reassuring while providing little opportunity to observe failure.","We would therefore want a forecast report to make its reference population visible. Which operating conditions were represented? How often did the event occur? How many examples support the probability range being used? Those details make it possible to distinguish an estimate with evidence behind it from a number that merely looks precise."]},{"title":"Read the errors from both sides","paragraphs":["For ≥M5 flares within 72 hours, Table 12 reports headline means under the paper’s five-seed/bootstrap protocol: TSS 0.966, precision 0.727, and recall 0.966. [2]","These metrics answer different questions. Recall asks how many positive cases were detected. Precision asks how many positive predictions were correct. Saito and Rehmsmeier explain why precision and recall deserve particular attention when positive cases are scarce: an evaluation can otherwise obscure the false alarms within the predicted-positive group. [5]","True Skill Statistic subtracts the false-positive rate from the true-positive rate. It is a discrimination score, not a percentage of predictions that are correct. For a review team, the accompanying counts are often the most useful starting point. Read the detected cases, the missed cases, and the false alarms together, then ask what each would have meant for the person receiving them."]},{"title":"The same forecast can support different alert policies","paragraphs":["A probability threshold connects an estimate to a decision. Changing it changes which observations trigger an alert. If missed events have a different cost from false alarms, a threshold chosen for a balanced metric need not minimize expected cost. Elkan's work on cost-sensitive learning formalizes this separation between estimating probabilities and choosing actions. [6]","The separate diagnostic below makes that choice tangible. Catching the additional positive windows also produces more false-alert windows. The trade-off is visible rather than hidden inside a single score.","In an imagined maintenance workflow, requesting another reading might be inexpensive, while stopping equipment might disrupt a shift. Those actions need not share a threshold. We would first specify the consequence of each action and the evidence required to justify it. We would then evaluate the resulting policy on observations that were not used to choose it.","That is also why a threshold from a research table should stay attached to its task and assumptions. Its value is to reveal how a decision changes, rather than supply a universal setting for another system."],"figure":"alert-thresholds","figureDescription":"M5–72h diagnostic; 104 positive windows. Threshold 0.460: 92 detected, 12 missed, 45 false alerts. Threshold 0.240: 104 detected, 0 missed, 115 false alerts. Miss:false-alarm cost 20:1; separate from headline results. Appendix G."},{"title":"Keep the limits close to the result","paragraphs":["The paper identifies fixed context, data gaps, changing solar cycles, scarce extreme events, and omitted image inputs as limits. Its calibration analysis above probability 0.8 contains only eight windows. [2]","An overall calibration summary cannot settle reliability in a sparsely observed group. Nor can success on one event definition establish performance on another. Our reading is that the useful next question is specific: which new condition would we have to test before allowing this estimate to support that decision?","The same care applies to explanations. A highlighted signal can guide investigation, but deciding that it caused an event requires additional evidence. Keeping prediction, explanation, and action distinct gives each claim a test it can actually pass or fail."]}],"sources":[{"title":"ICLR 2026 proceedings: EVEREST","href":"https://proceedings.iclr.cc/paper_files/paper/2026/hash/0b6566f471cd601cbfc895962623124e-Abstract-Conference.html"},{"title":"EVEREST — final paper and appendices","href":"https://proceedings.iclr.cc/paper_files/paper/2026/file/0b6566f471cd601cbfc895962623124e-Paper-Conference.pdf"},{"title":"Lin et al. (2017): Focal Loss for Dense Object Detection","href":"https://openaccess.thecvf.com/content_iccv_2017/html/Lin_Focal_Loss_for_ICCV_2017_paper.html"},{"title":"Guo et al. (2017): On Calibration of Modern Neural Networks","href":"https://proceedings.mlr.press/v70/guo17a.html"},{"title":"Saito and Rehmsmeier (2015): evaluating precision and recall under imbalance","href":"https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432"},{"title":"Elkan (2001): The Foundations of Cost-Sensitive Learning","href":"https://cseweb.ucsd.edu/~elkan/rescale.pdf"},{"title":"TSALM 2026 — official accepted-paper list","href":"https://tsalm-workshop.github.io/accepted-papers/#zilinskas2026forecasting"}]}],"representations":{"html":"/","agents":"/for-agents","markdown":"/llms-full.txt","json":"/evidence.json","discovery":"/llms.txt"}}