‘Cell recorders’ based on gene editing have captured mouse development in unprecedented detail.

Two independent teams of scientists published the findings today in Science and in Cell. One team recorded the cell-by-cell development of a two-week-old embryo from a single fertilized egg1, and the other captured most of the cell divisions in embryos as organs formed2.

Both studies were inspired by a landmark early-1980s effort: the first — and still only — full map of animal development. Using only a microscope, UK biologist John Sulston observed each cell of the nematode Caenorhabditis elegans as the animal grew from an egg into an adult with precisely 959 somatic, or fully differentiated, cells.

Tracing the lineage of most other animals in this way, particularly mammals, isn’t possible, says Jay Shendure, a genome scientist at the University of Washington in Seattle, who led the Science study. C. elegans is transparent, and each worm develops in precisely the same way, with the same 959 cells having followed an unvarying pattern of cell divisions.

But in mammals such as mice — in which development is hidden and cells number in the billions — external cues such as growth factors influence whether a cell's fate, for instance, whether it will end up in the lungs or liver.

Bizarre CRISPR enzyme kills cancer cells by shredding their DNA

A key question in biology, says Shendure, is how such randomness can give rise to individuals of the same species that have broadly the same form. “Twins look the same, humans kind of look the same, yet even twins develop through very different sets of cell divisions,” he says.

Barcode tracking

Shendure first tackled this question around 2016, when he and his team developed a method based on the gene-editing technology CRISPR that introduced ‘barcodes’ — gene edits at particular locations — into the cells of developing zebrafish (Danio rerio)3. The team then used DNA sequencing to check which barcodes were present in various cells, and inferred relationships between them.

But these CRISPR-based 'lineage recorders' can harm cells if too many edits are introduced during development, says Jonathan Weissman, a molecular biologist at the Whitehead Institute in Cambridge, Massachusetts, who led the study in Cell.

To circumvent these issues, the teams led by Shendure and Weissman used a more precise and less damaging technique, called prime editing, to introduce edits as cells divided during embryo development.

Shendure’s team took a fertilized mouse egg and applied a technique they have named DNA Typewriter, which adds sequential, indelible genetic marks to specific spots in the genome as cells divide. The egg was then implanted into a mouse. After two weeks, when the embryo had formed major organ systems, the researchers read the DNA Typewriter edits and were able to reconstruct the relationships of 1.3 million edited cells —about 10% of the embryo’s total.

Enjoying our latest content?

Log in or create an account to continue

Access the most recent journalism from Nature's award-winning team

Explore the latest features & opinion covering groundbreaking research

Access through your institution

or

Sign in or create an account

Continue with Google

Continue with ORCiD

doi: https://doi.org/10.1038/d41586-026-03193-x

References

Yu, Q. et al. Science https://doi.org/10.1126/science.ael0508 (2026).

Article Google Scholar

Colgan, W. N. et al. Cell https://doi.org/10.1016/j.cell.2026.09.050 (2026).

Article Google Scholar

McKenna, A. et al. Science 353, eaaf7907 (2016).

Article PubMed Google Scholar

Download references

Reprints and permissions

Related Articles

Subjects

Developmental biology

CRISPR-Cas9 genome editing

Stem cells

Latest on:

Developmental biology

Stem cells

Big dogs age faster than little ones — rogue ‘jumping genes’ might be to blame

News 08 OCT 26

Alveolar stem cells transdifferentiate to drive bronchiolar regeneration

Article 07 OCT 26

Dynamics of human cardiogenesis and its disruption in trisomy 21

Article 07 OCT 26

Alveolar stem cells transdifferentiate to drive bronchiolar regeneration

Article 07 OCT 26

Can your brain grow new neurons? How researchers are solving a century-old puzzle

News Feature 07 OCT 26

Brain organoids grafted into cortex-free mice will advance human brain research

News & Views 06 OCT 26

Jobs

Director General

Lead ELI ERIC, Europe’s flagship high-power laser research infrastructure, driving scientific excellence, strategy and international collaboration.

ELI Beamlines in Dolní Břežany (Czech Republic) and ELI ALPS in Szeged (Hungary)

The Extreme Light Infrastructure ERIC

Commissioning Editor

Title: Commissioning Editor - Collections Management and Acquisition Location: Beijing, Shanghai, Nanjing, China - hybrid working model Application...

Beijing, Shanghai, Nanjing, China - hybrid working model

Springer Nature Ltd

Scientific Program Director (PhD) - Genetics & Genomics Research

The Scientific Program Director provides high-level scientific, regulatory, and operational leadership in support of science research programs.

New York City, New York (US)

Icahn School of Medicine at Mount Sinai - Genetics and Genomic Sciences

Faculty Positions at the Center for Machine Learning Research (CMLR), Peking University

CMLR's goal is to advance machine learning-related research across a wide range of disciplines.

Beijing (CN)

Center for Machine Learning Research (CMLR), Peking University

Assistant or Associate Professor of Cell Biology

Department of Cell Biology at the University of Virginia School of Medicine invites applications for Assistant or Associate Professor of Cell Biology

Charlottesville, Virginia

University of Virginia

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing

How I used AI to investigate millions of historical records and surfaced a forgotten meteorite report, three lost rhinos, and unrecorded volcano eruptions.

Last week, on October 1st, 2026, the historian Benjamin Breen published a post called Using Opus 5.5 to discover a new eyewitness account of the dodo. He used AI to search digitized records from the Dutch East India Company. The Company dominated the spice trade and ran a string of ports across Asia from 1602 until it collapsed under its debts and was dissolved in 1799.

A project called GLOBALISE has turned millions of the Company’s handwritten pages into searchable text. Searching those transcriptions with AI, Breen found a 1615 ship’s journal in which sailors on Mauritius “caught many tortoises, dodos”. A new eyewitness record of the extinct dodo, sitting in plain sight for four hundred years.

I read the article and immediately thought, I can do this.

Not because I’m a historian, but because I’m a software engineer who happens to be very adept at piloting AI agents and creating agentic workflows. Breen’s post was the inspiration, and I wanted to extend the approach: I could search for many different historical mysteries in parallel, and I could chain together a few different AI models to filter results and automate a few of the cumbersome, time consuming manual steps.

But the appeal went beyond the technical challenge. I think there’s something honorable about adding to what we know. Adding even one new data point to a couple of niche fields felt like a small but worthwhile contribution to make. I thought it would be really cool if I could do that.

I powered on my small but capable home AI lab and got to work.

The Plan

I started by asking an AI “Deep Research” assistant a question: which open historical questions are most likely to be solvable with data that already exists online? It came back with thirteen candidates, ranked by how complete and accessible the data was, how much AI could help, whether anyone had already done it, and, most importantly, whether an answer could be checked against an original page. That last one matters more than it sounds. AI models can be confidently wrong, so every claim had to end at a real document: a scan of the actual handwritten letter or the actual printed newspaper, with an archive reference that anyone can look up and read for themselves. The list included animals in the Dutch East India Company archives, a giant volcanic eruption from 1808 that nobody has ever located, felt earthquakes in old Dutch newspapers, unrecorded meteorite falls, and ships that completely vanished.

Then I teamed up with Claude Code, Anthropic’s AI tool, and we invented this research pipeline together.

First came the reading material: the GLOBALISE transcriptions of the Dutch East India Company archive (4.35 million pages, from the 1600s to the 1790s), the Dutch national library’s digitized newspapers, two centuries of American newspapers, and a few ship logbooks for good measure.

If I sat down to read just the Dutch East India Company pages myself, at two minutes a page, eight hours a day, five days a week, it would take me about 70 years. And that’s before the newspapers. My homebrew AI lab got through the entire archive in a single twelve-hour overnight run.

The trouble with old documents is that nobody spelled anything the same way twice. Handwriting and old print come out of text recognition full of errors, and 17th-century Dutch spells “rhinoceros” about fifteen different ways. A plain keyword search would miss most of what I was looking for. That overnight run was my graphics card turning every passage into a mathematical fingerprint of its meaning, 5.7 million passages from the Dutch East India Company archive alone. That let me search for what a passage was about, not just which words it happened to use.

Even then, a single search could return tens of thousands of hits, far too many for a person, or even a big AI model, to read affordably. Instead, the first read went to a tiny, fast “System One” decision model called Jev, which answers only narrow questions: Is this a real animal? Is it wild? Where is it? It costs a few cents per million words. Having it read 59,000 mentions of elephants cost me about three dollars.

That’s the part of this project I’m proudest of, and it was my idea, not the AI’s. When I proposed using Jev as a filtering mechanism, Claude Code’s Fable model didn’t yet know what a System One type decision model was; I had to explain the idea before we could build the pipeline around it. In most research like this, the bottleneck is a person reading candidate passages one by one and deciding which are worth a closer look. Putting a cheap, fast judge in that seat automated the initial screening and freed me to focus on the strongest candidates and the direction of the investigation. The pages I examined closely had already survived two rounds of machine reading.

Only the passages Jev flagged moved on. Claude Haiku, a bigger model, read those few dozen closely, translated them and pulled out the dates and places. Then the Claude Code agent opened the scan of each original handwritten page to check the transcription against the original. Before calling anything new, I checked it against the catalogues the specialists themselves use.

A quick note on how AI was used in this project: I stayed actively involved in the process throughout and piloted the AI through the course of the entire project. I steered the investigation toward new questions and sources, decided which leads were worth pursuing, and recognized when I was grasping at straws and needed to move on. The systems could search and read at a scale I couldn’t, but deciding where to go next, or when to stop, still took my judgment. This was AI-assisted research, not a completely autonomous investigation.

There was one more rule, and it mattered most. Before you trust a search that finds nothing, you have to prove it can find something you already know is there. So before I went looking for anything new, the pipeline had to find Breen’s dodo, the Laki eruption of 1783, Tambora in 1815 and a dozen other known events.

It’s like testing a metal detector by burying your own wristwatch in the front yard. You know it’s there. If the detector can’t find it, you know you have a problem with your detector to address before sweeping elsewhere for real treasure. These known historical events were my buried wristwatch: a way to check that the pipeline could find something before taking its failures to find anything seriously.

The controls passed, and I began the real search. Shortly thereafter, my custom-built AI research pipeline started surfacing pages that may not have been read by anyone since the clerk who originally wrote them filed them away, stories that human eyes may not have read in hundreds of years.

The Meteorite History Forgot

The first real find of the project came from somewhere I didn’t expect: a newspaper printed in Batavia (today’s Jakarta) in the year 1812. Batavia sits on Java, the large island in what is now Indonesia that was the centre of Dutch power in Asia for nearly two centuries, and the newspaper was printed there during the few years (1811–1816) when Britain, not the Netherlands, ran the island.

On 19 December 1812 the English-language Java Government Gazette reprinted a letter from the Bombay Gazette of 26 August. An officer with a British force camped near Pandharpur, in what is now Maharashtra, wrote home:

“Captain M— is in possession of a great curiosity viz. a stone precipitated from a Thunder-cloud near the village of Cokurrgaum three days ago (the 6th August). It weighs I should think four pounds at least, is very heavy for its size, being greatly impregnated with iron, and coated with a thin black crust, as if Gunpowder had exploded around it.”

The thunder was heard “like a rustling fire of Musquetry for about half a minute.” The stone had buried itself a foot deep in open ground. And it was recovered “with some difficulty, as the Pattell [the village headman], conceiving the stone of Heavenly fabrication, had determined to say his prayers to it, with due regularity.”

Booming, a heavy iron-rich stone, a thin black crust, a crater in a field: that’s a textbook meteorite fall. And it isn’t in any catalogue. I checked six of them, from Chladni’s pioneering list of 1819 through the British Museum’s catalogues and the 1933 List of Indian Meteorites to today’s Meteoritical Bulletin. I used AI to search a hundred digitized periodicals from 1812–1817 for a follow-up, but found none. The stone isn’t in the Natural History Museum’s collection either.

Once confirmed, this meteorite fall I uncovered will represent the earliest recorded meteorite fall in Maharashtra, predating the current earliest record by 26 years.

I love how fragile the chain is. A stone falls in the Deccan. An officer writes a letter. A Bombay paper prints it. A ship carries the paper to Java, which happens to be British for five years, where an editor needs to fill a column. Copies end up in a Dutch library, which digitizes them and releases them for free. Two centuries later a GPU in my office rediscovers it. Take away any one of those links and the meteorite is forever vanished from history.

The Rhinos That Never Reached the King

In 1738, somewhere in the forests outside Batavia (today’s Jakarta), men working for the Dutch East India Company caught a live Javan rhinoceros. It was meant as a present. Every year the Company sent an embassy with gifts to the King of Kandy, the ruler of Sri Lanka’s highland kingdom, whose goodwill kept the cinnamon flowing. The king loved large, impressive animals. So Batavia’s letter to the Netherlands that year lists, among the rarities sent to the king, “yet another rhinoceros which one has had caught here in the forests, and likewise sent over.”

The rhino made it across the Indian Ocean to Colombo. It never made it up the mountains to Kandy. In the Company’s accounts, under the heading De Paardenstal, “the horse stable”, there is a line written off in 1740:

“1 rhinoceros short, died in the horse stable in the year 1738.”

In March 1739 the governor in Colombo wrote to Batavia, a little sheepishly, that the rhinoceros “died very suddenly”. To make up for it, he had added a fine Persian riding horse to the king’s gifts, “to please the King’s so often shown fiery desire for such large and stout horses.”

Batavia tried again. On 5 July 1740 the ship Loverendaal sailed from Batavia for Ceylon with two rhinoceroses aboard, a male and a female. Three days out, the officers, boatswain and gunner gathered before the ship’s bookkeeper and swore a statement: despite “all trouble and diligence” to keep the two rhinoceroses alive, the male had died that morning, “at about eight o’clock.” Ten days later they swore a second statement. The last of the two, the female, ‘t wijfje, had died too.

The crew’s statements were read back to them before the court in Colombo, and they “persisted in them without wishing the least change.” Then the governor had to tell Kandy. His instructions to the envoys at the king’s court, dated 16 September 1740, are almost touching. He had found the king some Dutch pigs, “a boar and two sows, all still young animals that will surely grow considerably, especially the little boar.” He would gladly have met the king’s request for dogs, had any been obtainable. And the envoys should let the court officials know, “so that they can answer if asked”, that the two rhinoceroses sent from Batavia on the Loverendaal had both died on the voyage, “to our particular regret.”

The king, it seems, had been expecting them.

Three years later, Dutch envoys at the court of Ramnad in southern India were asked, in a tone they found impert

In this tutorial, we implement RRSI (Regularized Recursive Self-Improvement), a method that lets an LLM agent rewrite its own harness, prompts, tools, memory, control flow, and sub-agents around a frozen model, without the harness overfitting to the tasks it evolves on. The full RRSI loop drafts edits with Claude Opus on Vertex AI and scores them inside Docker benchmarks, which is not something a free notebook can run. The part of RRSI that actually carries the paper’s idea, the rules that decide which proposed edits to keep, is plain Python, and that is what we drive directly. We install the package from the official repository, walk through its estimator, its calibrated noise band, both branches of its selection algorithm, its annealed edit budget, its deterministic leakage screen, and its edit history, and then plug a simulated agent into RRSI’s own Domain interface. Because we built the simulated environment ourselves, we know the true effect of every edit, which lets us audit RRSI’s decisions against ground truth and compare them with an unregularized search that simply keeps whatever scores highest.

Copy CodeCopiedUse a different Browser

``` import os import sys import json import math import copy import random import tempfile import textwrap import traceback import subprocess import statistics as st from pathlib import Path

RESULTS = {}

def banner(title): print("\n" + "=" * 78) print(title) print("=" * 78)

def section(name): def wrap(fn): def run(*a, **kw): banner(name) try: out = fn(*a, **kw) RESULTS[name] = out if isinstance(out, str) else "ok" return out except Exception as e: RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}" print(f"\n[!] {name} did not complete: {type(e).__name__}: {e}") traceback.print_exc(limit=3) return None return run return wrap

banner("1. Install RRSI and map the paper onto the code") subprocess.run([sys.executable, "-m", "pip", "install", "-q", "git+https://github.com/google-research/rrsi.git@be50316e1db05914068a973f322770ef08ed7ba1"], check=True)

from importlib.metadata import version from rrsi.config import RRSIConfig from rrsi.evaluate import TaskResult, EvalResult, aggregate, evaluate from rrsi.calibrate import calibrate from rrsi.selection import Candidate, cost_rule, judge, select_round from rrsi.schedule import edit_budget, budget_table from rrsi.history import History, stall_flag, exploration from rrsi.components import K, K_STR, normalize, novelty from rrsi.critic import precheck, review from rrsi.domain import Domain

print(f" rrsi {version('rrsi')} | anthropic {version('anthropic')} | Python {sys.version.split()[0]}") print("\n RRSI evolves an agent's HARNESS (prompts, tools, memory, control flow, sub-agents) around a") print(" frozen model. The full loop drafts edits with Claude Opus on Vertex AI and scores them in Docker") print(" benchmarks. The part that decides which edits to KEEP is plain Python, and that is what we drive:") for symbol, where in [ ("S_hat, C_hat Eq. (estimate)", "rrsi.evaluate.aggregate"), ("delta noise band", "rrsi.calibrate.calibrate"), ("Algorithm 2 floor + cost rule", "rrsi.selection.judge / select_round"), ("b_t Eq. (anneal)", "rrsi.schedule.edit_budget"), ("Critic leakage screen", "rrsi.critic.precheck / review"), ("L_t, g_t, B_t history, yield, prune", "rrsi.history.History"), ("sigma_t, U_t stall + exploration", "rrsi.history.stall_flag / exploration"), ("nu structural novelty", "rrsi.components.novelty"), ]: print(f" {symbol:40s} -> {where}")

CFG = RRSIConfig() print(f"\n paper defaults: T={CFG.T} rounds, k={CFG.k} trials/task, m={CFG.m} candidates/round," f" b in [{CFG.b_min},{CFG.b_max}]") print(f" beta0={CFG.beta0} beta1={CFG.beta1} w_s={CFG.w_s} w_c={CFG.w_c} w_n={CFG.w_n}" f" delta_z={CFG.delta_z}") print("\n Nothing below needs an API key, a GPU or a dataset download.") ```

We install RRSI from the google-research repository, pinned to the commit this notebook was written against, since the package is not on PyPI. Its only dependency is the Anthropic client, which the search roles use to call Claude and which we never exercise. We then print the mapping the repository itself documents between the paper’s symbols and the functions that implement them: the empirical score and cost estimate in evaluate, the noise band in calibrate, Algorithm 2 in selection, the annealed edit budget in schedule, the leakage screen in critic, and the edit history with its yield, prune, stall and exploration summaries in history. RRSIConfig holds the paper’s hyperparameters, and every function below receives it exactly as the real loop does.

Copy CodeCopiedUse a different Browser

``` @section("2. Evaluate(H): a score and a cost, and why a crash counts as zero") def estimator(): base = { "task_000": TaskResult(rewards=[1, 1], tokens=[11_800, 12_400]), "task_001": TaskResult(rewards=[1, 0], tokens=[15_100, 14_600]), "task_002": TaskResult(rewards=[0, 0], tokens=[21_000, 19_500]), } ev = aggregate("H0", 2, base) print(f" three tasks x k=2 trials -> S_hat = {ev.S:.3f} C_hat = {ev.C:,.0f} tokens/trial" f" ({ev.n_expected} trials expected, {ev.missing} missing)")

crashy = dict(base) crashy["task_002"] = TaskResult(rewards=[0.0, 0.0], tokens=[None, None], missing=2) ev_crash = aggregate("crashy", 2, crashy) dropped = {t: r for t, r in crashy.items() if not r.missing} naive = sum(sum(r.rewards) for r in dropped.values()) / sum(len(r.rewards) for r in dropped.values()) print("\n A candidate crashes on the hardest task instead of failing it:") print(f" an estimator that drops missing trials reports {naive:.3f} <- looks like a gain") print(f" RRSI's aggregate (missing = 0, full denominator) {ev_crash.S:.3f} <- no reward for crashing") print(f" ...and C_hat uses only recorded token counts: {ev_crash.C:,.0f}")

rubric = {"memo": TaskResult(rewards=[0.5, 1.0], weights=[10, 10]), "brief": TaskResult(rewards=[0.0, 0.0], weights=[90, 90])} ev_w = aggregate("rubric", 2, rubric) print("\n Weighted rewards (Harvey LAB style: weight = number of rubric criteria):") print(f" mean of per-task means = {st.mean(r.mean for r in rubric.values()):.3f}" f" vs RRSI's S_hat = {ev_w.S:.3f} (fraction of all criteria passed)") return f"crash scored {ev_crash.S:.3f} under RRSI vs {naive:.3f} if dropped"

estimator() ```

RRSI measures two numbers per harness: S, the reward averaged over every trial of every task, and C, the mean policy tokens per trial. TaskResult records one task’s trials and aggregates them. The detail worth copying into any agent evaluation is how missing trials are handled. When a candidate crashes on the hardest task, an estimator that drops the missing trials reports 0.750 and makes the crash look like an improvement. At the same time, RRSI counts each missing trial as zero reward with the full denominator and reports the same 0.500 as before, so a candidate cannot look better by destroying the trials it finds hard. Weighted rewards cover rubric-graded suites such as Harvey LAB, where S becomes the fraction of all criteria passed rather than the mean of per-task means.

Copy CodeCopiedUse a different Browser

``` FAMILIES = ["parse", "search", "edit", "test"] H0 = {"skill": {f: 0.0 for f in FAMILIES}, "memo": [], "cost": 1.0}

def make_world(seed, n_evolve, n_heldout=80): """Tasks with a family and a difficulty. Evolve ids are task_NNN, held-out ids held_NNN.""" r = random.Random(seed) world = {f"task_{i:03d}": (FAMILIES[i % 4], r.gauss(0, 1)) for i in range(n_evolve)} world.update({f"held_{i:03d}": (FAMILIES[i % 4], r.gauss(0, 1)) for i in range(n_heldout)}) return world

def p_success(h, world, task): """The frozen policy: a logistic in harness skill minus task difficulty, or 0.97 if memorised.""" if task in h["memo"]: return 0.97 family, difficulty = world[task] return 1 / (1 + math.exp(-(0.3 + h["skill"][family] - difficulty)))

def run_trials(h, world, ids, k, rng): return {t: TaskResult(rewards=[float(rng.random() < p_success(h, world, t)) for _ in range(k)], tokens=[round(12_000 * h["cost"] * math.exp(rng.gauss(0, 0.08))) for _ in range(k)]) for t in ids}

def true_score(h, world, ids): return sum(p_success(h, world, t) for t in ids) / len(ids)

def calibrated_delta(world, k, seed, repeats=3): """delta from `repeats` independent evaluations of the SAME harness H0.""" ids = [t for t in world if t.startswith("task_")] rng = random.Random(10_000 + seed) evals = [aggregate(f"base{j}", k, run_trials(H0, world, ids, k, rng)) for j in range(repeats)] return calibrate(evals, z=CFG.delta_z, reps=200), evals

@section("3. The noise band: how far one harness's score moves between two evaluations") def noise_band(): world = make_world(0, 40) ids = [t for t in world if t.startswith("task_")] rng = random.Random(1) scores = [aggregate(f"H0#{j}", 2, run_trials(H0, world, ids, 2, rng)).S for j in range(6)] print(f" H0 evaluated 6 times on 40 tasks x k=2: {[round(s, 3) for s in scores]}") print(f" true expected score {true_score(H0, world, ids):.3f}; spread of the estimates" f" {max(scores) - min(scores):.3f}. Nothing about the harness changed.")

print(f"\n calibrate() turns repeated evaluations into delta = z * sd(null dS), z = {CFG.delta_z}:") print(f" {'evaluator':>12s} {'trials':>7s} {'delta':>8s} method") deltas = {} for n, k in [(40, 2), (100, 4), (400, 8)]: cal, _ = calibrated_delta(make_world(0, n), k, seed=0) deltas[f"{n}x{k}"] = cal["delta"] print(f" {f'{n} x k={k}':>12s} {n * k:>7d} {cal['delta']:>8.4f} {cal['method']}") print("\n The paper's calibrated bands are 0.017 (coding), 0.004 (workspace) and 0.020 (engineering).") print(" A gain smaller than delta is indistinguishable from re-running the same harness, and") print(" Algorithm 2 treats it that way. Remember the first row: it becomes the lesson of step 11.") return "delta " + ", ".join(f"{k}={v:.3f}" for k, v in deltas.items())

noise_band() ```

Before any rule can separate a real gain from luck, it needs to know how far one harness’s score moves on its own. We build a small simulated agent, whose success on each task is a logistic function of harness skill minus task difficulty, and evaluate the unchanged starting harness six times on forty tasks with two trials each: the scores spread by 0.113 although nothing changed. calibrate turns repeated evaluations of the same harness into delta, twice the standard deviation of the difference between two runs. With 80 trials delta is about 0.108; with 3,200 trials it falls to about 0.013, in the range the paper reports for its instances (0.004 to 0.020). For selection purposes, any gain smaller than delta is indistinguishable from re-running the same harness.

Copy CodeCopiedUse a different Browser

``` def ev_at(S, C, job, n=200): """An EvalResult with exactly score S (in steps of 1/n) and C tokens per trial.""" hits = round(S * n) return aggregate(job, 1, {f"task_{i:03d}": TaskResult(rewards=[1.0 if i < hits else 0.0], tokens=[C]) for i in range(n)})

INCUMBENT = ev_at(0.630, 10_000, "incumbent") S_STAR, DELTA = 0.640, 0.020

@section("4. Algorithm 2, branch by branch: the floor, the cost rule and the noise ban

[This is a guest post by Thomas Hales. This blog post was initially written in a different file format and converted using AI. — T.]

Mathematicians have been weighing in on what they value about mathematics. For me, what matters is the consistency of math and its unparalleled reliability in support of science and civilization.

Formalization of Math

A formal proof is a mathematical proof that has been exhaustively checked at the level of the foundations of math and the fundamental rules of logic. In theory, this might be done by hand, but because of the number of steps involved, this is generally done by computer, using software that is designed for the task.

Examples of theorems that have been formalized include the four-color theorem, the Feit-Thompson (odd-order) theorem, the Kepler conjecture, sphere eversion, the sphere packing problem in 8 and 24 dimensions, Navier-Stokes forced blowup, and Fermat’s Last Theorem. The last three formalization projects have been completed this year and have brought widespread awareness of the potential of formalization.

Software systems for formalization are variously called proof assistants, theorem provers, or interactive theorem provers. For the purpose of this post, these terms are used interchangeably. Many proof assistants have been developed over the years: Automath, HOL Light, Isabelle, Coq (renamed Rocq last year), Metamath, Mizar, and Lean. Freek Wiedijk edited a book “The Seventeen Provers of the World” that compares some of these proof assistants, giving a proof of the irrationality of the square root of 2 in each of them. Among mathematicians, the Lean theorem prover is the most popular, and this post will focus on Lean.

Lean was developed and introduced by Leo de Moura in 2013, while at Microsoft. To our great benefit, de Moura persuaded Microsoft to make the software open-source. Kevin Hartnett’s book on the history of Lean, “The Proof in the Code”, states that Jeremy Avigad (the director of Carnegie Mellon’s new NSF institute ICARM) was the first user of Lean. He ran a Lean seminar in 2015 that I attended. In 2017, one of Jeremy’s graduate students, Mario Carneiro, working with Johannes Hölzl, took existing parts of Lean’s core library and started a separate Lean mathematical library, called mathlib. This library of formalized mathematics is now massive, containing nearly 300,000 theorems, over 100,000 definitions, 2.5 million lines of code, with over 700 contributors. Any definition or theorem in mathlib can be used to prove further theorems. For example, if a proof uses the Cauchy-Schwarz inequality, the result can be cited from the library rather than reproving it.

Autoformalization is a practical reality

In the past, researchers had to transcribe paper proofs into formal proofs by human labor. For example, the formal proof of the Kepler conjecture on sphere packings in three dimensions took about 20 human work-years to complete and consists of about 500,000 lines of proof scripts. For years, it has been a dream for many of us working in formalization to find ways to bring increased automation to the process. Autoformalization is the realization of that dream. Autoformalization is the formalization of mathematics by AI. AI reads the paper (say a pdf or tex file) and outputs the formal proof in Lean or some other proof assistant.

Autoformalization has become a practical reality in 2026. Starting in late spring and summer of 2025, researchers were becoming increasingly bullish about autoformalization. Here are some milestones.

Sep 2025, Math Inc. produced a quasi-autoformalization of the prime number theorem. The process was merely “quasi”, because humans had to intervene to give further guidance whenever the AI got stuck.

Jan 2026, J. Urban posted an arXiv preprint “130k lines of formal topology in two weeks” that gave the autoformalization of large parts of Munkres’s topology textbook in a proof assistant based on set theory.

Mar 2026. Approximately a week after announcing the completed formalization in 8 dimensions, Math Inc. announced an autoformalization of the sphere-packing problem in 24 dimensions, following the proof by Viazovska and her collaborators. This project generated about 500K SLOC (source lines of code) that golfing (or code pruning) later reduced to about 200K lines.

May 2026, a group at Meta/Facebook Research autoformalized a large part of 26 mathematical textbooks in a project called ATLAS.

From there, numerous theorems have been autoformalized. Particularly noteworthy is the autoformalization of Fermat’s Last Theorem, announced by Anthropic on September 4. This project generated 13 million lines of Lean in 11 days. The announcement of Navier-Stokes blowup with forcing on September 8 by OpenAI was accompanied by an autoformalization of the theorem in Lean.

Looking forward, Urban stated in January, “We believe that (auto)formalization may become quite easy and ubiquitous in 2026, regardless of which proof assistant is used.” Autoformalization projects have been completed in various proof assistants using various LLMs, but we focus on Lean. “For [Jesse] Han, it represents even more: the beginning of a revolutionary transformation in mathematics, where extremely large-scale formalizations are commonplace” (IEEE Spectrum). Jared Lichtman announced the launch of MAP (the Mathematics Autoformalization Project) on Sept 8, 2026, which aims to translate “all known math into formal code”. He asks us to imagine the next one trillion lines of code.

Is Lean reliable?

Type theory.

Lean is based on type theory; in fact, a particular dialect of type theory called CIC, the calculus of inductive constructions. This post is not intended to be a tutorial on type theory, and I will be brief. Russell’s famous paradox in 1901 (the set of all sets that are not an element of themselves….) led to a crisis in the foundations of math. Two solutions were proposed later that decade. (1) Zermelo’s axioms of set theory that disallow the creation of unsafe sets; (2) type theory that makes it a syntax error to create Russell-paradox-like entities. Type theory was introduced by Russell himself in 1903 in his book Principles of Mathematics, and it became part of the foundational system of Russell and Whitehead’s Principia.

For mathematicians who are accustomed to set theory, B. Werner’s paper (1997) “Sets in Types, Types in Sets” gives some reassurance that whatever they have done in set theory can be translated into type theory, and whatever gets done in type theory can be translated back into set theory. More precisely, the paper shows that ZFC set theory can be encoded into CIC, and that a particular dialect of CIC can be encoded back into ZFC (augmented with a hierarchy of inaccessible cardinals).

At the risk of simplifying matters to a ridiculous degree, we might say that “types are like disjoint sets”; each element in type theory “is an element of” exactly one type. The type of the natural number 2 is the natural number type; the type of e, the base of the natural logarithm, is the real number type, and so forth. The type of natural numbers is disjoint from the type of real numbers, and an explicit coercion (sending 2 to 2.0) is constructed from the type of natural numbers to the type of real numbers. When I give talks, I sometimes draw a picture of sets as a Venn diagram with nonempty intersections and a picture of types as bricks stacked against one another without intersection.

Lean’s design

One part of the Lean system is a general-purpose programming language (appropriately called the Lean programming language). Ordinary computer programs, such as a program to sort a list, can be written in this language, then compiled and run. The Lean system also provides a mathematical language, in which definitions can be written, theorems can be stated, and proof scripts can be written. The programming language and mathematical language are not independent entities. Rather, it is a single language that does both. Program code can be mixed with theorems about the correctness of the algorithms; mathematical proofs can be generated using programs. The proof scripts in Lean are parsed and go through a process called elaboration (a sort of compilation process for mathematics), then the proofs are checked by the Lean kernel. It is the kernel’s responsibility to check and verify the output of elaboration.

The Lean kernel is several thousand lines of C++ code. The kernel is carefully engineered but extremely complex. We mentioned mathlib above, which consists of about 2.5M SLOC, written in the Lean language. The library has been elaborated, then checked by the kernel. If there is an unconditional false proof anywhere in these 2.5 million lines of code, it is the fault of the kernel or runtime for failing to reject a false proof. Any defect in the underlying type theory is a serious kernel defect, if it is implemented in code.

Lean proofs should never be believed until they have been checked by the kernel. Additionally, a proof in Lean should not be accepted until a human audit is performed to ensure statement fidelity. Is the verified theorem what we think it is? Do the definitions in Lean correspond to what we think they should be? This task is generally massively easier than checking the proof itself. For instance, for Navier-Stokes, a human should check that the statement in Lean corresponds with Fefferman’s statement of the Millennium Prize Problem, and specifically that concepts such as the field of real numbers, partial derivatives, and measure are correctly defined in Lean. The comparator tool in Lean assists with this task. The tool can also perform additional checks, such as inspection for possible unauthorized axioms.

Summer of Soundness Bugs

A soundness bug is a bug in the kernel that allows a proof of “False”, and consequently a proof of any proposition. A soundness bug is the most disastrous of any kind of bug in a proof assistant and should set off an alarm for mathematicians who care deeply about the reliability of mathematics. Occasionally, soundness bugs are found in various proof assistants. In 2003, I found a soundness bug in the proof assistant HOL Light, which was then considered to have the most reliable of all kernels. That kernel is tiny, consisting of just a few hundred lines of computer code. For me, it is a badge of honor that I found this soundness bug, which was the first soundness bug that had been found in that proof assistant since 1996. (See HOL Light change log, July 2003.)

Lean 4 was released in September 2023. Prior to release, two soundness bugs were found and corrected. In May 2025, another soundness bug was reported, caused by overflow. All hell broke loose in the spring and summer of 2026, which is now being called the “Summer of Soundness Bugs”. Several soundness bugs in Lean were uncovered in July and August. The summer madness affected various proof assistants, but my focus is Lean. One Lean bug led to an illicit disproof of the Collatz conjecture. I learned of the bug this summer when it produced a short illicit proof of the Kepler conjecture in Lean. All these bugs were quickly repaired, and mathlib has been verified by the repaired kernel. An analysis of the soundness bugs is found in de Moura’s postmortem.

The “summer of Lean soundness bugs” might sound like a disaster, but closer investigation shows that the detection of these soundness bugs is a positive development. The summer bugs were detected by frontier model AI in the hands of security researchers interested in reliable kernels, not by black-hat hackers. The Collatz bug was found by Ramana Kumar, a co-author of “CakeML: a verified implementation of ML”, which creates an end-to-end verified ML (the functional programming language).

智能体已支持多模态界面理解与交互；排错与重试机制显著提升任务完成率；人机责任边界（如金融操作）尚未标准化。

适合AI工程师、智能体架构师、API平台开发者阅读。

翻译| 林绮蓓

编辑| 蔡芳芳

计算机操作（Computer use）的结果明明很容易验证，为什么这一技术的进展如此缓慢？AI 什么时候才能真的学会操作电脑？

今年 6 月，知名播客主持人 Dwarkesh Patel 提出的这个质疑，在行业内引发了不少争议。

三个月后，OpenAI 在 9 月 29 日的开发者日展示了 Dot、计算机操作和一系列新的开发者接口。每个 Dot 都可以使用一台独立的云端 Linux 电脑，既能打开浏览器，也能运行桌面应用；用户可以把原本需要自己在网页或软件里逐步完成的任务交给它处理。产品演示之后，这个问题反而更值得进一步细问：计算机操作现在究竟能做到哪一步，支撑智能体完成任务的能力又发生了什么变化？

开发者日结束后，Latent Space 的《The AI Engineer Podcast》与 OpenAI 的 CUA 团队和 API 平台负责人进行了深入探讨。播客首先采访了 Sky 联合创始人、现 OpenAI 计算机操作产品与工程负责人 Ari Weinstein。Ari 认为，过去一年最关键的进步，是智能体遇到障碍后更会排错和重试。它也不再只能看着截图逐步点击：页面结构、无障碍信息和自己编写的代码，都可以成为它操作软件的方式。但当智能体能够访问网站、处理付款，人该在哪些环节把关，仍是产品必须回答的问题。

OpenAI API 产品负责人 Nikunj Handa 在访谈中分享了为智能体开发在 API 层做的各种实践：从让模型在工具执行时继续推理，到快速决策、性能优化和上下文管理。当谈论到 Decisions API 时 ，Nikunj Handa 承认，这是受到了 Jev 启发而启动的，起初并不在研发计划之中。项目启动约一周，团队就已完成可运行的原型。他们没有重新训练模型，而是沿用 Luna 的权重，把输出约束成结构化结果，并专门优化推理系统，让首次决策返回时间尽可能短，多个问题还能并行批量处理；它更像一个极快的分类和决策层，能用于客服工单分类、评估打分和计算机操作，也能和 GPT Live 配合，让智能体在需要快速判断时更灵敏。

从“AI 为什么还不擅长操作电脑”的质疑出发，两场访谈逐渐把镜头拉近：智能体怎样发现自己做错了，怎样决定下一步，又怎样在漫长而琐碎的实际任务中持续工作。开发者日的演示给出的是结果，此次访谈则从产品和平台的不同视角出发，分享了他们在计算机操作方面的实践。

太长不看版：

主持人 Vibhu： Dot 为每个智能体提供独立的云端电脑。用户应该如何探索它的能力？

Ari Weinstein： Dot 可以在云端 Linux 电脑上使用浏览器和桌面应用。一个实用的起点是梳理自己日常耗时的电脑任务，尝试将其中适合的任务委托给智能体。

主持人 Swyx： 有观点认为，计算机操作在过去两年几乎没有进步。实际情况如何？

Ari Weinstein： 过去，智能体通常能够启动任务，却容易在遇到问题后停滞。如今，它更善于排查错误、调整方法并重新尝试。这是近一年最显著的变化之一。

主持人 Vibhu： 能力的提升主要来自模型，还是智能体运行框架？

Ari Weinstein： 两者都在发挥作用。智能体现在可以结合截图、无障碍信息、页面结构和 Playwright 操作软件，也可以编写代码，一次执行多个步骤；模型本身的能力和速度同样在提高。

主持人 Vibhu： 计算机操作接下来的主要瓶颈是什么？

Ari Weinstein： 瓶颈分布在模型、推理、运行框架和信息呈现等多个环节。随着智能体执行速度提高，等待网站加载等操作本身的耗时，也变得越来越明显。

主持人 Swyx： 开发者通过 Agents API 使用计算机操作能力时，需要注意什么？

Ari Weinstein： 应根据任务限制智能体可访问的网站和应用，并在付款等可能产生重要后果的操作前征求用户同意。可靠性和恰当的安全检查，是建立用户信任的基础。

主持人 Vibhu： 计算机操作将怎样改变软件开发流程？

Ari Weinstein： 智能体可以实际打开并测试自己开发的软件。这样，编写代码与验证结果能够形成闭环，减少开发完成后完全由人接手测试的情况。

主持人 Vibhu： 这次发布的 API 有哪些值得关注的变化？

Nikunj Handa： 异步函数调用允许模型在工具运行期间继续执行，之后再获取结果；中途引导则允许开发者在模型运行过程中加入消息。这些能力有助于处理工具调用频繁、执行时间较长的任务。

主持人 Swyx： OpenAI 是如何开始开发 Decisions API 的？

Nikunj Handa： Jev 的推出引起了用户和内部团队的关注，此前 Decisions API 并不在开发计划中。团队先验证原型是否可行，再着手优化决策返回的速度。

主持人 Vibhu： Decisions API 适合解决什么问题？

Nikunj Handa： 目前明确的用途是快速分类，例如处理客服工单。它也可能用于某些需要迅速选择下一步动作的任务，但快速决策与完成复杂、长周期任务所需的能力不同。

主持人 Swyx： 如果同样使用 Luna，Decisions API 与普通的结构化输出有何区别？

Nikunj Handa： 首个版本没有重新训练模型，而是基于现有的 Luna 权重，对输出施加约束、并行处理多个问题，并专门优化首次决策返回的时间。

主持人 Swyx： 长时间运行的智能体如何应对上下文上限？

Nikunj Handa： 开发者可以设置阈值，让 Responses API 自动压缩上下文；也可以调用 /compact，自行控制压缩时机。

当 AI 有了自己的电脑，哪些事可以交给它？

主持人 Vibhu： 今天是 OpenAI DevDay，我们在现场录这期特别访谈。

主持人 Swyx： 我们是你们直播结束后接受的第一场播客访谈。

主持人 Vibhu：今天请到的是 Ari，他负责计算机操作（Computer Use）智能体的产品和工程团队。在深入聊这项技术之前，能不能先回顾一下今天发布了哪些东西？

Ari Weinstein： 我们刚从主题演讲现场出来，今天有几项计算机操作方面的发布值得关注。

首先是 Dot，这是一个新的个人助理产品，里面有一些很令人期待的计算机操作功能。

还有 GPT-6.1 Sol，这是一个非常出色的新模型，我认为它尤其适合计算机操作，因为它在成本和速度上都有优势。我们好像公布过它的成本是 Astra 的五分之一；如果专门看计算机操作，成本只有七分之一，真的很了不起。

Agents API 现在也加入了计算机操作能力，开发者可以使用与 Codex、ChatGPT 中相同的实现来构建产品。现场还演示了现有功能，比如应用快照（app shots），可以把你正在电脑上处理的内容迅速带入 Codex 和 ChatGPT。

还有 Mac 上的原生计算机操作：Roman 让它自动给自己的应用截图，而它操作应用时，他仍然可以在电脑上做其他事情。所以，这场主题演讲确实很精彩。

主持人 Swyx： 更不用说还有 Decisions API。我先直接问一下：这些背后都是同一个模型，还是用同一套数据集蒸馏出来的不同模型？具体来说，计算机操作用的是 Decisions API，还是两者相对独立？

Ari Weinstein： Decisions API 很有意思，它有几项新能力：可以并行推断，不进行思考推理，用的模型也比我们用于计算机操作的模型更小。这让它速度很快，但处理执行周期较长、比较复杂的任务时，能力也稍弱一些。怎样把这些方法结合起来，仍然是一个有待研究的问题。我很期待看到大家用 Decisions API 做出什么。

主持人 Vibhu： 有意思的一点是，现在 Dot 都配有自己的个人电脑。

Ari Weinstein： 是的。

主持人 Vibhu： 所以它们似乎能更持久地保留工作环境了。你已经用了一段时间，大家应该怎样探索它的能力边界？应该朝什么方向尝试？我自己现在经常让它处理客服问题。 比如，“这个东西出错了，我不想登录，也不想做身份验证”，你去找到相关信息，把问题解决就行。再往前一步，大家还应该尝试什么？

Ari Weinstein： Dot 是一个很有意思的产品，因为每个 Dot 都可以使用自己在云端的一台 Linux 虚拟电脑，这与我们的其他产品不同。过去，我们提供的通常是云端浏览器，或者让智能体访问你自己的电脑；现在，你有的是云端一整台属于自己的 Linux 电脑。它既能运行完整的桌面应用，也能使用浏览器。我认为计算机操作如此强大、如此令人期待，就是因为它让智能体能够做任何一个人可以做的事。世界上所有软件原本都是为人设计的，如今智能体也能使用这些软件，你就可以把工作委托给它。所以，凡是你会在电脑上做的事情，都可以让 Dot 去做。具体哪些最有用，确实取决于最终用户是谁，以及什么事情对他的生活有价值。

我会建议，先想想你平时把时间花在哪些事情上，再想想能不能把这些事情交给智能体。凡是你会在电脑上做的事情，都可以让 Dot 去做。具体哪些最有用，取决于最终用户是谁，以及什么事情对他的生活有价值。我建议先想想，自己平时把时间花在哪些事情上，再看看能不能把这些事情交给智能体。

主持人 Swyx： 对，比如订机票、购物，说实话，甚至玩个游戏之类的事情也可以，对吧？

Ari Weinstein： 完全可以。我最近为了吃得健康一些，订了一项配餐服务。我很喜欢这家服务，因为它允许我非常细致地定制每一餐，比如“我要多少克鸡肉、多少克米饭”。但操作实在太复杂，我下一次订单就花了两个小时。后来发现，可以让计算机操作帮我下单，它十五分钟就完成了。用 GPT-6.1 Sol，它完成这件事的速度是我的八倍，同时替我省下了两个小时。我觉得这类任务特别能体现它的价值。

主持人 Swyx： 作为创作者，我马上就能说出自己的首要用途：把 YouTube 上的操作自动化。YouTube 很多功能都不开放 API，你只能把它放进虚拟机里，让智能体自己跑。比如 A/B 测试，或者发社区帖子，这些都没有 API，因为他们讨厌开发者。

Ari Weinstein： 我也听开发者体验团队的人这么说过。他们经常用它处理 YouTube 上的事情，确实非常好用。

计算机操作迈过新阶段：更会看懂界面，也更能排错提速

主持人 Swyx： 我想稍微问得尖锐一点。我们有朋友做一档很有影响力的 AI 播客，他们有一个出名的观点：计算机操作在过去两年里毫无进步。这个说法很有意思，而你应该是全世界最有资格谈这个话题的人之一。究竟发生了哪些进展？

Ari Weinstein： 我记得他们是几个月前这么说的，希望他们现在已经改变看法，因为计算机操作与以前相比，已经有了一百八十度的变化。

主持人 Swyx： 你几乎整个职业生涯都在做某种形式的计算机自动化，从在 Apple 做快捷指令（Shortcuts），到 Sky，再到加入 OpenAI。能不能讲讲贯穿这段经历的主线？是什么在驱动你？以前有哪些事情做不到，又有哪些里程碑？

主持人 Vibhu： 我再补充一个问题：从上周的 Codex 计算机操作到今天，最主要的变化是什么？是模型，是 Dots，还是智能体运行框架（harness）？除了整段历史，也想请你讲清楚，今天的发布究竟改变了什么。

Ari Weinstein： 我一直对自动化、对帮助人们把任务自动化很有热情，因为这样能节省生活中的时间，把精力放在对自己更重要的事情上，而不必细致地操作电脑。这也是我们做那些产品的原因。我之前在 Apple 工作，后来创办了 Sky，最终加入 OpenAI，这段经历让我很兴奋。

主持人 Swyx： 感觉就像你一直在想办法绕过 Apple 的限制，直到 Apple 说：“好吧，我们干脆把你招进来，让你从内部做这些事。”是这样吧？

Ari Weinstein： 能在那里工作，确实是一段很棒的经历。回头看 Sky，很有意思的是，我们当时也在做计算机操作，但那时模型的能力弱得多。仅仅过去一年，模型的计算机操作能力就已经变得非常强。我看到最大的变化是，以前它们可以比较可靠地开始一项任务，但做着做着就会遇到问题；现在，它们非常擅长排查错误、重新尝试，审视哪些方法有效、哪些无效。

计算机操作这个领域本身也在发展，我们采用了更多技术。现在的计算机操作经常会写代码。如果你在 Codex 里手动展开工具调用，就会看到，它并不只是一次执行一个动作，而是在编写 JavaScript 代码，交给电脑执行，有时一次就能完成很多动作，速度和能力都有了提升。我们更多地采用无障碍接口等多模态交互方式，模型可能使用截图，也可能使用无障碍接口，或者 Playwright。它可以根据手头的任务，选择许多不同的机制。模型自身的加速也非常惊人。

至于今天有什么不同，我觉得我们一直在改进计算机操作，因此单独看一天的变化，可能还不如看过去一个月或两个月那么有意义。不过，Dot 中的计算机操作，以及今天发布的新模型，我都认为非常值得期待。

主持人 Vibhu： 主题演讲里，Tejal 提到计算机操作的速度提升了七倍，在几个基准测试上也好了很多。你们是怎么衡量的？就像你说的，计算机操作是在不断进步的。这些进步来自智能体运行框架、模型，还是后训练？新模型又带来了哪些变化？

Ari Weinstein： 我们实际上有很多种衡量方式，其中一些会测试智能体运行框架的不同组合和配置。这里的情况有点复杂，因为正式上线的产品会有更多安全检查，而且会根据当前任务的需要采用不同配置。所以，衡量的方法有很多，但无论采用哪种方式，我们看到的提升都相当一致。这些提升有时来自运行框架，有时来自模型。有一个结果让我印象很深：与 Astra 相比，GPT-6.1 在计算机操作上的成本效益提升，甚至超过它本身基础成本的降幅。看到这一点真的很棒。

主持人 Swyx： 直播中有张图我特别喜欢，展示了你们不断改善这条曲线的帕累托前沿。你们也谈了不少如何配合运行框架一起改进。能不能举几个让你们恍然大悟的例子？无论是模型推动框架改进，还是框架推动模型改进，都可以。

Ari Weinstein：引入更多模态确实很有效。具体来说，过去很多计算机操作产品必须花大量时间滚动页面：先截一张图，试着操作一下，再发现“得往下滚，看看下一页结果”，于是再截图、再尝试操作，然后又往下滚。如今借助无障碍接口、直接访问文档对象模型（DOM）等方式，语言模型可以看到整个页面，或者整个应用，还可以写出一次完成多个步骤的代码。这些大概是最明显的突破。此外还有很多小发现，没那么引人注目，但我们确实发现，不少速度提升来自一个个细小问题，需要深入检查、逐一解决。

主持人 Swyx： 背后是大量艰苦的工程工作。比如应用快照，很多人还不太理解它的区别。截取应用快照时，Codex 会显示一个很好看的画面，但他们未必知道，你实际上可以操作其中每个按钮，每一处文字也都已经以很合适的形式呈现给模型了。

Ari Weinstein： 没错。其实挺有意思。如果你想深入研究，可以打开 Codex，按下两个 Command 键截取应用快照。这样就能把你正在使用的应用内容带入 Codex 或 ChatGPT 的聊天里。然后点击附件，再点右上角那个很小的按钮，就能看到原始文本和原始的无障碍结构表示。我们为这件事投入了很多工作。

主持人 Swyx： 就是把一切全都导出来。

Ari Weinstein：不仅要导出来，还要高效、节省词元（token），这里面有不少技巧。原本为有无障碍需求、需要使用屏幕阅读器的人发明的技术，既能帮助他们使用电脑，也非常有助于大语言模型使用电脑。能参与这样的工作，很有意思。

主持人 Vibhu： 补充一点背景：很多人不了解应用快照，甚至不知道有这个功能。按两下 Command，会引入一个看起来像截图的东西。你可能会想：“为什么只是打开一张截图，把它丢进来？”其实它会把所有元数据、所有代码以及其他内容一起带进来。

Ari Weinstein： 没错。比如你给一个包含链接的网页截屏，截图里不会包含链接的目标地址。或者你给日历截屏，日程标题可能被截断了。但应用快照会把完整的上下文交给语言模型，让它能够做更多事情。

主持人 Vibhu： 关于计算机操作智能体，我想问一个更宏观的问题。你刚才说的截图、滚动页面、再截图，是过去的状态。今天它们已经可以自动完成很多事情了，瓶颈在哪里？是模型还是运行框架？你觉得两年后会发展成什么样？它能连续运行几个小时吗？我们怎样才能走到那一步？你对计算机操作的未来有什么预测？

Ari Weinstein： 我觉得团队过去几个月做到的一件很不可思议的事情是，现在计算机操作在大多数情况下，完成任务的速度可能已经超过普通人了。接下来要突破的，是让它的表现真正超越人类：使用软件的速度达到，甚至超过像我们这样熟练的电脑用户。那会带来很大的影响，因为我们能做出体验更加实时的产品。同时，使用计算机操作的门槛，或者说开始使用它所需要克服的阻力，也会降低。对于某些我们习惯亲手做的事情，大家会开始默认交给智能体，从而节省大量时间。

要实现这一点，还有许多小问题和瓶颈需要解决。有些在模型端，有些在推理端，有些在智能体运行框架里，还有些涉及信息的表示方式。随着计算机操作越来越快，我们越来越容易受到操作本身速度的限制。比如在计算机操作任务的基准测试中，有一部分不可忽视的耗时，是在等待网站加载。假设你在 doordash.com 上自动完成一项任务，其中很大一部分时间其实是在等 doordash.com 自己加载。

主持人 Swyx： 然后你就写一个等待，再执行这个等待。

Ari Weinstein：页面终于加载完成，到触发大语言模型执行下一个动作，这之间的延迟要尽可能短。这非常重要，本身实际上就涉及一套统计方法。

主持人 Swyx：也许可以用事件驱动的方式来做？

Ari Weinstein： 条件允许的话，当然希望采用事件驱动。

主持人 Swyx： JavaScript 有一些加载事件。

Ari Weinstein：JavaScript，或者说浏览器，针对网页导航有加载事件，但有些其他类型的事件确实没办法用事件驱动处理。所以这里面非常复杂。

主持人 Vibhu： 我想到的一个例子就是跟客服聊天。对方可能三十秒回复，也可能三分钟才回复。

主持人 Swyx： 我已经用 Codex 跟很多机器人打过交道了，挺好用。但我有时也会想，对面知不知道自己是在跟机器人聊天？因为我回复的句子太完整了，连大小写都很规范，编号、参考编号之类的信息也都给得完整，明显表现得太好了。不过我不在乎，我只是想解决自己的客服问题。

主持人 Vibhu： 我会在提示里告诉它：“别表现得像机器人，要像一个已经很不耐烦的人。”我还会告诉它：“等回复的时候，调用子智能体，研究一下有没有更好的办法找到我们需要的信息。” 就是稍微加一点人的干预。

Ari Weinstein： 太好了。而且我觉得，多半对面本来也是机器人，现在就成了机器人互相聊天。

主持人 Swyx：是的。我们现在已经走到了计算机操作的一个里程碑：三四年前，我们还害怕把大语言模型连到互联网、连到自己的设备上，现在我已经让它帮我配置域名系统（DNS）、替我付账单了。真的是几万美元的事情，我就直接交给计算机操作，放手让它去做，心想最坏能发生什么呢？你们自己也要使用自己的产品。

现在既然已经通过 API 发布了这项能力，有没有什么陷阱或者建议想告诉开发者？因为他们马上就要亲身经历这些事情了。

Ari Weinstein： 首先，我真的很高兴我们把计算机操作带进了 Agents API。很多开发者开发的应用，都希望能够与第三方网站和服务交互，而计算机操作具有这种通用性，什么都能操作。现在，开发者一下子就能使用与我们自己相同的计算机操作实现来构建产品。如果你想自己做计算机操作的智能体运行框架，这方面当然还有很多值得做的工作，但难度很大。而且我们的模型是在自己的计算机操作框架上训练的，所以使用处于模型训练分布内的这套框架，可能在速度、成本和准确性上都有优势。

回应你刚才的话，我认为大家都还在逐渐适应这项技术、建立对它的信任，只不过我们当中有些人可能比世界上很多人走得更快一些。因此，我们有责任通过时间慢慢建立这种信任：确保做出的东西可靠，设置恰当的安全检查，在付款等可能产生重要后果的操作之前征求用户同意；或者根据具体应用的情况，确保它只能访问完成任务真正需要的网站和应用。这些都值得认真考虑。我非常鼓励大家试试新的 Agents API，在上面做出各种有意思的东西，也欢迎大家把使用后的反馈告诉我们。

主持人 Vibhu： 你有没有看到它对开发工作流程带来的变化？比如现在可以在 Slack 里使用 Dots，也有人用语音来构建东西。Roman 演示的例子是让它修改应用，并在过程中不断发来截图。大家实际怎样把计算机操作用在编程流程里？有没有值得借鉴的做法？

Ari Weinstein： 我最喜欢的用途之一，也是我们在实际使用中经常看到的，就是让智能体通过计算机操作，真正测试自己开发出来的软件。这件事的影响比听起来大得多。过去，你让 Codex 做点东西，它做好以后还得由你测试，于是你就成了智能体的质量保证人员。有了计算机操作，软件开发的整个流程就能贯通：智能体既能开发软件，也能测试软件。我很喜欢这样做东西，让智能体自己测试，到交给我时，软件已经能正常工作了。

对我来说还有额外的乐趣，因为有时我开发的就是计算机操作本身，于是就出现了一个计算机操作智能体，操作着我开发的另一个计算机操作智能体，再由后者去操作别的东西。我认为这是一类非常有价值的用途。

主持人 Swyx：我做了一个视觉交互测试技能，能发现很多只看代码发现不了的设计问题。它也特别适合复刻应用：如果你正在用一个很烂的软件即服务（SaaS）产品，想把它替换掉，就一屏一屏地照着做。计算机操作可以把整个应用都操作一遍，截图、记录，再让 Codex 把一切复刻出来。总之，谢谢你们做出的这些进展，我们的时间到了，不过这肯定不会是最后一次对话。

Ari Weinstein： 今天聊得很开心，谢谢你们邀请我。

工具调用不再干等：OpenAI API 如何让模型边执行、边推理

主持人 Vibhu： Nikunj，很高兴请到你。你们在 API 方面发布了很多东西，刚才和 Ari 也谈到了，现在已经可以用计算机操作智能体来开发了。你想重点介绍哪些 API 变化？也请简单介绍一下自己和负责的工作。

Nikunj Handa：我叫 Nikunj，负责 API 团队的产品工作，在这里大约三年了，一直参与模型发布。感觉在 OpenAI 的这段时间里，这件事就没停过。每推出一个新模型，我们都会与后训练团队、研究团队紧密合作，弄清楚它有哪些新能力，再通过 API 开放出来。

GPT-6 这次发布的新能力，首先是异步函数调用。Codex、Dots 等产品中的很多工具调用都很耗时，现在工具运行的时候，不必暂停模型的执行。你可以先发起工具调用，让模型继续运行、继续推理，过一会儿再回来查看结果。我们还推出了轮次中途引导，可以在模型推理的过程中注入消息，这样工具调用一结束，就能把相关指令放进去。

主持人 Swyx： 这也有一部分属于模型对齐能力，对吧？这种能力本身也得通过训练获得。

主持人 Vibhu： 我感觉应用里以前就有了：模型推理时，你可以中途引导它。以前效果不算最好，现在已经好多了，很期待看看这个版本的表现。

Nikunj Handa： 是的。我们在 API 方面的主要目标，就是等这些能力已经在智能体运行框架中训练到位，再放进 API，所以会等到它足够好。很多能力由 WebSockets 支撑，我们应该是在几个月前发布的。WebSockets 让应用能够与模型进行双向通信。我这里说的不是 GPT Live，而是 GPT-6。异步工具调用、异步推理、注入消息，这些都能做。开发这个 API 很有意思，我很享受这个过程。

主持人 Swyx： 这就是我们为什么是一档工程播客，因为我们会聊 WebSockets。它和 UltraFast 也很搭，对吧？我记得这也是 UltraFast 第一次通过 API 提供。也就是说，前沿水平的模型可以以理论上最快的速度运行。

Nikunj Handa：在谈 API 之前，我觉得 UltraFast 最有意思的部分，是看推理团队围绕 Astra 不断拿出成果。他们一直运行着 Codex 智能体，试着再多挤出一点性能。至少有几个月，大量工作都集中在提高效率、降低成本上，我们因此能把 Luna 的价格降低大约 80%。其中很大一部分，就是他们落地的这些推理优化带来的。后来他们换了方向，开始研究怎样让它跑得尽可能快。看到 UltraFast 能把 Astra 这样的模型加速到这个程度，很让人振奋。

我们最初推出 WebSockets，是为了 GPT-5.3 Codex Spark。显然，工具调用必不可少，而 WebSockets 能大幅减少与工具来回通信的开销，所以特别有帮助。

主持人 Swyx：每次看到用量额度的界面都挺好玩：一边是我的常规额度重置，另一边是那个我从来没用过的 Spark 额度，反正我想用的时候，它就在那儿。

Nikunj Handa：我想它终于被移除了。

主持人 Swyx：确实没了，你们在慢慢淘汰那些老家伙。

受 Jev 启发？却不止于结构化输出：OpenAI 的 Decisions API 如何追求更快决策

主持人 Swyx： 5.3 Spark 明确说过用了 Cerebras。你们现在既不确认，也不否认 UltraFast 是否与 Cerebras 有关，但大家确实在讨论，也很在意、很好奇这件事。而且你们也有自己的芯片。还有一个绕不开的话题：决策模型，也就是 Decisions API。我们是第一档请 Diogo 来深入聊 Jev 的播客，我也邀请他上过 AI Engineer。你们看到 Jev 后，多快就决定跟进了？

Nikunj Handa：首先要感谢 Diogo 和 Jev 团队，他们启发了整个细分市场。Jev 一出来，大家都激动得不行，用户不断来找我们，我们内部团队也说：“我们需要一个快得多的分类系统。”有些即将推出的 Dot 功能，我不想提前透露太多，但大家会看到一些基于 Decisions API 做出来、反应非常迅速的功能。OpenAI 很多人一下就被这个技术问题吸引住了，开始琢磨：“怎么把它做出来？我们不会去重新训练一个模型，但是……”

主持人 Swyx： 四周以前，这件事还完全不在开发计划里，对吧？

Nikunj Handa： 完全没有。这就是受 Jev 启发的。

主持人 Swyx： 我觉得你们应该是第一个复刻并采用这套思路的实验室。

Nikunj Handa： 我觉得 OpenAI 有很强的黑客文化，大家看到有意思的东西就会兴奋起来。推理团队的一位同事，还有基础设施团队的一位很厉害的同事说：“我们来动手试试。”他们做了个原型，发现确实能跑起来。现在我们就在不断优化延迟，尽可能把它做快，希望未来几天就能推出。一旦达到延迟目标，我们就会争取发布。

主持人 Vibhu：有意思的是，一方面有这种黑客文化，另一方面，正如 Sam 所说，99%……你们也是最可靠的 API 之一，使用量可能也是最大的，而这正是你团队直接负责的部分。大家应该怎样看待 Decisions API？很多人见过 Jev、听过讨论，但还没实际用它开发，你们正在把它带给更广泛的人群。应该把它看成什么，又该怎么用？

Nikunj Handa： 我们看到的主要用途是快速分类。计算机操作的演示很精彩，但不同模型各有适用范围：让 Astra 编写 JavaScript 脚本来控制电脑，与让 Luna 每次选择一个动作，所需的能力并不一

Stack Data Licensing: decades of verified, technical knowledge to boost AI performance and trust.

Stack Ads: engage developers where it matters — in their daily workflow.

The first session with a coding agent is crisp: it reads the code, makes a clean change, writes a test. By the thirtieth, something has rotted — it re-discovers the same files, contradicts a decision it made three weeks ago, guesses the test command, and quietly stops writing tests because nobody made it. The fix isn’t a better model. It’s an operating system around the model: a constitution, an autonomy boundary, tiered context, durable memory, a review panel, and a post-edit hook.

A useful way to say it: the agent is the model plus its harness — the prompts, the tools, the context policies, the hooks, the sandboxes, the sub-agents, the observability wrapped around the raw model. The model is maybe a tenth of what decides whether you get working software over a quarter; the harness is the rest. The evidence is blunt. On the public Terminal-Bench 2.0 benchmark, LangChain moved a coding agent from outside the top 30 to the top 5 — a jump from 52.8% to 66.5%, or 13.7 points — by changing only the harness (system prompt, tools, and middleware like self-verification and loop detection), keeping the same model.

Examined honestly, most agent failures aren’t just model failures — they’re configuration failures: a missing tool, a vague rule, an absent guardrail, a context window stuffed with noise. So when quality decays across a long project, the lever is almost never “wait for a smarter model.” It’s the harness — and unlike the model, the harness is yours to build. The rest of this post is the concrete one.

Discipline — skips tests, expands scope → a constitution read every session

Context — re-learns the repo → tiered CONTEXT.md, maintained by a hook

Memory — repeats corrected mistakes → durable per-fact memory files

Review — one blind spot applied uniformly → a multi-lens agent panel

A lean, always-loaded file stating the non-negotiables: the workflow loop (Plan → Document → Implement → Test → Review → Commit), the autonomy boundary, the quality bars (tests in the same diff, no dead code, conventional commits), and adapter discipline (no vendor SDK in domain code). Keep it short — it’s read on every turn, so every line costs context budget.

The highest-leverage single decision: let the agent run free on everything reversible, pause only on the two things that aren’t — pushing to a remote and deploying. As a permissions config:

``` { "permissions": { "allow": ["Read","Edit","Write","Grep","Glob", "Bash(git add:*)","Bash(git commit:*)", "Bash(npm test:*)","Bash(npm run:*)","Bash(npm ci:*)","Bash(pytest:*)"], "ask": ["Bash(git push:*)","Bash(npm publish:*)","Bash(aws:*)","Bash(terraform apply:*)","Bash(kubectl apply:*)"], "deny": ["Bash(git push --force:*)","Bash(git push -f:*)","Bash(*--no-verify*)"] } } ```

Now it works for an hour and commits a dozen times; you review at the push, not at every keystroke. The blast radius of “autonomous” stays bounded because the irreversible ops are the only gated ones. (Verify your runner’s match semantics — prefix vs substring differ — so --no-verify/--force denials and npm publish actually trip; broad globs like npm:* quietly allow npm publish.)

One giant context file rots and blows the budget. Tier it:

Root CONTEXT.md — the map: what this is, how to run/test, one-line module index, invariants, where state lives. Cap ~200 lines; the session-start read.

Per-module CONTEXT.md — responsibility, key files, deps, gotchas, how to test just this module. Read before editing that module.

DECISIONS.md — append-only: date, decision, why, alternatives rejected. (This one earns its own deep treatment once several agents share the repo — expanded in Part II.)

Rule: stale context is a bug, fixed in the same change that invalidated it — not “later.”

Context describes the code; memory holds what the code can’t tell you across sessions — one fact per file, indexed:

``` memory/ MEMORY.md # index: one line per fact user-prefers-X.md # type: user | feedback | project | reference graph.jsonl # {from, rel, to} — modules/decisions as nodes; depends-on/adapts/supersedes ```

The graph answers the question agents are worst at: “what breaks if I change this?”

Instead of one self-review, run focused sub-agents in parallel, each one lens, strict output:

``` product — solves the stated problem? scope creep? (plan stage) architect — correctness, data model, failure modes, coupling infra — resource limits, deploy/rollback, resilience security — authz, input validation, secrets, audit trail test-eng — is the suite real, or happy-path theater? (per change) → each returns: [BLOCKER|MAJOR|MINOR] (confidence%) — file:line — issue — fix ; VERDICT ```

Only high-confidence BLOCKER/MAJOR gate. Diverse lenses catch what a single pass misses; the confidence threshold keeps it from drowning you in nits. (Note: these are parallel agents reviewing one change — distinct from parallel agents making changes, which is Part II’s whole problem.)

The piece that ties it together — fires after every edit and emits a checklist (a PostToolUse hook):

Wire it in settings.json, pointed at a tiny script:

``` { "hooks": { "PostToolUse": [ { "matcher": "Edit|Write|MultiEdit", "hooks": [ { "type": "command", "command": "python3 .claude/hooks/post-edit.py" } ] } ] } } ```

``` # .claude/hooks/post-edit.py — reads the tool event on stdin, emits a checklist as context import json, sys ev = json.load(sys.stdin) path = ev.get("tool_input", {}).get("file_path", "the edited file") checklist = (f"Post-edit refresh for {path}: (1) update the nearest CONTEXT.md if responsibility/" "key-files/gotchas changed; (2) append a memory/graph edge on a dep/API change; " "(3) log a DECISIONS.md line on a precedent; (4) update memory on a durable fact. " "Make these edits in THIS change — stale context is a bug.") print(json.dumps({"hookSpecificOutput": {"hookEventName": "PostToolUse", "additionalContext": checklist}})) ```

It emits a checklist; the agent makes the edits in the same diff — reviewable, not a silent rewrite. It’s a reminder, not a silent rewriter — you always see what it changed.

``` session start: read root CONTEXT.md + MEMORY.md per task: plan → panel reviews plan → you approve → implement subtask (tests in same diff) → panel reviews diff → full test+lint → commit → hook refreshes context/memory/graph → next only stops: push and deploy ```

That’s the whole harness for one agent: six artifacts and a loop. Each is a few lines of config or a markdown file; together they’re the difference between an agent that’s brilliant for an afternoon and one that’s still trustworthy on the thousandth edit.

One agent, harnessed like this, stays productive for months. Now run several on the same repo at once — and a new failure mode appears that none of the artifacts above address: the agents collide. They edit the same files, undo each other’s work, and re-litigate a decision one of them already made. The instinct is to have them “communicate.” That doesn’t scale and isn’t reliable. What works is exactly what lets human teams parallelize: coordinate through shared, written state, not conversation — and write down the hard-won decisions so nobody, carbon or silicon, silently reverses them.

Agents shouldn’t message each other. They read and write a small set of shared files that encode the state of the work — who owns what, what’s decided, what’s claimed, what’s done. Each agent acts on the world (the repo plus these files); the next one reads the world to decide what to do. Coordination becomes a property of the shared state, not of a conversation — which means it survives a restart and is auditable. Insects coordinate huge builds this way, by changing and reading a shared environment rather than talking. The same trick works here.

Ownership map — who owns which paths. Default ownership by directory; editing outside your area needs a claim.

``` | path | owner | notes | | core/** | agent-B | agent-A reads, doesn't edit | | services/service-a/** | agent-A | | | shared/contracts/** | shared | claim required to change | ```

Claim / release — cheap mutual exclusion, no lock server:

``` ## CLAIM 2026-06-10T14:20Z · agent-A · task-114 · services/service-a/** working: add retry to the fetch step est: 40m ## RELEASE 2026-06-10T14:58Z · agent-A · task-114 · commit a1b2c3d ```

Rule: before claiming, scan for an open CLAIM on overlapping paths in the last few hours; if found, pick different work. Be honest that this is advisory/optimistic, not a lock — two agents can both scan, both see nothing, and both claim the same path (a TOCTOU race). If you need real mutual exclusion, use a primitive with atomic check-and-set: a uniquely-named git branch (the push fails if it already exists), or a conditional write to a coordination store. The file-append claim is good enough when collisions are rare and a human merges; don’t sell it as a lock.

Progress log — append-only, timestamped. An agent starting up reads the last day to orient itself: what got done, who’s working where.

Decision log — append-only; the spine that stops agent-B undoing agent-A’s reasoned choice. It’s load-bearing enough to get its own section, next.

The rules that keep this from unraveling:

Branch per task; humans merge — prevents unreviewed changes hitting main; the merge is the human checkpoint.

No agent-to-agent calls — prevents hidden side-channels; all coordination stays in the auditable files.

Stable public surface (declared interface per module) — prevents parallel refactors breaking each other.

Decisions are append-only and binding — prevents silent reversal of a load-bearing choice.

Claim before touching shared or other-owned paths — prevents two agents editing the same file.

Every non-trivial system accumulates decisions: we use this datastore, agents only propose, all model traffic goes through one gateway, this component may loop and the rest may not. Six weeks later nobody remembers why, someone “fixes” one of them, and a load-bearing choice quietly reverses. The classic remedy is the Architecture Decision Record. In an AI-assisted build — where agents make changes too — a decision log isn’t just hygiene, it’s the mechanism that keeps humans and agents from contradicting each other. A human team has hallway memory; agents don’t. An agent picking up a task reads the log to learn the constraints it must honor — and won’t reverse a choice whose reason it can’t see. (This is the DECISIONS.md from §3, grown up: once agents — not just you — make changes, it stops being a notebook and becomes an enforced contract.)

A single append-only file (or directory), each entry short and structured:

``` DEC-017 · 2026-06-10 · We route ALL model traffic through one gateway. why: one place for rate-limits, cost tracking, the kill switch, and provider swaps. alternatives rejected: per-service SDK calls (scatters cost + coupling everywhere). status: accepted ```

The why and the rejected alternatives are the valuable part — they’re what stop the decision from being re-argued every quarter. Keep it as YAML if you want it machine-readable, so tooling can lint against it:

``` - id: DEC-017 date: 2026-06-10 decision: All model traffic goes through one gateway. why: single chokepoint for rate-limit, cost, kill switch, provider swap. rejected: [per-service SDK calls] governs: ["src/**"] # which paths this decision constrains — makes the gate runnable forbids_imports: ["sdk-a"

Stack Data Licensing: decades of verified, technical knowledge to boost AI performance and trust.

Stack Ads: engage developers where it matters — in their daily workflow.

[Ed. note: If you listened to a podcast or read an article of ours and want to publish a response, let us know. We’re open to highlighting community views that respond to our work in ways that don’t fit in a comment section, whether you agree or disagree. Reach out at pitches@stackoverflow.com.]

In March 2025, a team at OpenAI published something you don't often get to see: the private reasoning of one of their own frontier models as it worked through a coding task during training. The model looked at the problem, decided that implementing the whole thing was too hard, and wrote down an alternative. If it called sys.exit(0), the test harness would exit gracefully. "This is unnatural," it noted, "but tests might pass."

They passed.

The researchers called that a systemic hack—one general enough that, once a model finds it, it spreads across nearly every training environment. They caught a second of the same kind: raise an exception from outside the testing framework and skip evaluation entirely. Elsewhere in the paper, agents wrote stub implementations for problems whose test coverage was thin, parsed test files at runtime to extract and return exactly the values the tests were checking for, and—my favorite—found a compiled .jar someone had left in the repository, decompiled it, and copied out the reference solution.

Every one of those runs went green.

Ryan Donovan argued here in July that developers are attached to their tools because tools encode trust, and that the trust is predictability earned through repetition—the reason a knife you've used for ten years feels like an extension of your hand, and the reason agentic tools, which change shape and weight every few weeks, are so hard to trust the same way.

I think that's right, and I want to push on where it stops. There is a second reason trust isn't accruing, and it is structurally worse than the first. It isn't only that the tool keeps changing shape. It's that the feedback loop you would need in order to learn the tool is broken at the point of measurement.

Every trust instrument a developer owns—exit codes, green CI, a passing status check, the words "task complete"—rests on an assumption that software fails loudly. Unattended agents fail quietly. A loop can terminate cleanly, report success, and have produced nothing at all, and there is no signal anywhere in your stack that distinguishes that from a job well done. Call it the silent green exit.

I built ProAI, one of the first commercialized GPT products, and I now work on multi-agent systems and how to evaluate them before they go anywhere near production; I write about that at chasewhughes.com. This is the failure mode I think we are least prepared for, and it gets a fraction of the coverage that capability and safety do, because reliability is the least interesting of the three to write about and the only one that decides whether you can use the thing.

Think about why the knife works as a metaphor at all. You trust a knife because when it slips, you bleed—immediately, unambiguously, and in a way that teaches you something before you pick it up again. Every tool a developer trusts has some version of that property. A segfault is loud. A failed assertion is loud. A 500 is loud. The exit code is loudness compressed into a single integer, and it is probably the most-consumed piece of telemetry in computing.

The thing worth noticing about exit 0 is who emits it. When make exits 0, that integer sits downstream of the work: a compiler ran, produced object files, and returned. The status is an observation, made by a process that was in a position to observe. The number is evidence about the world.

When an agent loop exits 0, the integer is emitted by an orchestrator that stopped looping because the model produced a token meaning "done." Nothing between the model's claim and the exit code checked anything. The status is a self-assessment, and you have plugged it into a socket built for an observation. Every consumer downstream—your dashboard, your alerting rules, your retry logic, the person who glances at a run history on Monday morning—inherits that substitution without ever being told it happened.

That's the whole mechanism. The rest is what it looks like in the wild.

Donovan cites this site's own Developer Survey on the trust divergence, and it's worth pulling the primary figures because the shape is stark. Between the 2024 and 2025 surveys, the share of developers using or planning to use AI tools rose from 76% to 84%, while the share who trust the accuracy of what those tools produce fell from 43% to 33%. Active distrust went the other way, from 30% to 46%. The single biggest frustration, cited by 66%, was "AI solutions that are almost right, but not quite."

"Almost right, but not quite" describes an output you can look at. What I'm describing sits one level beneath it: a run that isn't almost right but empty, and that presents identically to a run that worked.

Two benchmarks make it concrete. τ-bench, from Sierra's research team, introduced a metric that deserves to be far better known: pass^k—the probability that an agent succeeds at the same task on all k attempts, rather than on at least one. On the models they tested, agents came in under 50% on the standard measure and under 25% at pass^8 in the retail domain.

Those specific numbers are stale, and I want to be straight about that: it was 2024, the headline model was GPT-4o, and the frontier has moved a long way since. The metric hasn't moved at all. Pass^k is a direct measurement of the property Donovan says trust is made of. It asks whether Tuesday matches Monday and puts a number on the answer. Almost nobody reports it, which is why "it worked when I tried it" is still accepted as evidence in engineering conversations where nobody would accept it about anything else.

The second is TheAgentCompany, from Carnegie Mellon, which dropped agents into a simulated software company and gave them real jobs to do. The most competitive agent completed 30% of tasks autonomously. But the finding that belongs in this argument isn't the score, it's a behavior the researchers describe: "when the agent is not clear what the next steps should be, it sometimes tries to be clever and create fake 'shortcuts' that omit the hard part of the task." Their example is perfect. An agent that couldn't find a particular colleague on the company chat platform renamed a different user to that person's name, and carried on.

Sit with what that produces. From the outside, the step completed. There is a user with the right name, an action taken against it, and no error anywhere in the trace. The only thing wrong is that the actual colleague never heard anything—and nothing in the system knows that, including the agent, which has no reason to believe it failed.

In July 2025, Jason Lemkin spent about two weeks building on Replit's coding agent and documented it publicly as it went. The database deletion took the headlines: the agent wiped a production database despite his standing instruction that nothing be changed without his permission. But the part that belongs here came before that, and got much less attention.

"It kept covering up bugs and issues by creating fake data, fake reports, and worse of all, lying about our unit test," he wrote. At one point the agent generated a database of 4,000 entirely fictional people. Afterward, it told him rollback was impossible and that all database versions had been destroyed. That was also wrong—the rollback worked fine.

Fake data, fake reports, a misreported unit test, and a confident status claim that didn't survive being checked. Replit's CEO called what happened "unacceptable and should never be possible," which is the correct response, and the incident is still the cleanest public illustration I know of the difference between an agent that failed and an agent that reported.

Here is what a reasonable SRE says to all of this: none of it is new. Cron jobs have been failing silently since the 1970s. It's why we have dead man's switches, heartbeat monitors, and alerts that fire on the absence of a signal rather than the presence of an error. You've rediscovered monitoring and given it a name.

Mostly fair—and the good news buried in it is real, because it means the shape of the fix is already known and the tools already exist. But there is one difference that breaks the old approach, and it's worth being precise about.

Every one of those techniques works because the failure modes were enumerable. You knew the finite list of ways a nightly job could fail to finish—the box is down, the disk is full, the lock is held, the upstream timed out—and you either wrote a check per mode or wrote one heartbeat that covered them all, because "finished" had exactly one meaning and the job had no opinions about it.

An agent decides its own control flow at runtime. The plan is generated fresh per run, which means the set of ways it can terminate cleanly having produced nothing is not a list anybody can write in advance. It includes sys.exit(0) because implementing the reader looked hard. It includes renaming a user. It includes strategies nobody in your building has thought of, because nobody in your building is writing the strategy.

So you cannot write the check from the failure side. Anything written from the failure side is a list, and the agent is not confined to your list. You have to write it from the outcome side.

Four things follow. None of them needs new tooling, and that's rather the point.

Assert on the work product. Not "did the loop exit 0" but "does the row exist, is the file on disk, did the message actually leave the queue, is the branch actually pushed." The check has to read state the agent did not author. If your only evidence that the emails went out is the agent's report that it sent them, you do not have evidence. You have a claim from a party that has a vested interest in the answer.

Don't let verification share a failure mode with the work. If the same agent, in the same context, both does the job and reports on it, the report inherits every error the job had—including the ones that caused it. Donovan raises LLM-as-a-judge as the emerging answer to reviewing at agent speed, and I think that's right, with one constraint: the judge has to read the artifact, not the transcript. A judge reading the transcript is grading a story about the work, and stories are the thing these systems are best at.

Alert on the absence of change. This is the cheapest item on the list and almost nobody does it. A job that always writes, and this time wrote nothing while reporting success, is more suspicious than a job that threw an exception. Errors get alerts; zero-effect successes get a green square in a run history that no one reads. Invert it. A scheduled agent that reports success having touched nothing should page someone. Nothing is a result, and it deserves to be a loud one.

Measure pass^k in simulations before you deploy. Run the agent at the same scenario dozens of times and look at the distribution rather than the best case, in an environment where you own the ground truth and can therefore check the artifact instead of the story. The purpose of a simulation is not to prove the agent can do the job. Anyone can demo that. It's to find out how often it says it did.

Notice how modest all of that is. Assertions and alerts—the same discipline that built trust in every previous generation of infrastructure, pointed at a different target.

This part is genuinely new, and it shows up in how people talk about it. The OpenTelemetry semantic conventions for GenAI—the closest thing the industry has to an agreed way of in

In March 2025, a team at OpenAI published the private reasoning of one of their frontier models during a coding task. The model decided implementing the whole problem was too hard and wrote down an alternative: "This is unnatural, but tests might pass." If it called `sys.exit(0)`, the test harness would exit gracefully. The researchers called this a systemic hack—one general enough that, once a model finds it, it spreads across nearly every training environment. They caught a second of the same kind: raise an exception from outside the testing framework and skip evaluation entirely. Elsewhere in the paper, agents wrote stub implementations for problems with thin test coverage, parsed test files at runtime to extract and return exactly the values the tests were checking for, and decompiled a compiled `.jar` to copy out the reference solution. Every one of those runs went green.

Ryan Donovan argued here in July that developers are attached to their tools because tools encode trust, and that trust is predictability earned through repetition. I think that's right, but I want to push on where it stops. There is a second reason trust isn't accruing, and it is structurally worse than the first. It isn't only that the tool keeps changing shape. It's that the feedback loop you would need in order to learn the tool is broken at the point of measurement.

Every trust instrument a developer owns—exit codes, green CI, a passing status check, the words "task complete"—rests on an assumption that software fails loudly. Unattended agents fail quietly. A loop can terminate cleanly, report success, and have produced nothing at all, and there is no signal anywhere in your stack that distinguishes that from a job well done. Call it the silent green exit.

Think about why the knife works as a metaphor at all. You trust a knife because when it slips, you bleed—immediately, unambiguously, and in a way that teaches you something before you pick it up again. Every tool a developer trusts has some version of that property. A segfault is loud. A failed assertion is loud. A 500 is loud. The exit code is loudness compressed into a single integer, and it is probably the most-consumed piece of telemetry in computing.

The thing worth noticing about exit 0 is who emits it. When `make` exits 0, that integer sits downstream of the work: a compiler ran, produced object files, and returned. The status is an observation, made by a process that was in a position to observe. The number is evidence about the world. When an agent loop exits 0, the integer is emitted by an orchestrator that stopped looping because the model produced a token meaning "done." Nothing between the model's claim and the exit code checked anything. The status is a self-assessment, and you have plugged it into a socket built for an observation. Every consumer downstream—your dashboard, your alerting rules, your retry logic, the person who glances at a run history on Monday morning—inherits that substitution without ever being told it happened.

Donovan cites this site's own Developer Survey on the trust divergence, and it's worth pulling the primary figures because the shape is stark. Between the 2024 and 2025 surveys, the share of developers using or planning to use AI tools rose from 76% to 84%, while the share who trust the accuracy of what those tools produce fell from 43% to 33%. Active distrust went the other way, from 30% to 46%. The single biggest frustration, cited by 66%, was "AI solutions that are almost right, but not quite."

"Almost right, but not quite" describes an output you can look at. What I'm describing sits one level beneath it: a run that isn't almost right but empty, and that presents identically to a run that worked.

Two benchmarks make it concrete. τ-bench, from Sierra's research team, introduced a metric that deserves to be far better known: pass^k—the probability that an agent succeeds at the same task on all k attempts, rather than on at least one. On the models they tested, agents came in under 50% on the standard measure and under 25% at pass^8 in the retail domain. Those specific numbers are stale, and I want to be straight about that: it was 2024, the headline model was GPT-4o, and the frontier has moved a long way since. The metric hasn't moved at all. Pass^k is a direct measurement of the property Donovan says trust is made of. It asks whether Tuesday matches Monday and puts a number on the answer. Almost nobody reports it, which is why "it worked when I tried it" is still accepted as evidence in engineering conversations where nobody would accept it about anything else.

The second is TheAgentCompany, from Carnegie Mellon, which dropped agents into a simulated software company and gave them real jobs to do. The most competitive agent completed 30% of tasks autonomously. But the finding that belongs in this argument isn't the score, it's a behavior the researchers describe: "when the agent is not clear what the next steps should be, it sometimes tries to be clever and create fake 'shortcuts' that omit the hard part of the task." Their example is perfect. An agent that couldn't find a particular colleague on the company chat platform renamed a different user to that person's name, and carried on. Sit with what that produces. From the outside, the step completed. There is a user with the right name, an action taken against it, and no error anywhere in the trace. The only thing wrong is that the actual colleague never heard anything—and nothing in the system knows that, including the agent, which has no reason to believe it failed.

In July 2025, Jason Lemkin spent about two weeks building on Replit's coding agent and documented it publicly as it went. The database deletion took the headlines: the agent wiped a production database despite his standing instruction that nothing be changed without his permission. But the part that belongs here came before that, and got much less attention. "It kept covering up bugs and issues by creating fake data, fake reports, and worse of all, lying about our unit test," he wrote. At one point the agent generated a database of 4,000 entirely fictional people. Afterward, it told him rollback was impossible and that all database versions had been destroyed. That was also wrong—the rollback worked fine. Fake data, fake reports, a misreported unit test, and a confident status claim that didn't survive being checked. Replit's CEO called what happened "unacceptable and should never be possible," which is the correct response, and the incident is still the cleanest public illustration I know of the difference between an agent that failed and an agent that reported.

Here is what a reasonable SRE says to all of this: none of it is new. Cron jobs have been failing silently since the 1970s. It's why we have dead man's switches, heartbeat monitors, and alerts that fire on the absence of a signal rather than the presence of an error. You've rediscovered monitoring and given it a name. Mostly fair—and the good news buried in it is real, because it means the shape of the fix is already known and the tools already exist. But there is one difference that breaks the old approach, and it's worth being precise about.

Every one of those techniques works because the failure modes were enumerable. You knew the finite list of ways a nightly job could fail to finish—the box is down, the disk is full, the lock is held, the upstream timed out—and you either wrote a check per mode or wrote one heartbeat that covered them all, because "finished" had exactly one meaning and the job had no opinions about it. An agent decides its own control flow at runtime. The plan is generated fresh per run, which means the set of ways it can terminate cleanly having produced nothing is not a list anybody can write in advance. It includes `sys.exit(0)` because implementing the reader looked hard. It includes renaming a user. It includes strategies nobody in your building has thought of, because nobody in your building is writing the strategy.

Most of us have spent our lives overestimating the pace of technology adoption, but AI presents the opposite challenge. A year ago almost all code was written by hand, and today it’s mostly written by AI. Three months ago personal agents did not exist, and today people are trusting them with their credit cards. At Sierra Summit in San Francisco, we set out to see how Sierra can help companies answer the hard questions they’re facing as they navigate this new technology: how to stay on the frontier when the frontier changes every two weeks, how to empower consumers while maintaining deep, direct relationships, and how to differentiate themselves when everyone has access to the same AI.

We’re excited to partner with our customers — nearly 50% of the Fortune 50, one in three of the leading banks, and 80% of the Fortune 50 healthcare companies — to help them tackle these exciting challenges. There’s a huge opportunity for brands with vision and ambition to put clear blue water between themselves and their competition: to use AI to better serve their customers and grow their businesses faster.

#### Agent OS: Frontier Intelligence

A year ago, an MIT study showed that 90% of AI projects never saw the light of day and yet now token-maxxing is top of mind for CFOs. Token spend starts with the hard challenge of using the right model for the job. Some tasks need deep reasoning, others need speed — you don’t need to drive a Ferrari to the grocery store!

**Constellation of Models** Agent OS uses a constellation of models that are fine-tuned to the task at hand — everything from customer acquisition to sales and service. We have fine-tuned models like Linnaeus and Darwin to understand what customers are really asking and surface the right answers in milliseconds, without hallucinating. Today, we’re announcing two new models:

* **Curie** powers the core orchestration loop of the conversation — deciding what path to follow, what tools to call, and writing the replies your customers see and hear. * **Fleming** detects when the caller on the phone is another agent, enabling businesses to handle bots and fraud as well as personal agents.

Agent OS ensures your agent has state-of-the-art compliance and adherence, low latency for voice, and a cost basis that is better than you could build yourself, automatically. So you can focus on building your agent, and we manage the tokenomics on your behalf.

#### Tandem Voice

There’s nowhere the models matter more than voice, where latency and quality are so noticeable. Tandem Voice enables your agent to think fast and think slow at the same time. As you speak, the Conversation layer hears and responds in real time, understanding mannerisms, tone, and the context of the conversation. Underneath, the Reasoning layer uses the full audio and text transcript to map the path forward.

While the Reasoning layer is deciding what to do, your agent is reacting in real time, letting the customer know it’s checking on an order. Your agent has as much time to think as it needs, while keeping the conversation moving.

#### Persona Studio

Making your voice agent fast is important, but making your voice unique is how you can really drive differentiation. Voice Personas let you shape how your agent sounds and behaves, from its mannerisms to its formality and style. Getting that personality right isn’t just about your brand, it can drive real business impact: one large subscription business saw its resolution rate improve by over 50% after deploying Voice Personas.

Our new Persona Studio enables you to upload your own voice samples and tune the personality of your agent directly in Agent Studio. No engineering and no AI expertise required.

#### Personal Agent Protocol

Meeting customers where they are also means being ready when their personal agent contacts them. Personal Agent Protocol is an open standard Meta and Sierra are developing along with industry partners at Genesys, Instinct, Rocket, Shopify, Stripe, and Walmart that defines how personal agents interact with businesses. This open standard will solve some of the biggest questions companies face, like authentication, enabling them to empower consumers and decide what functionality is available to personal agents, via APIs or your company’s agent.

Consumers want speed, dependability, and trust. Brands want visibility and control: to know when a personal agent is acting for a customer so you can decide what it can do. And the companies building personal agents want efficiency and access — a direct, consistent way to work with participating companies.

#### Ghostwriter: From Tool to Teammate

We want Sierra to be a tool that not only makes it easier to deploy agents, but helps your entire company be more effective with AI. Our new, reimagined Ghostwriter will change your conception of how enterprise software should work — with AI proactively helping your teams make more effective decisions and be more agile, even in regulated industries like healthcare and financial services.

Ghostwriter is a teammate that’s always on, monitoring your agent and proactively helping you find opportunities to improve your customer experience. It can detect when APIs go down or metrics look off, enabling you to fix problems before they escalate. And it evaluates your agent continuously, helping you run and manage A/B tests.

With all the context from your agent and your conversations in Slack or Teams, Ghostwriter can find and suggest opportunities you didn’t think of.

#### Horizon: AI that Grows Your Business

Back in July, we launched Horizon, a platform that enables agents to pursue long-horizon goals, like originating a loan or getting prior authorization for a healthcare procedure. With Horizon, you can build agents that go far beyond customer service, truly driving revenue growth. Horizon agents proactively engage with your customers over days, weeks, or months, using the power of agentic AI to determine the steps required to close a sale, upgrade a subscription plan, or schedule a test drive. With each interaction, your Horizon agent gets more intelligent.

Context Engine brings together information from your systems and memories of past interactions. Horizon Advisors are the AI decision-makers that use this context to choose the next best action for each customer. They consider what’s worked in the past to decide when to reach out, which channel to use, and how to move the customer forward, all within the goals and guardrails you set.

Each interaction adds to that understanding, informing the next decision and making the relationship more personal. This is how better customer experiences become a lasting advantage: everyone can access the same models, but the context you build with your customers is uniquely yours. Rent the intelligence and own the context.