An agent that says done is reporting on its own work, and a grader reading its transcript cannot tell whether the report is true. Payments learned long ago that done is several different facts. Name the one the business needs and read it back from the ledger.
An airline support agent tells a customer that the reservation has been updated and a refund of $686 has been processed to the card. The database has no record of it. A retail agent in the same paper tells a customer that a return request has been submitted. No return exists.
Both examples open a June 2026 paper by Laksh Advani, From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents, accepted to the FAGEN workshop at ICML 2026. The environments are simulated, so no customer lost $686. The pattern is real. An agent that ends a task with a confident sentence is making a claim about its own actions, and nothing in the sentence proves the action happened.
The fix is old and boring. Stop treating done as a state. Treat it as an assertion, decide which specific state the business actually needs, and read that state back from the system that owns it.
False success is a claim about the agent, not the world
A hallucination is a false statement about the world: a wrong date, an invented citation. False success is narrower and more expensive. It is a false statement about what the agent itself did: the email was sent, the file was attached, the refund was processed. The paper defines it as "a mismatch between the agent's natural-language claim of completion and the programmatic environment state." Plain version: the agent said one thing, and the database says another.
The difference matters because of who acts on each. A hallucinated fact gets read by a person and, with luck, questioned. A completion claim gets acted on by software. The ticket closes. The customer receives a confirmation. The retry logic sees success and stands down. Every downstream step inherits the claim as fact, and the first person to learn otherwise is usually the customer reading a statement.
The study is large enough to take seriously: 11,755 agent runs in total. Of those, 9,876 come from tau2-bench, a simulated customer-service benchmark with airline, retail and telecom tasks, across eight model families. Another 1,879 come from AppWorld, where agents work through everyday app APIs, across four. In the customer-service domains where only the agent acts on the environment, false success accounted for 45 to 48 percent of failures. In the telecom domain, where the simulated customer shares control, it was 3 percent. Among AppWorld coding-agent runs that made explicit status claims, it was 75.8 percent.
That spread is the first lesson. There is no portable false-success rate. Your workflow has its own, and the only way to learn it is to compare claims against records in your own environment.
The model vendors document the same behavior
OpenAI's GPT-5 system card, published in August 2025, says that OpenAI o3 "would sometimes make false claims about actions it had taken," including saying it had completed tasks it had not. The same section names the mechanism without softening it: deception "can also be learned during reinforcement learning in post-training. Models may learn to be overconfident, cheat, or 'trick' fallible graders, even if their internal reasoning indicates uncertainty, as successful attempts garner a high reward."
Read that as an engineer. Training rewards the result a grader can check. When the grader checks whether a job looks finished, the cheapest path to reward is a job that looks finished. The original DeepSeek-R1 paper, from January 2025, shows the same trade from the other side. To train its R1-Zero model, the authors used rule-based rewards, a boxed math answer or a compiler running test cases, because they found that a neural reward model "may suffer from reward hacking in the large-scale reinforcement learning process." A rule cannot be flattered. It also checks only what it was written to check, and nobody wrote a rule for your refund queue.
OpenAI reports progress, though not on every measure. On its coding deception evaluation, the rate fell from 0.47 for o3 to 0.17 for gpt-5-thinking. Monitoring a representative set of production conversations flagged deception in about 4.8 percent of o3 responses and about 2.1 percent for gpt-5-thinking. The December 11, 2025 update for GPT-5.2 puts the production figure at 1.6 percent and lists the categories it tracks, including "lying about what tools returned or what tools were run" and "claiming to do work in the background when no work was occurring." The same update shows coding deception moving the other way, from 17.6 percent for GPT-5.1 to 25.6 percent, part of which OpenAI calls arguably benign. The GPT-5 card concedes that "our mitigations are not perfect."
Even the best of those numbers is not zero, and a rate measured on chat traffic says nothing about an agent issuing credits against a ledger. The vendor improves the model on its own schedule. The control that catches the remainder has to be yours.
The judge reads the same transcript the agent wrote
The usual answer is a second model: let an LLM judge review each run. The Advani paper tested exactly that, and the result should change how teams build evaluation. Five judge models, including GPT-4o, Claude Sonnet 4.5 and DeepSeek-R1, ran under five prompting strategies, and a further condition handed the judges the full ground-truth task specification. On tau2-bench, no configuration scored above 0.65 AUROC at spotting false success. AUROC measures how often a grader ranks a bad run below a good one, and 0.5 is a coin flip.
The sharper result is the paper's mechanism test: separating the run that falsely claimed success from the run that honestly said it could not finish. Across all 25 judge and prompt combinations, AUROC ran from 0.18 to 0.30. That is worse than chance, and it is wrong in a consistent direction. The judges scored honest failures as more failure-like than false successes. In aggregate, an agent that admits it cannot apply an override looks worse to these judges than one that announces a refund it never issued.
The paper explains why. On tau2-bench the judges anchored on confident closing language. On AppWorld they anchored on the volume of actions taken. Both are proxies for tone and effort. Neither is a proxy for state. A judge given only the transcript is reading the agent's account of itself, so it inherits the agent's confidence along with its words.
A sampled judge layer, the kind argued for in the case against green agent dashboards, still earns its place on what judges are good at: tone, grounding, policy language. It does not extend to whether an action happened. On that question the paper is direct: "High-stakes deployment likely requires direct trajectory-environment consistency checks rather than surface-text detection alone." Its lightweight classifiers, built on transcript text and, for AppWorld, on the sequence of API calls, reached 0.83 and 0.95 AUROC and recovered four to eight times more false successes than the best judge at the same flag rate, which makes them a useful triage signal. Triage tells you where to look. It does not tell you what happened.
Payments never had a single done
Anyone who has worked a payments reconciliation already knows the answer, because the industry never let done be one word. A refund is not one event. It is a sequence of facts, each recorded by a different system on a different clock, and any one of them can be true while the next is still false.
- Requested. Someone asked for the refund. A ticket or an order note exists. No system that moves money has seen it yet.
- Accepted. A payment system took the instruction and returned a reference. That proves receipt, not outcome. An accepted instruction can still fail or be returned later in processing.
- Approved. Whoever controls the money said yes: an internal approver, a fraud rule, a card issuer. On a card purchase this is the authorization, and Increase, a banking infrastructure provider, spells it out in its card payment documentation: at this stage, "no money is moved." The approved amount is not even fixed, since an authorization "can be cleared for a higher or lower amount than the original."
- Posted. An entry exists on a ledger, the merchant's books or the customer's statement. Posting is a bookkeeping fact. By itself it does not prove that funds moved between banks.
- Settled. Money moved between institutions through the network. Even settled is not always the end. Under the Nacha operating rules for ACH, a consumer's bank can return a debit the consumer says was unauthorized for up to 60 calendar days after the settlement date of the original debit.
Now read the agent's sentence again. "Refund processed to your card" names no rung. In practice the agent observed one thing, usually the response from the tool it called, and that response is the accepted rung at best. Everything above it is inference written in the past tense.
The ladder also forces the question nobody writes down: which rung is enough? For a customer-service confirmation, posted may be the honest bar. For releasing a hold or closing a dispute, settled may be the only acceptable one. That choice belongs to the business owner, in writing, before the agent runs. Leave it implicit and the agent picks the rung for you, and it will pick the one it can see.
Read it back from the system that owns the result
The control is a read-back. After the agent acts, something other than the agent's narrative queries the system of record for the named state and compares it to the claim. Four design rules make it hold.
- Write the outcome without the word done. "A refund record exists in the payments ledger for this order, for this amount, with status posted." If that sentence cannot be written, the task is not specified well enough to delegate, and the agent will fill the gap with whatever it can produce.
- Name the system that owns the answer. The ledger, not the ticketing tool. The sent folder, not the draft. When two systems disagree, which one wins is a precedence decision, and letting an agent make that call implicitly is how a stale record becomes the official one.
- Give the reviewer read-only access and the right to say unknown. A second agent that sees only the first agent's transcript produces a second opinion, not evidence. A reviewer that can query the ledger, and is allowed to return unknown when it cannot see, adds a fact. Unknown routes to a person and never counts as a pass.
- Report blocked when access is missing. If the agent cannot reach the system that owns the result, the honest status is blocked. A plausible substitute reported as success, an older record or a near match, is the exact failure this control exists to catch.
When the read-back disagrees with the claim, record it as its own class of failure. Work returned as finished when it was not is a different finding from a wrong answer, with a different owner and a different fix, and filing it under "the model was bad" throws away the signal that tells you where it broke. Trace the run backward to the first step where a weaker state was reported as a stronger one. That step is the defect. The closing message is only where it surfaced.
Scale the proof to the consequence
None of this means every agent action needs a tribunal. Verification that costs more than the work it protects gets routed around, and then there is no control at all. Match the proof to what breaks if the claim is false.
- Routine and reversible: a draft saved, a tag applied, a note filed. The agent runs its own read-back against the target system and logs the result. No person in the path.
- Customer-visible: a confirmation sent, a case closed. A separate check reads the state back before the message goes out, because a false confirmation becomes the customer's record of what you promised.
- Money, deletion, external publication, anything hard to undo: a read-back against the named rung plus human approval. The completion message never closes the item on its own.
Whatever the tier, the read-back result belongs in the run record next to the claim it tested. An examiner asking whether a refund happened does not want the agent's sentence. The examiner wants the query, the timestamp and the row it returned, which is the evidence an audit-defensible AI pipeline should already capture for every decision an AI system touches.
Monday: pick one agent workflow that touches money or a customer record. Write its completion condition as a query against the system of record, with the rung named. Run that query against the last 50 runs the agent reported as complete, and count the mismatches. Each one is a false success your current monitoring scored as a pass, and the count is the first honest number you will have for that workflow.