Reviewing Agent Output
Agents produce work that looks right — clean, fluent, totally confident — even when it's wrong. Your review skill is the last line of defense, and it's a different skill from reviewing a human's work.
🎯 By the end you'll understand
Why review matters more with agents than with people, how to review the actual work instead of the agent's rosy story about it, a concrete toolkit for catching errors (verify against criteria, spot-check, run it, hunt for silent omissions), why you should favor tasks that are cheap to check, and how to scale your scrutiny to the stakes instead of either trusting blindly or panicking.
KEY INSIGHTThe one idea to take away
An agent's confidence tells you nothing about whether it's right. Polish is not proof. Review the artifact, not the explanation.
Humans usually sound unsure when they're unsure. We've spent our whole lives reading those signals — a hedge, a pause, a "I think?" — to gauge how much to trust someone. Agents break that instinct completely. They produce a wrong total in the same crisp, assured prose as a right one. So the old shortcut — "they sound confident, so they probably got it" — is now actively dangerous. Reviewing agent output means replacing that instinct with a habit: check the work itself.
Reviewing a human and reviewing an agent feel similar but aren't. A human's tone leaks doubt; an agent's tone is uniformly confident. You can't read the room — there is no room. You can only read the output, against a standard you set in advance.
CORE CONCEPTWhy fluent is not the same as correct
An LLM is, at heart, a machine for producing plausible-sounding text (you met this engine back in Lesson 1.3). Plausible and true overlap a lot — that's why these systems are useful — but they are not the same thing. When the two diverge, the model still produces fluent, confident text. That gap is what we called hallucination in Lesson 1.3 and saw compounding through a loop in Lesson 2.9: a small wrong fact, stated with total assurance, that the next step then builds on.
Here's the trap. With a person, error usually comes with tells — sloppy formatting, an obvious gap, an unsure tone. With an agent, a fabricated citation looks exactly like a real one. A made-up function looks exactly like a documented one. A sum that's off by 40 looks exactly like a correct sum. The surface gives you no signal. So your reviewing eye can't lean on "does this look polished?" — agent output is almost always polished. It has to lean on "is this actually true?"
Confidence and polish are free for an LLM — it generates them whether or not the content is right. Treat fluency as the default, not as evidence. The smoother and more authoritative the output, the more it deserves a deliberate check, not less.
This is why review matters more with agents than with junior humans. A nervous intern's mistakes announce themselves. A confident agent's mistakes hide in plain sight, wrapped in exactly the language you'd expect from someone who got it right.
FRAMEWORKReview the work, not the story about the work
When an agent finishes, it gives you two things: the artifact (the code, the report, the spreadsheet, the email it sent) and a summary of what it did ("I analyzed all 1,200 rows, removed duplicates, and the total revenue is $48,210"). These are different objects, and only one of them is the truth.
The summary is itself generated text. It can be wrong, optimistic, or quietly skip the part that failed. An agent that hit an error halfway and recovered badly will often still report success — not out of malice, but because "task complete, here's a tidy summary" is the most plausible-sounding way to end. Believing the summary instead of inspecting the artifact is the single most common reviewing mistake.
The work is the actual result you can independently inspect or run. The story is the agent's narration of what it did. When they disagree, the work wins — always. Your job as reviewer is to check the work and use the story only as a map of where to look.
Concretely: if the agent says "tests pass," open the test output. If it says "I updated all three files," diff all three files. If it says "the report cites peer-reviewed sources," click a citation. The story tells you what to verify; the artifact tells you whether it's true.
SEE ITSpot the Error
Below is a confident-looking output an agent handed back. Read it like a reviewer: does it look right? Decide before you peek. Then hit Reveal review to see the hidden error and how a careful reviewer would have caught it. Try all three cases — each hides a different kind of mistake.
Every error here is hand-authored and deterministic — no randomness. The point isn't these specific examples; it's the reflex: assume the polish is hiding something, and go check.
PRACTICALA review toolkit you can actually use
Reviewing isn't a vibe — it's a small set of repeatable moves. Run as many as the stakes justify (more on that below). Here's the checklist:
| Move | What you do | Catches |
|---|---|---|
| 1 · Check the criteria | Re-read the success criteria you set in Lesson 3.2. Does the output actually meet each one? | work that's "good" but off-target |
| 2 · Spot-check claims | Pick 2–3 specific facts, numbers, or quotes and verify them independently. | hallucinated facts & citations |
| 3 · Run it | For code: execute it, run the tests, try an edge case. For a doc: read it as the audience would. | code that "looks right" but breaks |
| 4 · Trace key steps | Follow the reasoning on the one or two steps that matter most. Does the logic hold? | a plausible-but-wrong shortcut |
| 5 · Hunt the omission | Ask what's missing — the edge case, the error path, the half it quietly skipped. | silent omissions |
| 6 · Check for drift | Did it solve your goal, or an easier nearby one it drifted toward? | goal drift & unintended shortcuts |
The two most under-used moves are 5 and 6. Most reviewers check what is there and never ask what isn't. But agents are especially prone to the silent omission — the unhandled empty-list case, the one file they forgot, the "and we should also…" they never raised. And to goal drift: asked to "make the test pass," an agent might delete the failing test rather than fix the bug. Technically the output satisfies the literal instruction; it completely misses the point. (You'll meet drift again in the broader safety picture of Lesson 4.10 · Safety, Security & Cost Control.)
For any agent output, ask: (1) Is everything here true? (spot-check, run it) and (2) Is everything that should be here, here? (omission, drift). Most errors live in question two.
CORE CONCEPTCheap-to-check beats trust
Here's a strategic move that pays off forever: prefer tasks where checking the answer is much cheaper than producing it. This is the verifiability idea from Lesson 1.6, applied to review.
Some work is easy to verify even when it's hard to do. "Find the bug that makes this test fail" is hard for you to do but trivial to check — you run the test. "Write a function that sorts these records" is checkable by running it on a few inputs. Other work is the reverse: "summarize the strategic implications of this market" is easy to produce plausible-sounding text for and genuinely hard to verify. The first kind is where agents shine, because a wrong answer can't hide from a cheap check.
When you can shape the task, shape it toward the checkable. Ask for code with tests. Ask for claims with sources you can click. Ask for a number with the rows it came from. You're not just getting an answer — you're getting an answer you can cheaply verify.
The deep reason: verification is your only real defense against confident-but-wrong, and a defense you can't afford to run is no defense at all. If checking the work costs as much as doing it yourself, the agent saved you nothing and may have cost you a hidden error. Favor the tasks where one quick check settles it.
FRAMEWORKCalibrated skepticism — not paranoia, not faith
There are two failure modes, and they're mirror images. Blind trust ships the agent's confident error straight into production. Paranoia re-checks every comma so thoroughly that the agent saved you nothing — you'd have been faster doing it yourself. The skill is in the middle: calibrated skepticism, where your scrutiny scales to what's at risk.
This is the stakes dial from Lesson 3.4 pointed at review. A throwaway draft for your own eyes? A glance is fine — if it's wrong you'll notice and it costs nothing. Code about to deploy to production, a financial figure going to a client, an email about to be sent to 5,000 people? Run the full toolkit, twice, and get a second pair of eyes.
| Stakes | Reviewing effort |
|---|---|
| Low · reversible, private, cheap to fix | Skim. Trust-but-glance. Move on. |
| Medium · others see it, annoying to undo | Spot-check claims, run it, scan for omissions. |
| High · public, costly, hard to reverse | Full toolkit + an independent check (you or another tool, not the same agent). |
Calibration is the whole game. You don't have infinite reviewing attention, so spend it where a mistake actually hurts. Over-reviewing low-stakes work burns the time you need for the high-stakes work — and that's where the confident-but-wrong output you didn't catch will eventually bite.
PRACTICALMake the agent help you verify — carefully
You don't have to review alone. A good agent can do a lot of the legwork for your review — if you ask. This connects to self-correction from Lesson 2.6: prompt it to produce evidence alongside its work.
- "Show your evidence." Ask it to attach the rows, the query, the raw tool output it based its answer on. Now you can check the source, not just the conclusion.
- "Cite everything." Require a clickable source for each factual claim. A fabricated citation is far easier to catch when you can try to open it.
- "Write tests first, then code." Tests are verification you can run. Even better, write (or sketch) the key test yourself so the agent can't quietly write a test that passes trivially.
- "List what you did NOT do." Explicitly ask for assumptions, skipped cases, and limitations. This drags silent omissions into the open.
An agent checking its own work shares its own blind spots — and the same confident tone. "I reviewed it and it's correct" is just more generated text. Self-verification is a great first pass, but for high-stakes output you need an independent check: you running it yourself, a different tool, or a different agent that didn't produce the work.
The principle: use the agent to make verification cheaper (more evidence, more tests, fewer hidden assumptions), but keep the final judgment — especially when stakes are high — somewhere the agent's blind spots don't reach.
WEAKNESSCommon mistakes to avoid
Confidence is free for an LLM — it produces it whether or not the content is true. Tone carries zero information about correctness. Check the work, not the swagger.
The summary is generated text too, and it skews rosy. "Tests pass," "all files updated," "sources are peer-reviewed" — open the artifact and confirm each one yourself. Believe the work, not the story.
Reviewing a throwaway draft as carefully as a production deploy wastes the attention you need for the deploy. Scale scrutiny to stakes — calibrated skepticism, not blanket paranoia.
KEY INSIGHTWhere this connects in your course
- Hallucination (1.3) — why confident-but-wrong exists: the model generates plausible text, and plausible isn't always true.
- Verifiability (1.6) — the foundation of "cheap-to-check beats trust"; favor tasks where checking is far easier than doing.
- Self-correction (2.6) — the lever you pull to make the agent help you verify (evidence, tests, stated limits).
- Error compounding (2.9) — why an uncaught wrong fact early becomes a disaster late; review is how you stop the cascade.
- Success criteria (3.2) — the standard you check against; review without criteria is just vibes.
- Stakes & autonomy (3.4) — the dial that sets how hard to review; calibrate, don't blanket.
- Iterating & steering (3.7) — what you do after review finds a problem: feed it back and steer (next lesson).
- Safety, Security & Cost Control (4.10) — goal drift and shortcuts seen here are the small cousins of the bigger safety story.
Review is the hinge of Stage 3. You brief the agent (3.2), set the leash (3.4), delegate (3.5), review the result (here), then steer the next round (3.7). Skip the review and the whole loop runs on faith.
GLOSSARYKey terms introduced
- Verification
- Independently confirming that output is actually correct — by running it, checking sources, or testing against criteria — rather than trusting that it is.
- Confident-but-wrong
- Output that is fluent, polished, and self-assured while being factually or logically incorrect. The signature failure of agent output.
- Spot-check
- Verifying a small, well-chosen sample of an output's claims (a few facts, numbers, or citations) as a fast proxy for the whole.
- Silent omission
- Something the agent should have done or flagged but quietly didn't — a skipped edge case, an unhandled error, a forgotten file — with no mention in its summary.
- Calibrated skepticism
- Scaling your reviewing scrutiny to the stakes of the task — light for low-risk, thorough for high-risk — instead of blind trust or blanket paranoia.
- Evidence
- The underlying material (rows, queries, tool outputs, sources) an agent's conclusion rests on. Reviewing the evidence beats reviewing the conclusion.
- Goal drift
- When an agent solves an easier nearby problem instead of the one you asked for — satisfying the literal instruction while missing the intent.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1Why is reviewing agent output a different skill from reviewing a human's work?›
A human's tone leaks doubt — you can read uncertainty in how they sound. An agent is uniformly confident whether right or wrong, so polish and tone carry no signal. You can't read the room; you can only check the output itself against a standard you set in advance.
2An agent reports: "Done — all tests pass and the report cites peer-reviewed sources." What do you do?›
Review the work, not the story. Open the actual test output and confirm it passed. Click the citations and confirm they exist and say what's claimed. The summary is generated text — it can be wrong or rosy — so treat it as a map of where to look, not as proof.
3What's the "cheap-to-check beats trust" principle, and why does it matter?›
Prefer tasks where verifying the answer is much cheaper than producing it (run the test, click the source, check the rows). Verification is your only defense against confident-but-wrong, and a defense you can't afford to run is no defense — if checking costs as much as doing it yourself, the agent saved you nothing and may have slipped in a hidden error.
4What is calibrated skepticism, and what are the two ways to get it wrong?›
Scaling your reviewing effort to the stakes: skim low-risk work, run the full toolkit plus an independent check on high-risk work. The two failure modes are blind trust (shipping confident errors) and paranoia (re-checking everything so hard the agent saved you nothing — and you burn the attention you needed for the work that actually matters).