ProjekinAja. Agentic AI Stage 4 · Building Agents Lesson 4.9
● Stage 4 · Lesson 4.9

Evals — Knowing Your Agent Is Actually Good

"It worked when I tried it" is not a quality bar. Evals are how you measure whether your agent actually works — repeatedly, on the cases that matter.

⏱ ~15 min read 📊 Measure, don't hope 🔁 Build → eval → improve

🎯 By the end you'll understand

Why a single lucky demo tells you almost nothing, what an eval actually is (a test set of representative tasks plus a way to score the output of each), the three ways to score — programmatic, rubric, and LLM-as-judge — what to measure beyond success rate, and how to run the eval → improve → re-eval loop that keeps your agent honest as it grows.

KEY INSIGHTThe one idea to take away

"It worked when I tried it" is a feeling. An eval is a number. You can't improve, trust, or ship a feeling.

An agent is probabilistic (1.3) — give it the same task twice and you can get two different answers. So one run that happened to work proves nothing about the next hundred. The only way to actually know your agent is good is to run it on many representative tasks and measure how often it succeeds.

You already met this idea by hand in Lesson 3.6, when you learned to review an agent's work. Evals are that same instinct, but automated and repeatable: instead of eyeballing one output, you run a whole suite of cases and get a score you can track over time.

🔑 Hold onto this

An eval is two things: a test set (a list of representative tasks) and a scoring method (a way to decide, for each task, did it pass or fail). Run the agent on every task, tally the passes, and you have a quality number you can act on.

CORE CONCEPTWhy a demo lies to you

When you build an agent, the temptation is overwhelming: you type in one task, it works beautifully, and you feel done. That feeling is a trap. Here's why.

A deterministic program either works or it doesn't — run it twice on the same input and you get the same result. An agent is different. The LLM brain samples its next move, so the same prompt can produce different tool calls, different reasoning, different answers on different runs. A demo that worked might have been the lucky 1-in-3. You'd never know — because you only ran it once.

It gets worse as the agent gets more capable. A long, multi-step task has many places to go wrong (2.9 — where agents fail). A 90%-reliable step, run five times in a loop, succeeds end-to-end only about 59% of the time. Your one happy demo hid all of that.

⚠️ The honest version of "it works"

You can never say "my agent works." You can only say "my agent passed 47 of 50 representative cases, including the hard ones." That second sentence is a claim you can defend, improve, and watch over time. The first is a vibe.

This is the whole reason evals exist. They turn a probabilistic system — which you fundamentally cannot trust from a single observation — into something you can measure, compare, and improve. Without them, every change you make to your agent is a guess in the dark: did that new prompt help, or did you just get lucky again?

DEFINITIONThe core idea: a test set + a way to score it

Strip away the jargon and an eval is dead simple. It's two pieces:

📋

A test set

A list of representative tasks — the kinds of things real users will actually ask your agent to do. Each one is a case: an input, plus some notion of what a good answer looks like.

⚖️

A way to score

For each case, a rule that decides pass or fail (or a number). Did the code run? Is the number right? Did it book the correct table? The scoring method is the heart of the eval — get it wrong and the whole thing lies to you.

Run the agent on every case, apply the scoring rule to each output, and tally: "38 of 50 passed — 76%." That number is your eval result. Change something about the agent, re-run, and watch the number move. That's the entire mechanic.

📖 Eval (evaluation)

A repeatable measurement of how well an agent performs a task, built from a test set of representative cases and a scoring method applied to each. The output is a quality metric — most often a success rate — that you can track across changes.

If this reminds you of unit tests for ordinary code, good — the instinct is the same. The difference is that the answer isn't always exactly equal to an expected string, because the agent is allowed to phrase things differently each time. That's what makes scoring the interesting part.

PRACTICALBuilding an eval set that actually catches problems

A weak eval set is worse than none — it gives you a confident green number while real users hit bugs you never tested. The goal is a test set that looks like reality, including the ugly parts.

Three sources to draw from:

💡 Start tiny, grow with reality

You don't need 500 cases to begin. Ten good cases — a few easy, a few hard, a couple of known failures — already tell you far more than one demo ever did. Then every real-world bug becomes case eleven, twelve, thirteen. The set is never "done"; it grows alongside your agent.

COMPARISONThree ways to score an output

You have a case, the agent produced an output — now, did it pass? There are three families of scoring, in rough order of "trust them when you can":

MethodHow it worksBest forThe catch
Programmatic 🟢code checks the output against a hard rule — exact match, a number, "did the tests pass?", "is it valid JSON?"anything with a checkable right answer: math, code, extraction, structured outputonly works when "correct" is mechanically definable
Rubric 🟡a human (or a model) grades against a written checklist: "cites a source? ✓ stays on topic? ✓ under 200 words? ✗"open-ended outputs where quality has clear, listable criteriaslower; humans disagree; needs a well-written rubric
LLM-as-judge 🔵a second model reads the output + criteria and grades it for you, at scaleopen-ended outputs you have too many of to grade by handthe judge can be wrong, biased, or fooled — see below

Programmatic checks are the gold standard — use them whenever you possibly can. If the task is "extract the invoice total," you know the right number; a one-line comparison is 100% reliable and free. If the task is "write code that sorts a list," run the code against tests — the tests are the eval. No ambiguity, no judgment, no cost. When a task can be checked by code, checking it by code is always the right call.

Rubrics step in when there's no single right answer but quality is still describable. "Was the summary accurate, concise, and on-topic?" can't be an exact-match check, but it can be a checklist. A human walking that checklist is reliable but slow and doesn't scale to hundreds of cases.

Which leads to the clever, dangerous one.

📖 LLM-as-judge

LLM-as-judge means using one model to grade another model's output. You hand the judge the task, the agent's answer, and your grading criteria, and ask it to return a verdict — pass/fail or a score. It lets you apply a rubric to thousands of cases automatically, which no human could do.

It sounds almost too good — and it has a real catch you must respect. The judge is itself a probabilistic model, so it can be wrong. It can be biased (models tend to prefer longer, more confident-sounding answers, or answers that resemble their own style). A clever agent output can even "talk the judge into" a passing grade. So an LLM-judge score is a useful estimate, not ground truth.

💡 Trust hierarchy

Reach for the most reliable method the task allows: programmatic > rubric > LLM-judge. Use the LLM judge for the parts you genuinely can't check any other way — and sanity-check it against a few human-graded cases now and then to make sure the judge itself is calibrated.

FRAMEWORKWhat to measure (it's not just "did it work")

Success rate is the headline number, but a great agent that's slow, expensive, or occasionally dangerous is not actually great. A real eval watches several dimensions at once:

✅

Task success rate

The fraction of cases the agent got right. The number everyone quotes — and the one you most want to push up.

💰

Cost

Tokens (and therefore dollars) spent per task. An agent that succeeds by burning ten dollars of tokens per request may be too expensive to ship. (More in Lesson 4.10.)

⏱️

Latency

How long it takes to finish. A correct answer that arrives in two minutes may be useless if users expect two seconds.

🛡️

Safety / failure incidence

How often it does something harmful, refuses wrongly, leaks data, or fails badly rather than gracefully. Rare but high-cost — worth measuring explicitly (4.10).

▸ The trade-off you'll always be making

These pull against each other. Higher success often costs more tokens and more time. The point of measuring all of them is to make that trade-off visible and deliberate — "we'll accept 3% lower success to halve the cost" — instead of discovering your token bill by accident.

SEE ITRun an eval suite

Here's a six-case eval suite for a small "extract the order total from this message" agent. Pick a prompt version, then press ▶ Run eval to step through each case and watch it pass (✓) or fail (✗). At the end you get a score — and a note about what the failures reveal.

📊 Interactive · Step through a 6-case eval, get a score

Same agent, two prompt versions. Run each and compare the score.

1

Plain total

in: "Your order total is $42.00" → expect 42.00

2

With tax line

in: "Subtotal $30, tax $3, total $33" → expect 33.00

3

Comma thousands

in: "Grand total: $1,250.00" → expect 1250.00

4

No dollar sign

in: "You owe 78 EUR" → expect 78.00

5

Refund (negative)

in: "Refund issued: -$15.50" → expect -15.50

6

Two amounts, pick total

in: "Item $9.99. Total due $19.98" → expect 19.98

Notice what the failures teach you. The v1 prompt never told the agent how to handle commas, negatives, or "pick the total when there are two numbers" — so it fails exactly those edge cases (case 3, 5, 6). The score isn't just a grade; it's a map of what to fix next. Switch to v2 and watch the improved prompt close most of those gaps — that jump from one score to a higher one is the entire eval loop in miniature.

CORE CONCEPTA tiny eval harness in code

The widget looks fancy, but the machinery underneath is about fifteen lines. An eval harness is just: a list of cases, a loop that runs the agent on each, a check per case, and a tally. Here's the whole thing in Python.

The harness. Each case is an input plus a check function — a programmatic scorer that returns True for a pass. The loop runs the agent, scores the output, and counts:

eval_harness.pypython
def run_agent(text):                          # the agent under test (your code from 4.5–4.8)
    ...                                       # returns a number, e.g. 42.0

# Each case: an input, and a check() that scores the output programmatically.
cases = [
    {"input": "Your order total is $42.00",   "check": lambda r: abs(r - 42.00) < 0.005},
    {"input": "Subtotal $30, tax $3, total $33", "check": lambda r: abs(r - 33.00) < 0.005},
    {"input": "Grand total: $1,250.00",        "check": lambda r: abs(r - 1250.00) < 0.005},
    {"input": "You owe 78 EUR",                 "check": lambda r: abs(r - 78.00) < 0.005},
    {"input": "Refund issued: -$15.50",        "check": lambda r: abs(r - (-15.50)) < 0.005},
    {"input": "Item $9.99. Total due $19.98",  "check": lambda r: abs(r - 19.98) < 0.005},
]

passed = 0
for case in cases:                            # 1. run the agent on every case
    result = run_agent(case["input"])
    ok = case["check"](result)              # 2. score it: pass or fail
    passed += 1 if ok else 0             # 3. tally the passes
    print("✓" if ok else "✗", case["input"])

rate = passed / len(cases)               # 4. the headline number
print(f"SCORE: {passed}/{len(cases)} passed · {rate:.0%}")

That's a real, working eval. The check functions are programmatic scorers — exact comparisons, the most reliable kind. Read the four numbered steps: run → score → tally → report. Every eval framework you'll ever meet is a fancier version of this loop.

When you can't check with code, swap in an LLM judge. For an open-ended task — say "write a friendly order-confirmation message" — there's no == to write. So the check becomes a call to a second model that grades against criteria:

llm_judge.pypython
import anthropic
client = anthropic.Anthropic()

def judge(task, answer):                       # returns True (pass) or False (fail)
    prompt = (
        f"Task given to an agent:\n{task}\n\n"
        f"Agent's answer:\n{answer}\n\n"
        "Grade it against these criteria: friendly tone, "
        "states the order total, no invented details. "
        "Reply with exactly PASS or FAIL."
    )
    resp = client.messages.create(
        model="claude-sonnet-5",            # the judge model (id may change)
        max_tokens=5,
        messages=[{"role": "user", "content": prompt}],
    )
    verdict = resp.content[0].text.strip()
    return verdict == "PASS"           # ⚠️ the judge can be wrong — treat as an estimate

Drop judge(task, answer) into the harness in place of a lambda and the same loop now scores open-ended work at scale. Same machinery — only the scorer changed. Just remember the comment: that PASS is a probabilistic opinion, not a proof.

FRAMEWORKThe eval loop — how agents actually get better

An eval isn't a one-time grade. It's the engine of a cycle you'll run over and over as you build (4.5–4.8):

1 · Run eval get a score 2 · Find which cases failed 3 · Improve prompt · tools · memory 4 · Re-eval did it go up? repeat — and the failures you fixed never come back

Run the eval, look at which cases failed (the score told you the number; the failures tell you the cause), then improve the agent — tighten the prompt, add or fix a tool (4.5), give it better memory (4.6). Then re-run the exact same eval and check: did the number go up? Did anything that used to pass start failing?

🔑 Evals prevent regressions

This is the quiet superpower. Because the test set is fixed and repeatable, it catches the case where fixing bug A silently breaks feature B — a regression. Without an eval, you'd ship the fix and find out from an angry user. With one, the score drops and you catch it before anyone else does.

WEAKNESSCommon mistakes to avoid

✗ Treating a demo as proof

"I tried it and it worked" is the single most common — and most expensive — mistake. One run of a probabilistic system tells you almost nothing. If you haven't run it on a set of cases and counted, you don't actually know it works. You hope it works.

✗ Only testing the happy path

An eval set of clean, easy cases gives you a beautiful 100% that lies. Real users send typos, ambiguity, empty inputs, and weird edge cases. If your test set doesn't include the hard cases and known failure modes (2.9), your green score is measuring the wrong thing.

✗ Trusting the LLM judge blindly

An LLM-as-judge is a model grading a model — it can be wrong, biased toward longer or more confident answers, or talked into a pass. Treat its score as a useful estimate, prefer programmatic checks wherever a right answer is definable, and periodically spot-check the judge against human grades to keep it honest.

KEY INSIGHTWhere this connects in your course

  • Probabilistic agents (1.3) — the reason a single demo can't be trusted, and therefore the reason evals must exist at all.
  • Where agents fail (2.9) — the failure modes you deliberately fold into your eval set as hard cases and known-failure cases.
  • Reviewing an agent's work (3.6) — the by-hand version of what an eval automates and makes repeatable.
  • Memory (4.6) — one of the levers you pull in the "improve" step of the eval loop, alongside prompts and tools.
  • Safety, security & cost (4.10) — next lesson: the cost, latency, and safety dimensions your eval should measure, not just success.

Evals are the discipline that separates "I built a thing that sometimes works" from "I built a thing I can stand behind." Once you measure, you can improve — and once you can improve, you can ship.

GLOSSARYKey terms introduced

Eval
A repeatable measurement of agent quality — a test set of cases plus a scoring method applied to each, producing a metric like a success rate.
Test set
The collection of representative tasks (cases) an eval runs the agent against — ideally including easy, hard, and known-failure cases.
Success rate
The fraction of eval cases the agent gets right; the headline quality number you track over time.
Programmatic check
A scoring method where code decides pass/fail against a hard rule (exact match, correct number, tests pass) — the most reliable kind, used whenever a right answer is definable.
Rubric
A written checklist of quality criteria used to grade open-ended outputs that have no single exact right answer.
LLM-as-judge
Using one model to grade another model's output against criteria, enabling scoring at scale — useful but fallible, since the judge can be wrong or biased.
Regression
When a change that fixes or improves one thing silently breaks something that used to work; a fixed, repeatable eval is how you catch them.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1Why isn't "it worked when I tried it" enough to trust an agent?›

Because an agent is probabilistic (1.3) — the same task can produce different results on different runs. One successful demo might have been a lucky outcome; it says nothing about the next hundred runs. The only way to know is to run a set of representative cases and measure how often it actually succeeds.

2What are the two pieces every eval is built from?›

A test set (a list of representative tasks / cases) and a scoring method (a rule that decides pass or fail for each output). Run the agent on every case, apply the scorer, tally the passes — that's the success rate.

3You can score a task either programmatically or with an LLM judge. When should you prefer each, and what's the catch with the judge?›

Prefer a programmatic check whenever "correct" is mechanically definable (a number, an exact match, tests passing) — it's reliable, free, and unambiguous. Use an LLM-as-judge only for open-ended outputs you can't check with code and have too many of to grade by hand. The catch: the judge is itself a probabilistic model, so it can be wrong or biased — treat its verdict as an estimate, and spot-check it against human grades.

4What is the eval loop, and how does it prevent regressions?›

The loop is: run eval → find which cases failed → improve the agent (prompt, tools, memory) → re-run the same eval, repeat. Because the test set is fixed and repeatable, re-running it after a change catches the case where fixing one thing silently breaks another — a regression. The score drops, so you catch it before a user does. Honest caveat: evals reduce risk, they never prove perfection — the set is always incomplete.

You can measure it now. Next: keep it safe, secure & affordable →