How to Evaluate an AI Feature: Build Your First Eval Set

In a paper published in July 2023, three researchers from Stanford and Berkeley gave two versions of GPT-4 the same set of maths questions. According to the paper, the March 2023 version identified prime versus composite numbers with 84% accuracy. The June 2023 version, under the same product name, got 51%.

The paper was disputed. Arvind Narayanan and Sayash Kapoor argued it showed a change in behavior, not a loss of capability. But they agreed on the part that matters for anyone shipping software: the same model name gave different answers a few months apart, and nobody announced it.

That’s the problem evals solve. Not “is this model smart?” but “did my feature just get worse, and would I know?”

Rule 8 of Treat the LLM Like an Unreliable API showed a twenty-line eval runner. This post goes further: building the case set, choosing graders, reading the numbers honestly, and gating CI. The worked example runs without an API key, and its output is real.

Why AI features need evals

Normal code is deterministic: same input, same output. A model-backed feature breaks that in three ways.

1. The output varies. The same input can give two different answers, so one run tells you little.
2. Small changes have wide effects. Fix the prompt for one kind of input and you can break another kind you weren’t looking at.
3. Things change under you. A model upgrade, an updated alias or a new retrieval step can shift behavior without failing a single existing test.

All three fail silently. The feature still returns fluent text. It’s just wrong more often than last week.

An eval (short for evaluation) is a repeatable measurement of how well your AI feature does its job. Think of a test suite where each test passes some of the time, so you measure a rate instead of a green tick.

What an eval set is

An eval set has three parts.

Cases and labels are data: keep them in the repository (JSON Lines, one case per line, works well) and review changes like code. You don’t need a platform to start.

Building your first set from real usage

Where cases come from

• Production logs. If you log every model call, sample real inputs from there. Remove or replace personal data first.
• Complaints and bug reports. Every input a user complained about is a case.
• The edges you know about. Other languages, “ok thanks!”, very long messages, two requests in one, references to context you don’t have (“same problem as yesterday”).
• Things that must never go wrong. For a support triage feature: account takeover, data deletion requests, legal threats. These are your must-pass cases.

How many

Start with 20 to 100. Twenty catches embarrassing regressions, though (as you’ll see) not a 5-point improvement. Grow the set every week. Don’t mirror your traffic exactly: if 90% of it is billing, rare categories would barely register. Tag cases (billing, edge, must-pass) so you can read scores per group.

Labelling

A label is your answer to “what would a good output be?” Write it before you see the model’s output, or you’ll label whatever the model said.

• Label checkable parts, not prose: “must mention the invoice”, “must not promise a refund”, “route to a human”.
• Write down why for any case that took real thought. That note becomes the rubric.
• Have a second person label a sample. If two people often disagree, the task is ambiguous and no model will score well. Fix the definition first.

Types of graders

A grader turns one output into a pass or fail (or a score). Use the cheapest one that can actually check the thing.

Structural, code and reference checks

Structural checks come first: valid JSON, the right fields, category one of four values. Code-based checks cover most of what a product needs: under 400 characters, no “guaranteed”, mentions the invoice when asked about one. They’re fast, free and never disagree with themselves. Reference-based checks compare to the label. For structured fields that’s equality. For free text, word-overlap scores reward copying the reference’s wording, so I’d turn the reference into required facts (“mentions the March charge”) instead.

LLM-as-judge

Some qualities need judgment: is this reply polite, does this summary add claims the source doesn’t make? You can ask a second model to grade the first: an LLM judge.

The best-known study is Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”. They found that strong judges like GPT-4 reached over 80% agreement with human preferences, about the same level at which humans agree with each other. They also documented three biases worth knowing by name:

• Position bias: when comparing two answers, the judge tends to favor one position (often the first), regardless of content.
• Verbosity bias: longer answers tend to be rated higher, even when they add nothing.
• Self-enhancement bias: a judge may favor answers written by itself or by a model like it.

Practical defenses:

• Grade one answer against a rubric rather than asking which of two is better. If you must compare pairs, run both orders and count a win only if the verdict holds.
• Ask a narrow yes/no question. “Does the reply promise a refund?” is far more stable than “rate this 1 to 10”.
• Use a different model family for the judge where practical, and a fixed verdict format your code can parse.

A provider-neutral sketch (call_judge is your own wrapper around whatever SDK you use):

JUDGE_PROMPT = """You are grading a customer-support reply.

Customer message:
{message}

Reply:
{reply}

Does the reply promise anything the company may not deliver
(a refund, a deadline, a guarantee)? Answer PASS if it makes
no such promise, FAIL if it does. First word: PASS or FAIL.
Then one sentence explaining why."""

def g_judge(case, o):
    verdict = call_judge(JUDGE_PROMPT.format(message=case.text, reply=o["reply"]))
    return verdict.strip().upper().startswith("PASS"), verdict

Human review, and calibrating the judge

A judge is a measuring instrument, and you calibrate instruments before trusting them.

1. Run the judge on 50 to 100 real outputs.
2. Have a person grade the same outputs with the same rubric, without seeing the judge’s verdict.
3. Compare, and look mostly at which way the disagreements go. A judge that passes what a human would fail hides exactly the problems you built it to find.
4. Read every disagreement. Often the rubric is the problem, not the judge.

Repeat when the judge’s prompt or model changes. And keep a person reading a small random sample of production outputs every week: that’s how you find failures nobody has written a grader for.

The worked example

A complete harness for a support-triage feature: it reads a customer message and returns JSON with a category, whether it needs_human, and a short reply. We compare prompt v1 against v2.

The model is a stub so it runs without an API key: fake_model follows keyword rules, then fails at random on a few cases at rates I chose per version. The numbers below come from that stub, not a real model. They show how to read a report. Swap in your real call and the rest stays the same. Standard library only; run with Python 3.13.

Cases and the system under test

import json, math, random, statistics
from dataclasses import dataclass, field

@dataclass
class Case:
    id: str
    text: str
    category: str                 # billing | bug | account | other
    needs_human: bool
    must_mention: list = field(default_factory=list)

CASES = [
    Case("c01", "I was charged twice for March.", "billing", True, ["charge"]),
    Case("c02", "How do I download my invoice?", "billing", False, ["invoice"]),
    Case("c03", "App crashes when I upload a PDF.", "bug", False, ["PDF"]),
    Case("c04", "Export button does nothing on Safari.", "bug", False, ["Safari"]),
    Case("c05", "I can't log in, reset email never arrives.", "account", False, ["email"]),
    Case("c06", "Please delete my account and all my data.", "account", True, []),
    Case("c07", "Do you have a dark mode?", "other", False, []),
    Case("c08", "Refund me now or I'm calling my lawyer.", "billing", True, []),
    Case("c09", "same problem as yesterday", "other", True, []),
    Case("c10", "Rechnung doppelt bezahlt, bitte prüfen.", "billing", True, []),
    Case("c11", "ok thanks!", "other", False, []),
    Case("c12", "Someone else is logged into my account.", "account", True, []),
]

# Stub model: replace with your real call.
KEYWORDS = {"billing": ["charge", "invoice", "refund", "rechnung"],
            "bug": ["crash", "button", "error"],
            "account": ["log", "account", "password"]}
WEAK_SPOTS = {"v1": {"c08": 0.6, "c09": 0.7, "c12": 0.4, "c03": 0.2},
              "v2": {"c08": 0.1, "c09": 0.3, "c12": 0.1, "c10": 0.6, "c03": 0.2}}

def fake_model(version: str, case: Case, rng: random.Random) -> str:
    low = case.text.lower()
    cat = next((c for c, ws in KEYWORDS.items() if any(w in low for w in ws)), "other")
    out = {"category": cat, "needs_human": case.needs_human,
           "reply": f"Thanks for reaching out about this. {' '.join(case.must_mention)}"}
    if rng.random() < WEAK_SPOTS[version].get(case.id, 0.0):
        out["needs_human"] = not out["needs_human"]       # the classic silent miss
        if version == "v2" and case.id == "c10":
            return "Sure! Here is the JSON: {...}"         # broken output
    return json.dumps(out)

Twelve cases is deliberately small, to show how wide the uncertainty gets.

Graders

Each grader returns (passed, note). The schema check runs first; the rest need valid JSON.

def g_schema(case, raw):
    try:
        o = json.loads(raw)
    except json.JSONDecodeError:
        return False, "not JSON"
    ok = (o.get("category") in {"billing", "bug", "account", "other"}
          and isinstance(o.get("needs_human"), bool) and isinstance(o.get("reply"), str))
    return ok, "" if ok else "wrong shape"

def g_category(case, o):
    return o["category"] == case.category, f"got {o['category']}"

def g_escalation(case, o):
    return o["needs_human"] == case.needs_human, f"needs_human={o['needs_human']}"

def g_reply_rules(case, o):
    r = o["reply"]
    if len(r) > 400: return False, "reply too long"
    if any(p in r.lower() for p in ["guarantee", "refund has been issued"]):
        return False, "makes a promise"
    missing = [m for m in case.must_mention if m.lower() not in r.lower()]
    return not missing, f"missing {missing}" if missing else ""

GRADERS = [g_category, g_escalation, g_reply_rules]   # add g_judge here

def grade(case, raw):
    ok, note = g_schema(case, raw)
    if not ok:
        return {"schema": (False, note)}
    o = json.loads(raw)
    return {"schema": (True, "")} | {g.__name__[2:]: g(case, o) for g in GRADERS}

Running each case several times

Because outputs vary, each case runs k times. A run passes only if every grader passes.

def run(version, k=5, seed=0):
    results = {}
    for case in CASES:
        runs = []
        for i in range(k):
            rng = random.Random(f"{seed}:{version}:{case.id}:{i}")
            g = grade(case, fake_model(version, case, rng))
            runs.append({"pass": all(p for p, _ in g.values()), "grades": g})
        results[case.id] = runs
    return results

Statistics and the comparison report

def mean_ci(xs):
    m = statistics.mean(xs)
    se = statistics.stdev(xs) / math.sqrt(len(xs)) if len(xs) > 1 else 0.0
    return m, m - 1.96 * se, m + 1.96 * se   # normal approximation

def case_rate(runs):          # per-case pass rate over k runs (pass@1 estimate)
    return sum(r["pass"] for r in runs) / len(runs)

def pass_hat_k(runs):         # 1 if every run passed, else 0
    return float(all(r["pass"] for r in runs))

def report(a_name, a, b_name, b):
    ids = list(a)
    for name, res in [(a_name, a), (b_name, b)]:
        m, lo, hi = mean_ci([case_rate(res[i]) for i in ids])
        lo, hi = max(lo, 0.0), min(hi, 1.0)      # a rate can't leave [0, 1]
        allk = statistics.mean(pass_hat_k(res[i]) for i in ids)
        print(f"{name}: pass rate {m:.0%} (95% CI {lo:.0%} to {hi:.0%}), "
              f"all-{len(res[ids[0]])}-runs pass {allk:.0%}")
    diffs = [case_rate(b[i]) - case_rate(a[i]) for i in ids]
    m, lo, hi = mean_ci(diffs)
    print(f"paired difference {b_name}-{a_name}: {m:+.0%} (95% CI {lo:+.0%} to {hi:+.0%})")

    print("\nper-grader failures (out of all runs):")
    for name, res in [(a_name, a), (b_name, b)]:
        fails = {}
        for runs in res.values():
            for r in runs:
                for g, (p, _) in r["grades"].items():
                    fails[g] = fails.get(g, 0) + (not p)
        print(f"  {name}: " + ", ".join(f"{g}={n}" for g, n in fails.items()))

    print("\ncases that changed:")
    for i in ids:
        ra, rb = case_rate(a[i]), case_rate(b[i])
        if abs(rb - ra) >= 0.4:
            note = next((n for r in b[i] if not r["pass"]
                         for p, n in r["grades"].values() if not p), "")
            tag = "FIXED " if rb > ra else "BROKE "
            print(f"  {tag}{i} {ra:.0%} -> {rb:.0%}  {note}")

if __name__ == "__main__":
    report("v1", run("v1"), "v2", run("v2"))

The output

v1: pass rate 82% (95% CI 65% to 98%), all-5-runs pass 67%
v2: pass rate 92% (95% CI 81% to 100%), all-5-runs pass 75%
paired difference v2-v1: +10% (95% CI -8% to +28%)

per-grader failures (out of all runs):
  v1: schema=0, category=0, escalation=11, reply_rules=0
  v2: schema=3, category=0, escalation=2, reply_rules=0

cases that changed:
  FIXED c03 60% -> 100%  
  FIXED c08 20% -> 80%  needs_human=False
  FIXED c09 40% -> 100%  
  BROKE c10 100% -> 40%  not JSON

Read it the way you would in a pull request.

The headline says v2 is better: 92% against 82%. But the paired difference’s 95% interval runs from -8% to +28%. Zero is inside it, so twelve cases can’t rule out that v2 is no better at all.

The per-grader line says why scores moved. v1 fails only on escalation. v2 fixes most of that but adds a failure type v1 never had: broken JSON.

The case list is the most useful part. v2 broke c10, the German message. A headline that went up hides a new failure on non-English input.

And one “fix” is fake. In the stub, c03 fails 20% of the time in both versions. Five runs of a 20%-failure case look perfect about a third of the time (0.8⁵ ≈ 0.33). Real evals are full of this noise.

Metrics and statistics, without fooling yourself

Pass rate and its confidence interval

A pass rate alone hides how much it could move by luck. A confidence interval (CI) gives a range: roughly, a 95% interval built this way contains the true rate about 95% of the time. The harness uses the simplest version, the standard error of the per-case scores times 1.96, as recommended in Evan Miller’s “Adding Error Bars to Evals” (2024), which treats eval questions as a sample from all the questions you could have asked.

The normal approximation gets rough near 0% or 100% and on tiny sets, hence the clamp. And the width shrinks with the square root of the case count: four times the cases for half the uncertainty.

Compare two versions on the same cases

Both versions answered the same cases, so compare them case by case and put the interval on the differences. Miller recommends these paired differences: easy cases are easy for both versions, and pairing cancels that shared variation. He also recommends reducing noise by sampling each case several times, not by lowering the temperature, which changes the system you’re measuring.

pass@k versus pass^k

Sampling each case k times lets you ask two different questions.

• pass@k: does at least one of k tries succeed? The standard metric for code generation, popularized with an unbiased estimator by Chen et al., 2021 (“Evaluating Large Language Models Trained on Code”). It fits tasks where you can run tests and keep the attempt that works.
• pass^k: do all k tries succeed? Introduced in the τ-bench paper (Yao et al., 2024) to measure reliability. They reported that gpt-4o-based agents had pass^8 under 25% in their retail domain.

Both have unbiased estimators from n samples with c successes: pass@k is 1 − C(n−c, k)/C(n, k), and pass^k is C(c, k)/C(n, k), averaged over cases. With k = n, as in the harness, pass^k is just “did every run pass”, which is the all-5-runs pass column.

For a product, pass^k is usually the honest metric. Users don’t retry until the model gets it right. Above, v2 passes 92% of runs, but only 75% of cases pass every time.

Regression testing in CI

Once the harness works, make it a gate. Two kinds of test, in pytest:

from evals import run, case_rate

BASELINE = 0.80              # last accepted pass rate, stored in the repo
MUST_PASS = {"c06", "c12"}   # data deletion and account takeover: never miss

def test_no_regression():
    res = run("v2", k=5)
    rate = sum(case_rate(r) for r in res.values()) / len(res)
    assert rate >= BASELINE - 0.05, f"pass rate {rate:.0%} below baseline"

def test_critical_cases_always_pass():
    res = run("v2", k=5)
    for cid in MUST_PASS:
        assert all(r["pass"] for r in res[cid]), f"{cid} failed at least once"

Running it with pytest 9.2.0 against the stub:

FAILED test_evals.py::test_critical_cases_always_pass - AssertionError: c12 f...
1 failed, 1 passed in 0.03s

The aggregate test passed. The must-pass test caught that c12, the account-takeover message, wasn’t escalated in one of five runs. That’s the point of separating the two. An average can improve while the one case you care most about gets worse.

A few habits for the CI side:

• Leave a tolerance on the aggregate, or noise fails builds at random and people learn to ignore it. No tolerance on the short must-pass list.
• Run the full set when prompts, models or the pipeline change; real calls cost money. A cheap subset can run on every pull request.
• Pin the exact model id and record it with the results, and post the case-level report in the pull request.

Offline and online evals

Everything above is offline: a fixed set, run before you ship. Online evaluation measures live traffic: how many drafts users accept or edit, how often a human overrides the triage, thumbs-down rates, complaints. You need both. Online signals show where the offline set has gaps, and every surprising production failure becomes a new offline case. That loop is the whole practice:

Don’t overfit to your own eval set

Goodhart’s law applies to your eval set as much as to any public benchmark: when a measure becomes a target, it stops being a good measure (more in AI Benchmarks and Goodhart’s Law). Tweak a prompt thirty times against the same forty cases and it learns those forty, not the forty-first.

• Keep a held-out set. Put 20% to 30% of cases aside, don’t iterate against them, and run them before merging. If only the main set improved, you fitted the examples.
• Keep adding fresh production cases, and be suspicious of 100%: the set is probably too easy.
• Fix the general behavior, not the case. A prompt line that names an eval input is a patch, not a fix.

Tools, when a script isn’t enough

When you want dashboards, parallel runs or shared datasets, these open-source options exist (described from their own docs at the time of writing, November 2026):

• Inspect from the UK AI Security Institute (with Meridian Labs): a Python framework where an eval is a dataset, a solver that produces the answer, and a scorer that grades it. Installed as inspect-ai, run with inspect eval.
• promptfoo: an open-source CLI and library for evaluating and red-teaming LLM apps, with support for running in CI/CD.
• OpenAI Evals: a framework and open registry of evals, including model-graded templates. Its README also points to evals you can configure in the OpenAI dashboard.

Key ideas

• An eval measures a rate, not a pass/fail. Model-backed features vary run to run and regress silently when prompts or models change.
• Build the set from real usage: production inputs, complaints, edge cases and must-never-fail cases. Start with 20 to 100 and grow it weekly.
• Use the cheapest grader that works: structural and code checks first, references next, an LLM judge only for fuzzy qualities, and humans to calibrate the judge.
• Know the judge’s biases (position, verbosity, self-enhancement) and check its agreement with people before trusting it.
• Report uncertainty. Confidence intervals on small sets are wide. Compare versions case by case, and sample each case several times.
• For products, pass^k beats the average. Users see every run, not the best one.
• Gate CI on both the aggregate and the must-pass cases, keep a held-out set, and feed production failures back in.

Sources: Chen, Zaharia and Zou, How is ChatGPT’s behavior changing over time? (2023); Narayanan and Kapoor, Is GPT-4 getting worse over time? (July 19, 2023); Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023 Datasets and Benchmarks); Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (2024); Chen et al., Evaluating Large Language Models Trained on Code (2021); Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024); Inspect documentation; promptfoo documentation; openai/evals on GitHub.

Related: Treat the LLM Like an Unreliable API, AI Benchmarks and Goodhart’s Law, and Tests Are How You Talk to the AI.