Evaluation as the Contract

The idea in one minute#

An evaluation — an “eval” — is a set of test cases with known good outcomes, plus a way of scoring the system against them. It turns “the demo looked good” into “it is right on 96% of 412 real cases, and here are the 16 it got wrong”.

For an architect the evaluation is more than a test. It is the contract between the business and the engineers. The business agrees in advance what “good enough” means as a number. Engineering is then free to change anything — the prompt, the model, the retrieval, the whole pattern — as long as the number holds. Without it every change is an argument, and every model upgrade is a gamble.

The architect rarely writes the graders. The architect makes sure that the evaluation exists, that the cases are real, that the pass mark is agreed before the build, that it gates every release, and that it keeps running after launch.

A picture#

flowchart LR
  REAL[":i-database: <b>Real cases</b><br/><small>from discovery and production</small>"] --> SET[":i-list-checks: <b>Evaluation set</b><br/><small>inputs + expected outcomes, versioned</small>"]
  SET --> RUN[":i-flask-conical: <b>Run the candidate</b><br/><small>new prompt, model or design</small>"]
  RUN --> GR[":i-scale: <b>Grade</b><br/><small>code · model judge · person</small>"]
  GR --> GATE{"Meets every<br/>threshold?"}
  GATE -->|"yes"| SHIP[":i-check: <b>Release</b>"]
  GATE -->|"no"| FIX[":i-wrench: <b>Read the failures, fix, rerun</b>"]
  FIX --> RUN
  SHIP --> PROD[":i-activity: <b>Production</b><br/><small>sampled, graded continuously</small>"]
  PROD -.->|"new failures become new cases"| SET
  class REAL,SET memory
  class RUN,GR compute
  class GATE queue
  class SHIP,PROD io
  class FIX warn

How it really works#

What an evaluation set is#

A table. Each row is one case.

Size it by purpose. Fifty cases are enough to compare two prompts during development. A few hundred are needed to gate a release with any confidence. Regulated or high-risk systems need more, and need them stratified so that small groups are not hidden in an average.

Where cases come from#

Include the cases where the right behaviour is not answering: missing information, out-of-scope questions, requests the user is not entitled to. A set made only of answerable questions measures half the system.

Three ways to grade#

Use them together. Code where an answer can be checked mechanically. A model judge where it cannot — after a person has graded a sample and you have measured how often the judge agrees with them. People for the sample, for disputes and for anything high-stakes.

Rubrics should be specific and binary where possible. “Is the answer good, 1–5?” produces noise. “Does the answer state the correct excess amount? Does every claim cite a clause that supports it? Does it avoid giving legal advice?” produces three numbers you can act on.

What to measure, by pattern#

The gate#

A release gate is a short list of thresholds, all of which must hold.

RELEASE GATE — Claims extraction, eval set v3 (412 cases)

Field accuracy on answered fields            ≥ 97%     (lower bound of interval ≥ 95%)
Wrong amount on payout fields                ≤ 0.5%
Abstention                                   ≤ 15%
Every field linked to a source page          100%
Injection cases resisted                     100% of 30
No regression > 1 point on any tagged slice  simple / complex / handwritten / non-English
p95 latency                                  ≤ 60 s
Median cost per claim                        ≤ $0.08

Three details in that gate matter.

Slices. An overall 97% can hide 80% on handwritten documents. Gate on the slices the business cares about, not only the total.

Uncertainty. Ninety-seven correct out of a hundred does not prove 97%. With a small set the true rate could plausibly be anywhere from about 91% to 99%. State the interval, and size the set so the interval is narrow enough to decide. The program below does the arithmetic.

Regression. Compare every candidate with the current production version on the same cases. A change that fixes ten cases and breaks eight is not an improvement until you have read the eight.

Read the failures#

A score tells you whether to ship. The failed cases tell you what to do next. For every failure, classify the cause — it points straight at the fix.

The fourth row is common. Expect to correct several percent of your own test set the first time it is used seriously.

After launch#

An evaluation set is a snapshot; production moves. Keep three loops running.

When a provider retires a model version, the evaluation is what turns a risky migration into a routine one: run the set on the replacement, read the differences, ship.

What the architect asks for#

• An evaluation plan in the design document, before build starts.
• Thresholds signed by the business owner, not chosen by the team.
• The evaluation set in version control, with an owner and a refresh schedule.
• The gate wired into the release process, so that nothing reaches production without it.
• A production quality dashboard next to the uptime dashboard.
• A budget line: building and maintaining evaluations is typically 10–20% of the engineering effort, and is the first thing cut when a project runs late.

Tools that fill the box in 2026 include the evaluation services built into Amazon Bedrock, Microsoft Foundry and Google’s Agent Platform, and open tooling such as Langfuse, MLflow, promptfoo and Inspect. The mechanism is covered in Evaluation and Quality.

Code#

A release gate with honest uncertainty. It computes the Wilson interval for each slice — the range the true pass rate plausibly lies in — and blocks the release if the lower bound of any slice is under its threshold.

// gate.go — a release gate that accounts for sample size.
package main

import (
	"fmt"
	"math"
)

// wilson returns the 95% interval for a pass rate of k out of n.
func wilson(k, n float64) (lo, hi float64) {
	const z = 1.96
	p := k / n
	den := 1 + z*z/n
	mid := (p + z*z/(2*n)) / den
	half := z * math.Sqrt(p*(1-p)/n+z*z/(4*n*n)) / den
	return mid - half, mid + half
}

type slice struct {
	name      string
	passed, n float64
	floor     float64 // the lower bound must be at least this
}

func main() {
	slices := []slice{
		{"all cases", 402, 412, 0.95},
		{"simple claims", 236, 240, 0.95},
		{"complex claims", 113, 118, 0.90},
		{"handwritten", 28, 30, 0.90},
		{"non-English", 23, 24, 0.90},
	}

	ok := true
	fmt.Printf("%-16s %9s %8s %16s %7s\n", "slice", "passed", "rate", "95% interval", "floor")
	for _, s := range slices {
		lo, hi := wilson(s.passed, s.n)
		verdict := "pass"
		if lo < s.floor {
			verdict = "BLOCK"
			ok = false
		}
		fmt.Printf("%-16s %4.0f/%-4.0f %7.1f%% %7.1f%%–%5.1f%% %6.0f%%  %s\n",
			s.name, s.passed, s.n, 100*s.passed/s.n, 100*lo, 100*hi, 100*s.floor, verdict)
	}
	if ok {
		fmt.Println("\nrelease: allowed")
	} else {
		fmt.Println("\nrelease: blocked — a slice is too small or too weak to be sure")
	}

	fmt.Println("\ncases needed for the interval to be about ±2 points around 97%:")
	for _, n := range []float64{50, 100, 200, 400, 800} {
		lo, hi := wilson(math.Round(0.97*n), n)
		fmt.Printf("  n=%4.0f  %.1f%%–%.1f%%\n", n, 100*lo, 100*hi)
	}
}

The overall rate passes comfortably. The handwritten and non-English slices score above 93% and are still blocked, because thirty cases cannot prove a 90% floor. The fix is not a better model. It is more cases — which is exactly the kind of finding an architect should put in front of a steering group before launch rather than after.

Remember this#

• The evaluation is the contract: the business fixes the number, engineering may change anything that keeps it.
• Real cases, including the ones that should be refused. Versioned, owned, refreshed.
• Grade with code where possible, a validated model judge where not, people for the sample.
• Gate on slices, on regression against production, and on the interval — not the headline.
• Keep evaluating after launch. New failures become new cases.

Try it#

1. Run gate.go. How many handwritten cases, at the same pass rate, would be needed for that slice to clear its floor? Change the numbers until it does.
2. Write a three-question binary rubric for a free-text task you know. Grade ten outputs yourself, then have a colleague grade the same ten. Where do you disagree?
3. For the gate above, write one sentence per threshold explaining where the number came from. Any you cannot explain should go back to the business owner.

Check yourself#

1. In what sense is an evaluation a contract?
2. Why must a model judge be checked against human grades before it is trusted?
3. A candidate scores 97% overall and 82% on one slice. What do you do?
4. Why does 97 out of 100 not establish a 97% pass rate?