Requirements and Success Metrics

The idea in one minute#

Ordinary software either does what the specification says or has a bug. An AI system is right some percentage of the time, and the central requirement is therefore a question no traditional specification asks: how often may it be wrong, in which ways, and what happens then?

Requirements for an AI solution have three layers. Business metrics say why the project exists — days to decision, cost per case. Task metrics say whether the AI part works — accuracy, groundedness, refusal rate, each with a pass mark. System metrics say whether it is usable — latency, availability, cost per task.

Every one is a number with a threshold, a way to measure it, and an owner. A requirement without a threshold — “it should be accurate” — is a wish. Getting the business to commit to the number, before the build, is one of the hardest and most valuable things an architect does.

A picture#

flowchart TB
  B[":i-building-2: <b>Business metric</b><br/><small>decision in 5 days, not 9</small>"] --> T[":i-flask-conical: <b>Task metrics</b><br/><small>≥ 97% of fields correct<br/>100% of answers cite a source</small>"]
  B --> S[":i-gauge: <b>System metrics</b><br/><small>p95 under 8 s · 99.5% available<br/>under $0.10 per claim</small>"]
  T --> E[":i-list-checks: <b>Evaluation set</b><br/><small>real cases with right answers</small>"]
  S --> L[":i-activity: <b>Load test and monitoring</b>"]
  E --> G[":i-check: <b>Release gate</b><br/><small>all thresholds met</small>"]
  L --> G
  G -.->|"measured after launch"| B
  F[":i-triangle-alert: <b>Failure behaviour</b><br/><small>what happens when it is wrong or down</small>"] --> G
  class B neutral
  class T,E compute
  class S,L queue
  class F warn
  class G memory

How it really works#

Three layers of metric#

A project can pass the lower two layers and fail the first — a fast, accurate assistant nobody uses. Write all three, and name who owns each.

Turning “accurate” into a number#

Ask the question from discovery again: “what would have to be true for you to trust the result without checking it?” Then work it into a threshold in four steps.

1. Define the unit. Accuracy of what — a field, a document, a whole case?
2. Find today’s baseline. How often are people wrong at the same task? Almost nobody knows, and finding out often changes the conversation. If two experienced handlers agree with each other 94% of the time, a 99% target is asking the machine to beat a standard no human meets.
3. Price the errors. Not all mistakes cost the same. Separate them.
4. Set the threshold per error type, and decide what happens below it.

Errors are not equal#

Every task has at least two ways to be wrong, and the business nearly always cares more about one.

Design follows from this. When a confident wrong answer is the expensive error, require the system to abstain — hand the case to a person — whenever it is unsure, and accept a lower automation rate. Two numbers then replace one: how often it answers, and how often it is right when it does.

System requirements that AI changes#

Always write latency and cost as percentiles — the median and the 95th. An average hides the slow tail, and the slow tail is what users remember.

Failure behaviour is a requirement#

Decide in advance what the system does when it cannot do the job. For each of these, the requirements document must have a sentence.

• The model is unsure → hand off to a person, with what it found so far.
• The provider is down or slow → fall back to a second model, or queue, or show the manual path.
• The input is out of scope → say so plainly; do not improvise.
• The output fails a check → retry once, then hand off.
• The budget for this request is exhausted → stop, return partial work, log it.

Human oversight, stated precisely#

“Human in the loop” means nothing until four things are specified.

A reviewer who must redo the whole task to check the result saves nothing. A reviewer shown the extracted value beside the highlighted source can check it in seconds. That design detail often decides whether the project pays back.

Non-goals#

List what the system will deliberately not do. Non-goals stop scope creeping in the build and stop over-claiming in the launch announcement.

NON-GOALS
- The system does not decide claims. A handler decides every claim.
- It does not handle motor or life claims in phase 1.
- It does not answer customers directly.
- It does not learn from handler corrections automatically; corrections feed the evaluation set.

The requirements page#

REQUIREMENTS — Claims document extraction                          v1.0

BUSINESS   Median days to decision          9 → 5        owner: Head of Claims
           Reading time per claim           25 → ≤ 6 min owner: Claims Ops

TASK       Field accuracy (answered fields) ≥ 97%        eval set v1, 400 claims
           Abstention rate                  ≤ 15% of fields
           Every field linked to its source 100%
           Wrong amount on a payout field   ≤ 0.5%       separate threshold
           Human baseline (two handlers)    94% agreement — measured, for reference

SYSTEM     p95 time to extracted claim      ≤ 60 s (background, not interactive)
           Availability, business hours     99.5%
           Cost per claim                   median ≤ $0.08, p95 ≤ $0.30
           Peak throughput                  150 claims in 10 minutes

FAILURE    Low confidence → field left blank and flagged
           Provider outage → queue up to 2 h, then manual path
OVERSIGHT  Handler confirms every extracted field before decision; target ≤ 3 min

Code#

Should a person check everything, a sample, or nothing? The answer is arithmetic: the cost of review against the expected cost of the errors it would catch.

// oversight.go — expected cost per case under four ways of working.
package main

import "fmt"

type mode struct {
	name      string
	minutes   float64 // handler time per case
	errorRate float64 // share of cases with an error that reaches the decision
	modelCost float64 // dollars per case
}

func main() {
	const (
		hourly    = 45.0  // loaded cost of a handler, dollars per hour
		errorCost = 400.0 // rework, complaint handling, possible wrong payout
		perDay    = 900.0
	)

	// Error rates are what reaches the decision AFTER any review.
	modes := []mode{
		{"people only (today)", 25, 0.020, 0},
		{"AI extracts, person checks every field", 6, 0.012, 0.08},
		{"AI extracts, person checks flagged fields", 3, 0.025, 0.08},
		{"AI extracts, nobody checks", 0, 0.060, 0.08},
	}

	fmt.Printf("%-44s %8s %8s %8s %10s\n", "mode", "labour", "errors", "model", "per case")
	base := 0.0
	for i, m := range modes {
		labour := m.minutes / 60 * hourly
		errs := m.errorRate * errorCost
		total := labour + errs + m.modelCost
		if i == 0 {
			base = total
		}
		fmt.Printf("%-44s %8.2f %8.2f %8.2f %10.2f   saves $%.0f/day\n",
			m.name, labour, errs, m.modelCost, total, (base-total)*perDay)
	}

	// At what error cost does removing the check stop being cheapest?
	fmt.Println("\nerror cost at which 'nobody checks' matches 'checks flagged fields':")
	labourSaved := 3.0 / 60 * hourly
	extraErrors := 0.060 - 0.025
	fmt.Printf("  $%.0f per error (above this, keep the check)\n", labourSaved/extraErrors)
}

With these inputs the cheapest way of working is the second: the model extracts and a person checks every field. Unchecked automation is the most expensive of the three AI modes, because each error costs $400. Removing the check would only win if an error cost less than about $64 — and for a regulated decision the comparison is settled by law before it is settled by arithmetic. The point is that “how much oversight” has an answer you can compute and defend.

Remember this#

• Three layers: business, task, system. Each metric has a threshold, a method and an owner.
• Measure the human baseline before setting an accuracy target.
• Errors are not equal. Set a threshold per error type, and let the system abstain.
• Latency and cost are percentiles, never averages.
• Failure behaviour, oversight and non-goals are requirements, not afterthoughts.

Try it#

1. Run oversight.go. Lower errorCost to 50. Which mode is now cheapest? Raise it to 2,000. What does that tell you about where full automation is reasonable?
2. For a task you know, name its two kinds of error and estimate the cost of each. Which should the system avoid, and what should it do when unsure?
3. Write the requirements page for that task. Leave no metric without a number, even if the number is “to be measured in week 2”.

Check yourself#

1. Why is “it should be accurate” not a requirement?
2. Why measure how often people disagree with each other before setting a target?
3. What two numbers replace “accuracy” when a system is allowed to abstain?
4. What four things must be specified for “human in the loop” to mean something?