A Worked Engagement

The idea in one minute#

This lesson runs one engagement from the first phone call to the decision after the pilot, using every tool in the course in the order a real project uses them. The customer is a mid-sized insurer. The request is “we want AI agents to settle claims automatically”. The recommendation, eight weeks later, is something smaller, cheaper and far more likely to succeed — and the lesson shows the reasoning and the numbers behind the difference.

Read it twice. First for the story: what happened each week and what was produced. Then for the judgement: at each step, ask what you would have done, and what the architect chose not to do. Most of the value of this job is in the options that were closed early.

The figures are illustrative. The arithmetic is real, and the program at the end reruns it.

A picture#

flowchart LR
  W0[":i-message-square: <b>Week 0</b><br/><small>“agents to settle claims”</small>"] --> W1[":i-search: <b>Weeks 1–2</b><br/><small>discovery: shadowing, data, people</small>"]
  W1 --> W3[":i-funnel: <b>Week 3</b><br/><small>six ideas scored; two chosen</small>"]
  W3 --> W4[":i-layers: <b>Weeks 4–6</b><br/><small>requirements, options, design, cost</small>"]
  W4 --> W7[":i-shield-check: <b>Week 7</b><br/><small>review: security, legal, well-architected</small>"]
  W7 --> W8[":i-building-2: <b>Week 8</b><br/><small>steering group approves a pilot</small>"]
  W8 --> PIL[":i-users: <b>Weeks 9–23</b><br/><small>prototype, evaluation build, pilot</small>"]
  PIL --> DEC[":i-scale: <b>Week 24</b><br/><small>gate report and decision</small>"]
  class W0 neutral
  class W1,W3 queue
  class W4 compute
  class W7 warn
  class W8,DEC io
  class PIL memory

How it really works#

Week 0 — the request#

The head of claims has seen a vendor demonstration and wants “AI agents that settle property claims end to end”. The board has asked every division for an AI plan. There is budget for this year and impatience.

The architect does not argue with the request. The reply is: “Give me two weeks with your team before I propose anything. I will come back with what is worth doing first and what it would cost.” The technique is the walk-back from Discovery.

Weeks 1–2 — discovery#

Three days sitting with handlers, six interviews, a sample of 200 closed claims.

FINDINGS
Work        120 handlers · ~900 property claims/day · 25 min reading per claim
            Decision takes 9 days; service standard is 5
Where time  60% reading documents to find ~14 facts
goes        25% waiting for missing documents
            15% deciding and writing
Data        Documents in one platform with an API. Two years of decided claims.
            22% of documents are photos or handwriting.
            Policy wording: 12,000 PDFs, updated weekly.
People      Sponsor wants speed. Handlers fear monitoring. Compliance has not been asked.
            Document platform owner worried about load.
Surprise    Two experienced handlers agree on extracted facts 94% of the time.
Not AI      A quarter of the delay is missing documents — a form change fixes most of it.

Two findings already reshape the project. The delay is mostly reading, not deciding — so the valuable thing is extraction, not settlement. And a quarter of the delay needs no AI at all.

Week 3 — ideas, gates and scores#

A half-day workshop produces six ideas. The scorecard from Qualifying and Prioritising Use Cases sorts them.

The sponsor’s original idea is not rejected. It is placed: not now, and here is the path that leads to it. Extraction is the first step on that path, because a system that cannot read a claim reliably cannot settle one.

Week 4 — requirements#

From Requirements and Success Metrics, agreed line by line with claims operations:

Business   Days to decision 9 → 5 · reading time per claim 25 → ≤ 6 min
Task       Field accuracy ≥ 97% on answered fields · abstention ≤ 15%
           Wrong payout amount ≤ 0.5% · every field linked to its source page
System     Extraction complete ≤ 60 s p95 · cost per claim ≤ $0.08 median
Oversight  Handler confirms every field. The handler decides every claim.
Non-goals  No decisions. No customer contact. Property line only.

The 97% target was set after learning that people agree 94% of the time. Without that measurement the business would have asked for 99.9%.

Week 5 — options#

Three ways to deliver, compared honestly. The pattern for the recommended option is a workflow, not an agent: classify the documents, extract the facts with a schema, look up the policy clause, present for confirmation. Every step can be written down, so by The Pattern Ladder nothing higher is justified.

Option C has the largest benefit on paper and the worst expected outcome, once its cost, its delay and its odds are counted. The program at the end makes that explicit.

Week 6 — the design#

flowchart LR
  DP[(":i-archive: <b>Document platform</b><br/><small>claims PDFs, photos</small>")] --> ING[":i-workflow: <b>Extraction workflow</b><br/><small>classify → extract → validate</small>"]
  ING --> GW[":envoyproxy: <b>Model gateway</b><br/><small>small model for simple, mid for complex</small>"]
  GW --> M[":i-brain: <b>Hosted models</b><br/><small>in region, no retention</small>"]
  ING --> KS[":i-search: <b>Policy search</b><br/><small>hybrid + rerank, existing service</small>"]
  ING --> ST[(":postgresql: <b>Extracted facts</b><br/><small>with source page links</small>")]
  ST --> UI[":i-monitor: <b>Handler screen</b><br/><small>field beside highlighted source</small>"]
  UI --> H[":i-user: <b>Handler</b><br/><small>confirms, decides</small>"]
  H -.->|"corrections"| EV[":i-flask-conical: <b>Evaluation set</b>"]
  ING -.-> OB[":opentelemetry: <b>Traces, cost, quality</b>"]
  class DP,ST memory
  class ING,KS compute
  class GW queue
  class M io
  class UI,H neutral
  class EV,OB compute

Eight decisions recorded as ADRs. The four that mattered most:

Week 6 — cost and business case#

From Cost, TCO and ROI:

Unit cost      $0.072 per claim naive → $0.024 with routing, caching and batch
One-off        $570k   (build $450k + rollout and training $120k)
Recurring      $556k / year — of which model tokens are about $6k
Benefit        $3.2M / year gross time saved × 50% realised × 80% adoption = $1.28M
Payback        about 9 months        3-year ROI  about 72%
Sensitive to   Adoption and realisation. Not to token price.

The sentence the sponsor remembered: “The model costs six thousand dollars a year. The system costs half a million. Plan and staff it accordingly.”

Week 7 — reviews#

Security and legal were met in week 1, so week 7 confirms rather than discovers.

Week 8 — the decision#

One page, from The Solution Design Document. The ask: approve a pilot of option B with fifteen handlers and three written gates. The steering group approves, and asks one question — “when do we get the agents?” The answer: “When this system has read a hundred thousand claims correctly, we will have the evaluation data to consider automating the simplest ones. That is phase three, and you will decide it on evidence.”

Weeks 9–23 — prototype, evaluation build, pilot#

Gate 1 passed in week 11 with 93% accuracy on fifty random claims and token cost 1.4 times the estimate. The evaluation set of 412 claims was agreed in week 14. The pilot ran twelve weeks.

GATE 2 REPORT — week 24

Field accuracy, all cases          97.6%   (interval 95.6–98.7)      target ≥ 97     PASS
  simple claims                    98.3%                                             PASS
  complex claims                   95.8%                                             PASS
  handwritten or photographed      93.3%   (30 cases — too few to be sure)           BLOCKED
Wrong payout amount                0.3%                              target ≤ 0.5    PASS
Abstention                         11%                               target ≤ 15     PASS
Reading time per claim             8.5 min (from 25)                 target ≤ 6      MISS
Adoption, final four weeks         13 of 15 handlers on > 80% of claims              PASS
Cost per claim, median             $0.031                            target ≤ 0.08   PASS
Severity-1 incidents               0                                                 PASS
Fallback model evaluated           yes, 96.9%                                        PASS

Week 24 — reading the result#

Two lines did not pass, and how the architect handles them is the lesson.

Handwritten documents. Blocked not because the score was bad but because thirty cases cannot prove a 90% floor. The decision: those documents are routed to the manual path, and a further 150 handwritten cases are collected. Automation of that slice waits for evidence.

Reading time. 8.5 minutes, not 6. Investigation of the traces showed handlers re-reading whole documents because the highlighted source was shown on a separate screen. This is an interface problem, not a model problem — and no model upgrade would have fixed it. A redesign puts the source beside the field.

The recommendation to the steering group: proceed to limited release for typed documents only, with the screen change, and a revised business case at 8.5 minutes — a payback of about a year instead of nine months. The group approves. A project with a missed target and an honest explanation keeps its sponsor; one with a hidden miss does not.

What the architect did not do#

• Did not start with the agent the sponsor asked for.
• Did not build a vector database because retrieval was involved.
• Did not accept “as accurate as possible” as a requirement.
• Did not choose the model by leaderboard — two were tested on the customer’s own claims.
• Did not leave security and legal until the design was finished.
• Did not round the reading-time result up to a pass.

Code#

The three options compared the way a steering group should see them: expected three-year outcome, weighted by the chance of reaching production and reduced by the months before any benefit arrives.

// options.go — expected three-year outcome of three delivery options.
package main

import "fmt"

type option struct {
	name          string
	once, run     float64 // one-off cost; running cost per year
	benefit       float64 // realised benefit per year once live
	monthsToValue float64
	pSuccess      float64 // chance of reaching production
}

func main() {
	const horizon = 36.0 // months
	opts := []option{
		{"A buy a product", 300_000, 560_000, 880_000, 5, 0.80},
		{"B build assisted extraction", 570_000, 556_000, 1_280_000, 8, 0.75},
		{"C build settlement agent", 1_600_000, 1_100_000, 3_400_000, 18, 0.25},
	}

	fmt.Printf("%-30s %12s %12s %12s\n", "option", "if it works", "if it fails", "expected")
	for _, o := range opts {
		live := horizon - o.monthsToValue
		// Success: full cost for three years, benefit for the months it is live.
		win := o.benefit*live/12 - (o.once + o.run*horizon/12)
		// Failure: cancelled after one year of running cost, no benefit.
		lose := -(o.once + o.run)
		expected := o.pSuccess*win + (1-o.pSuccess)*lose
		fmt.Printf("%-30s %12.0f %12.0f %12.0f\n", o.name, win, lose, expected)
	}

	fmt.Println("\nchance of success option C needs to match option B's expected outcome:")
	b, c := opts[1], opts[2]
	winB := b.benefit*(horizon-b.monthsToValue)/12 - (b.once + b.run*horizon/12)
	expB := b.pSuccess*winB - (1-b.pSuccess)*(b.once+b.run)
	winC := c.benefit*(horizon-c.monthsToValue)/12 - (c.once + c.run*horizon/12)
	loseC := -(c.once + c.run)
	fmt.Printf("  %.0f%%  (estimated today: %.0f%%)\n", 100*(expB-loseC)/(winC-loseC), 100*c.pSuccess)
}

Option C barely breaks even inside three years even when it works, because it spends half the horizon being built; its expected outcome is a large loss, and the last line shows that no realistic chance of success closes the gap — the figure it prints is above 100%. Option B has the best expected value and a survivable worst case. And B produces the evaluation data that would raise C’s odds — which is the real argument for doing them in that order.

Remember this#

• The request was agents; the problem was reading. Discovery found the difference in two weeks.
• The sponsor’s idea was placed on a path, not rejected.
• The accuracy target came from measuring people first.
• Weigh options by expected outcome: benefit × chance of success, minus cost and delay.
• A missed gate, explained with evidence, keeps trust. A hidden one ends the project.

Try it#

1. Run options.go. Extend the horizon to 60 months. Does option C now make sense, and what chance of success would it need? What evidence would move its odds there?
2. Rewrite the week-8 one-page summary for this engagement yourself, without looking back at the design-document lesson. Compare.
3. Run the same eight-week structure on a real request at your work. Stop at week 3 and present the scorecard to someone who can say no.

Check yourself#

1. What two discovery findings changed the direction of the project?
2. Why was a workflow chosen rather than an agent?
3. The handwritten slice scored 93.3% against a 90% floor and was still blocked. Why?
4. Reading time missed its target. What was the cause, and why would a better model not have helped?