The Design Review

The idea in one minute#

A design review asks the same questions of every AI design, in the same order, and treats an unanswered question as a finding. This lesson gives you the question list — thirty questions in six groups that mirror the design method — the most common findings, and six practice problems to work alone. Use the list on your own design before anyone else sees it; it is quicker than being asked in the meeting.

A picture#

flowchart LR
  D[":i-file-text: <b>Design document</b>"] --> Q1[":i-list-checks: <b>Requirements</b><br/><small>unit of work, quality bar</small>"]
  Q1 --> Q2[":i-gauge: <b>Numbers</b><br/><small>tokens, peak, occupancy</small>"]
  Q2 --> Q3[":i-workflow: <b>Architecture</b><br/><small>state, routes, context</small>"]
  Q3 --> Q4[":i-triangle-alert: <b>Failure</b><br/><small>slow, wrong, runaway</small>"]
  Q4 --> Q5[":i-coins: <b>Cost</b><br/><small>per unit, levers</small>"]
  Q5 --> Q6[":i-shield-check: <b>Security</b><br/><small>sources, sinks, blast radius</small>"]
  Q6 --> OUT{":i-scale: <b>Outcome</b>"}
  OUT -->|"no open blockers"| GO[":i-check: <b>Build the first stage</b>"]
  OUT -->|"blockers"| BACK[":i-recycle: <b>Revise</b>"] --> D
  class D neutral
  class Q1,Q2,Q3 io
  class Q4,Q6 warn
  class Q5,OUT queue
  class GO memory
  class BACK neutral

How it really works#

The thirty questions#

Requirements

1. What is one unit of work, and what is it worth?
2. What is the quality bar, as a number on a named dataset?
3. Which calls are interactive and which can wait?
4. What may the system read, and what may it change?
5. What happens, concretely, when it is wrong?

Numbers

6. What are peak input and output tokens per second, per model?
7. How many model calls per unit of work, and how does context grow across them?
8. What cache hit rate is assumed, and what makes it true?
9. For self-hosted models: what measured goodput per replica, at what latency target?
10. What occupancy is planned, and what is the capacity on the bad day?

Architecture

11. Is there one gateway in front of every model, and does any code bypass it?
12. Where does each kind of state live, and can any model replica serve the next call?
13. What exactly is in the context for each call, in what order, with what budget?
14. Which model serves which call site, and what evidence chose it?
15. Is every behaviour-changing artifact versioned and gated by evaluation?

Failure

16. What are the first-token and total deadlines, and what happens when each passes?
17. What is the fallback for each model route, and has its prompt been evaluated?
18. What is shed first under overload?
19. What stops a runaway loop, and where is that enforced?
20. Does a crash or deploy lose task progress or repeat a side effect?

Cost

21. What does one unit of work cost, and what are the two largest terms?
22. What halves each of those terms?
23. Who sees the bill, per tenant, and what enforces a budget?

Security

24. Which text entering any context is controlled by someone else?
25. What can model output cause, classified as read, write, send, execute?
26. Does any single context hold private data, untrusted content and an outbound channel?
27. Whose authority does each tool call carry, how is it scoped, how long does it live?
28. Where does model-written code run, and what can it reach?
29. What is the worst outcome of a fully manipulated agent, in one sentence?
30. How would you know it had happened?

The findings that recur#

Scoring a design#

A review ends with one of three outcomes:

• Blockers — must change before building: no evaluation plan, an unmitigated trifecta, an unbounded loop, lost progress on crash, permissions enforced by instruction.
• Risks — accepted knowingly, with an owner: a single provider, an untested scale point, a manual step.
• Notes — improvements for later.

Record them in the design document. A design with written, owned risks is in better shape than one with no findings because nobody looked.

Practice problems#

Work each through the six steps and the one-page template from The Design Method. Each takes about an hour. The line after each names the part most people get wrong.

1. A code-review agent for a 2,000-engineer company: comments on every pull request within five minutes. Untrusted input: the diff itself, including contributions from outside.
2. A customer-support agent that can look up orders and issue refunds up to a limit. Refund is an irreversible effect triggered by customer-written text.
3. A meeting assistant that transcribes calls, writes summaries and files follow-up tasks. Anything said in the meeting — or shown on a shared screen — reaches the context.
4. A self-hosted chat service for a hospital, on premises, 3,000 clinicians. Capacity on a fixed GPU budget; no overflow route; data residency.
5. A research agent that browses the web and writes reports with citations. What it may hold in memory, and what its browser sessions are logged into.
6. A multi-tenant agent-hosting product: customers upload agents that run on your infrastructure. Hostile tenants; isolation of execution, caches and model access.

For problem 6, the security treatment is worked in Securing an Agent Platform.

Presenting a design#

Whether in a review meeting or an interview, the order that works:

1. Restate the problem and the unit of work in two sentences.
2. Ask the requirement questions you cannot assume — quality bar, latency, permissions.
3. Do the numbers aloud, in tokens. State assumptions; round freely.
4. Draw the request path, then where state lives, then the control plane.
5. Walk one request through the diagram.
6. Break it: slow, wrong, runaway, hijacked. Say what happens for each.
7. Price it per unit of work and name the biggest lever.
8. Mark the trust boundaries and state the blast radius.
9. Say what you would build first.

Naming tools is the least important part. A reviewer wants to know that you can state what a box must do and how it fails; which project fills it follows from that.

Remember this#

• Thirty questions in six groups; an unanswered question is a finding.
• The recurring blockers: no evaluation, state in the process, unbounded loops, permissions by instruction, an unnoticed trifecta.
• Outcomes are blockers, owned risks and notes — written down.
• Present in the order of the method; tools last.

Try it#

1. Run the thirty questions against a design you have shipped. Count the ones you cannot answer today.
2. Work practice problem 2 on one page.
3. Exchange designs with a colleague and review each other’s with the findings table.

Check yourself#

1. Which five findings should always be blockers?
2. Why is an untested fallback route a finding even though it “exists”?
3. In what order would you present a design, and why are the numbers before the diagram?

You have finished the course. Continue with AI Security Engineering for the security half in depth, or go down a layer into Inference Engineering to see what happens inside the serving plane.