Well-Architected AI

The idea in one minute#

A well-architected review is a structured way to find the weaknesses in a design before production finds them. Each of the three large cloud providers publishes a framework built on the same idea: examine the workload through a fixed set of pillars — operational excellence, security, reliability, performance, cost, and sustainability — and for each one answer a list of questions honestly.

All three now have AI-specific versions. AWS publishes a Generative AI Lens, Microsoft a set of AI workload guidance in its Well-Architected Framework, and Google an AI and ML perspective. They agree far more than they differ, because the weaknesses of AI systems are the same on every cloud.

This lesson reduces them to thirty questions an architect can ask of any AI design in ninety minutes. The value of the review is not the score. It is the short list of findings, each with an owner, that you leave with.

A picture#

flowchart LR
  D[":i-file-text: <b>The design</b><br/><small>document, diagrams, ADRs</small>"] --> R
  subgraph R["Review through six pillars"]
    direction TB
    P1[":i-activity: <b>Operational excellence</b><br/><small>evaluate, observe, change safely</small>"]
    P2[":i-shield-check: <b>Security</b><br/><small>data, identity, agency</small>"]
    P3[":i-recycle: <b>Reliability</b><br/><small>quotas, fallback, degraded mode</small>"]
    P4[":i-gauge: <b>Performance</b><br/><small>latency, quality, retrieval</small>"]
    P5[":i-coins: <b>Cost</b><br/><small>unit cost, attribution</small>"]
    P6[":i-globe: <b>Sustainability</b><br/><small>right-sized models</small>"]
  end
  RA[":i-scale: <b>Responsible AI</b><br/><small>across all six</small>"] -.-> R
  R --> F[":i-list-checks: <b>Findings</b><br/><small>high / medium risk, owner, date</small>"]
  F --> B[":i-git-branch: <b>Backlog and ADR updates</b>"]
  class D neutral
  class P1,P4 compute
  class P2 warn
  class P3,P5 queue
  class P6 memory
  class RA warn
  class F,B io

How it really works#

The three frameworks#

Read one closely — the one for the cloud you work on — and skim the other two. The ideas transfer completely; only service names change.

What AI adds to each pillar#

Classic well-architected questions still apply: backups, least privilege, autoscaling. Below is what a model in the request path adds.

Operational excellence — can we change it safely and see when it degrades?

1. Is there an evaluation set, and does a release require passing it?
2. Are prompts, model versions and configuration in version control, deployable and revertible independently of code?
3. Is every model and tool call traced, with inputs, outputs, tokens and latency?
4. Is output quality monitored in production, not only uptime?
5. Is there a path for user feedback to reach the evaluation set?
6. Is there a runbook for “quality dropped” as well as for “service down”?

Security — what can it see, what can it do, and who is it acting as?

1. Does retrieval respect the asking user’s permissions?
2. Is all untrusted content — documents, web pages, tool results — treated as data that may contain instructions?
3. Are tools scoped to least privilege, with approval before irreversible actions?
4. Is it known where prompts and outputs go, how long they are kept, and whether they are used for training?
5. Are model endpoints authenticated, private where required, and rate-limited per tenant?
6. Are inputs and outputs screened for sensitive data and harmful content, in both directions?

Reliability — what happens at peak, and when the provider fails?

1. Is peak token demand calculated against the quota actually granted?
2. Is there a fallback model that has passed the evaluation?
3. Are timeouts, retries with backoff, and circuit breakers defined for model calls?
4. Is there a degraded mode — a simpler answer, a queue, a manual path?
5. Are prompts, indexes and evaluation sets versioned so a release can be rolled back?
6. For long tasks: is state checkpointed so work resumes rather than restarts?

Performance efficiency — is it fast enough and good enough, measured?

1. Are latency targets stated as percentiles and measured in the target region?
2. Is the smallest model that passes the evaluation the one in use?
3. Is retrieval quality measured separately from answer quality?
4. Are responses streamed where a person is waiting?
5. Is context trimmed deliberately — history, tool results, retrieved passages?

Cost optimisation — do we know the unit cost and who pays?

1. Is cost attributed per use case, team and tenant?
2. Are caching, batching and routing used where they apply?
3. Are there budgets and alerts, and hard caps on agent runs?
4. Has provisioned capacity been checked against real utilisation?
5. Is development and evaluation spend visible separately from production?

Sustainability — are we using no more compute than the task needs?

1. Is model size matched to the task?
2. Is unnecessary generation avoided — cached answers, capped output, no redundant calls?

Responsible AI runs across all six#

Every framework treats responsible AI as a cross-cutting concern rather than a seventh pillar. The questions an architect asks:

• Who is affected by this system’s output, including people who never use it?
• Is it fair across the groups it will serve, and was that tested?
• Is it transparent — do people know they are dealing with AI, and can they see sources?
• Is a person accountable for its decisions, by name?
• Can an affected person contest an outcome and reach a human?

These become legal obligations for some systems; see Governance and Regulation.

Shared responsibility#

A recurring theme in all three frameworks: responsibility is split between whoever makes the model, whoever hosts it and whoever uses it in an application. You are almost always the third.

The practical meaning: “the provider’s model is safe” says nothing about whether your application is. The bottom five rows are your review.

Trade-offs are the review#

Pillars pull against each other. A design that scores perfectly on all of them has not been examined closely. Name the trade and who accepted it.

How to run a review#

Ninety minutes, the right five people, one document.

1. Before. Circulate the design document and diagrams. Ask each attendee to mark the questions they think the design fails.
2. Who attends. The architect, the lead engineer, someone from security, someone who will operate it, and the product owner. Not the sponsor.
3. In the room. Go pillar by pillar. For each question the answer is yes with evidence, no, or not applicable with a reason. “We plan to” is a no.
4. Record findings, not discussion. Each finding has a risk level, an owner and a date.
5. After. High-risk findings block the next gate. Update the ADRs. Schedule the next review for the next phase.

FINDINGS — Claims extraction, review of 9 Oct 2026

#  Pillar       Finding                                           Risk   Owner     Due
1  Reliability  No fallback model has been evaluated              High   Eng lead  23 Oct
2  Security     Extraction prompt can be steered by text in a     High   Security  23 Oct
                submitted document; output not schema-validated
3  Ops          Quality is monitored only by handler complaints   Med    Architect 6 Nov
4  Cost         Evaluation runs not tagged separately             Low    Eng lead  20 Nov

When to review#

Remember this#

• Six pillars, the same on every cloud: operations, security, reliability, performance, cost, sustainability — with responsible AI across all of them.
• AI adds specific questions to each: evaluation gates, permission-aware retrieval, quotas and fallback, measured latency, unit cost.
• You own everything above the model: prompts, tools, data, access, quality, compliance.
• A review produces findings with owners and dates. “We plan to” counts as no.
• Trade-offs between pillars are expected. Write down which was chosen and by whom.

Try it#

1. Take a design you wrote in the previous topic. Answer the thirty questions with yes, no or not applicable. How many are “no”?
2. Open the AI guidance of one cloud’s well-architected framework. Find one question on each pillar that is not on this page, and decide whether it applies to your design.
3. Pick two pillars that conflict in your design. Write one sentence stating which you favoured and who agreed.

Check yourself#

1. Name the six pillars and one AI-specific question for each.
2. In the shared responsibility table, which rows belong to you?
3. Why is “we plan to add that” recorded as a no?
4. Give one example of a trade-off between two pillars.

Sources#

• AWS Well-Architected Framework — Generative AI Lens (published 19 November 2025)
• Azure Well-Architected Framework — design principles for AI workloads
• Google Cloud Well-Architected Framework — AI and ML perspective