AI System Design

Learn to design systems that have a model in the middle: what to draw on the whiteboard, which numbers to compute before you draw it, which tools fill each box in 2026, and how the design changes when the model stops answering questions and starts taking actions.

Classic system design teaches load balancers, queues, caches and databases. All of that still applies. An AI system adds a component that is slow, expensive, probabilistic, priced by the token and easily talked into things — and the design has to be built around those five facts.

This course starts at “what is different when there is a model in the request path?” and ends with complete designs: the infrastructure for an agentic AI platform, drawn box by box with the real projects named, sized with arithmetic you can rerun, and reviewed for failure and attack.

It assumes you can program in Go and have seen an ordinary web backend. It does not assume you have run a GPU, built a RAG pipeline or written an agent.

What a finished design looks like#

flowchart LR
  U[":i-users: <b>Users and apps</b>"] --> GW[":envoyproxy: <b>AI gateway</b><br/><small>auth, token limits, routing</small>"]
  GW --> AG[":i-bot: <b>Agent runtime</b><br/><small>loop, state, checkpoints</small>"]
  AG --> GW2[":envoyproxy: <b>Model route</b>"]
  GW2 --> API[":anthropic: <b>Hosted models</b><br/><small>frontier APIs</small>"]
  GW2 --> SELF[":vllm: <b>Self-hosted models</b><br/><small>vLLM on GPUs</small>"]
  AG --> MCP[":modelcontextprotocol: <b>Tools over MCP</b><br/><small>search, code, tickets</small>"]
  AG --> SB[":i-box: <b>Sandboxes</b><br/><small>gVisor, microVMs</small>"]
  AG --> MEM[(":postgresql: <b>State and memory</b>")]
  SELF --> K8S[":kubernetes: <b>GPU cluster</b>"]
  AG -.-> OBS[":opentelemetry: <b>Telemetry and evals</b>"]
  class U neutral
  class GW,GW2 queue
  class AG,SELF compute
  class API,MCP io
  class SB warn
  class MEM,K8S memory
  class OBS neutral

Every box in that picture is a lesson. By the end you can defend each arrow.

How this course works#

Every lesson follows the same shape:

1. The idea in one minute — the whole lesson in a few sentences.
2. A picture — the mechanism, with the real tools drawn in. Press Expand on any diagram to open it full size.
3. How it really works — the precise version, with names and numbers.
4. Code — where arithmetic or a mechanism is worth running, a small Go program that uses only the standard library.
5. Remember this, Try it and Check yourself.

Designs are always worked in the same order: requirements → numbers → architecture → failure → cost → security. Tools are chosen last. A tool named in a diagram is an example of what fills the box, not an endorsement; the tables say what else fits.

The topics#

flowchart LR
  A["How AI systems differ"] --> B["Building blocks"]
  B --> C["AI infrastructure<br/>from scratch"]
  C --> D["Agentic systems"]
  D --> E["Security by design"]
  E --> F["Worked designs"]
  class A neutral
  class B,C compute
  class D queue
  class E warn
  class F memory

How it connects to the other courses#

This course stays at the level of boxes, arrows and arithmetic. When a box deserves a whole course, it has one:

• What happens inside the “self-hosted models” box: Inference Engineering.
• What the GPUs under it are: GPU Engineering.
• How to see any of it running: Observability Engineering.
• How to keep it from being turned against you: AI Security Engineering.

What you need#

• Go 1.22 or newer and a terminal. Every program runs offline.
• No GPU, no cloud account and no API key.

A promise about names and versions#

Protocol versions, project statuses and release dates in this course were checked on 4 October 2026, and the lessons that depend on them list their sources. The field moves monthly. The method — requirements, numbers, architecture, failure, cost, security — does not, and each lesson separates the two so you know which half to re-check.