How AI Works, for Architects

The idea in one minute#

A large language model is a program that, given some text, predicts what text comes next. It was built by showing it an enormous amount of writing and adjusting billions of internal numbers until its predictions were good. Those numbers are then frozen. When you use the model, it is not learning and it is not looking anything up: it reads what you send, and writes a continuation, one small piece at a time.

Everything an architect needs follows from seven facts about that process. The model reads and writes in tokens; it can only see a fixed-size context window; it has no memory between calls; its knowledge stops at a date; its output varies; it can be confidently wrong; and it is billed and slowed by the token. Learn the seven and you can predict how any AI design will behave before it is built.

A picture#

flowchart LR
  subgraph CW["Context window — everything the model can see"]
    direction TB
    S[":i-scroll-text: <b>Instructions</b><br/><small>the system prompt</small>"]
    T[":i-wrench: <b>Tool descriptions</b>"]
    D[":i-file-text: <b>Documents you fetched</b>"]
    H[":i-message-square: <b>Conversation so far</b>"]
    Q[":i-user: <b>The new question</b>"]
  end
  CW --> M[":i-brain: <b>Model</b><br/><small>frozen numbers, no memory</small>"]
  M -->|"one token at a time"| O[":i-file-text: <b>Output</b><br/><small>text, or a request to use a tool</small>"]
  O -.->|"your code appends it and calls again"| H
  class S,T,D,H,Q memory
  class M compute
  class O io

The dotted arrow is the most important line. The model keeps nothing. If a conversation seems to remember, it is because your application sent the whole conversation again.

How it really works#

The words, in the order you need them#

The seven facts#

1. It works in tokens. Text is cut into tokens before the model sees it. In English one token is about four characters, or three-quarters of a word; other languages and code often use more. Prices, limits and speeds are all stated in tokens, so an architect estimates in tokens from the first conversation.

2. It sees only the context window. Each model has a maximum number of tokens it can take in at once — from a few hundred thousand to a million in 2026. That window must hold the instructions, the tool descriptions, any documents, the conversation so far, the question, and room for the answer. Bigger windows did not remove the problem: models attend less reliably to material buried in a very long context, and every token is paid for on every call.

3. It has no memory between calls. Two calls to a model are unrelated unless your application carries information from one to the next. “Memory” in an AI product is always a design decision about what to store and what to send back.

4. Its knowledge stops at a date. A model knows what was in its training data up to its cutoff, and nothing about your company. Anything recent or private has to be supplied in the context window. That single fact is why retrieval is the most common pattern in enterprise AI.

5. Its output varies. The model computes a probability for every possible next token and picks one, with some randomness. Ask twice, get two answers. You can reduce the variation; you cannot rely on removing it. Systems that need the same answer every time need rules or code around the model, not a setting inside it.

6. It can be confidently wrong. The model produces plausible text, and plausible is not the same as true. A made-up fact — a hallucination — arrives in the same fluent voice as a correct one. Good designs give the model the facts, ask it to cite them, and check.

7. It is priced and paced by the token. You pay for tokens in and tokens out, and output tokens cost several times more than input tokens. Output is also what takes time: the model writes tokens one after another, so a long answer is a slow answer.

What those facts decide#

Models come in sizes#

Every provider sells a range, and the range matters more than the brand.

On 13 October 2026 the published list prices of three large providers sat in three clear bands: roughly $0.10 in / $0.50 out per million tokens for the small tier, $2 / $10 for the mid tier and $10 / $50 for the top tier. A hundred-fold spread between the cheapest and the most capable model is the single biggest cost lever an architect has — bigger than any discount. Choosing a Model turns this into a method.

Beyond text#

• Multimodal models read images, scanned documents, audio and video as well as text. Much “document processing” that needed a separate system in 2022 is now one model call.
• Embedding models turn text into vectors for search. Small, cheap, and a separate choice from the model that writes the answer.
• Reasoning or “thinking” modes let a model work through a problem before answering. They improve hard tasks and cost extra output tokens and time.
• Traditional machine learning — forecasting, fraud scoring, recommendations on tables of numbers — is still the right tool for those jobs. An architect who reaches for a language model to predict next month’s sales has chosen wrong.

Where a model comes from, and why you rarely make one#

Pretraining      months, enormous cost, done by the provider        → a foundation model
Fine-tuning      hours to days, modest cost, optional, done by you  → a specialised copy
Inference        milliseconds to minutes, per call, done by you     → an answer

Nearly all enterprise work happens on the last line. Fine-tuning is a tool for specific situations, covered in The Pattern Ladder. Pretraining your own model is, for almost every organisation, the wrong answer to the question.

Code#

A context-window budget. It adds up what a single call sends, prices it at three model tiers, and then shows the fact that surprises most newcomers: because the whole conversation is sent again on every turn, a chat’s cost grows with the square of its length.

// budget.go — what one call sends, what it costs, and how a conversation grows.
package main

import "fmt"

type tier struct {
	name    string
	in, out float64 // dollars per million tokens
}

func main() {
	// One call of a support assistant, in tokens.
	parts := []struct {
		name   string
		tokens float64
	}{
		{"instructions", 1200},
		{"tool descriptions", 1800},
		{"retrieved documents", 6000},
		{"conversation so far", 2400},
		{"new question", 100},
	}
	window, output := 200_000.0, 500.0

	input := 0.0
	fmt.Println("one call sends:")
	for _, p := range parts {
		input += p.tokens
		fmt.Printf("  %-20s %6.0f tokens\n", p.name, p.tokens)
	}
	fmt.Printf("  %-20s %6.0f tokens  (%.1f%% of a %.0fk window)\n\n",
		"total input", input, 100*input/window, window/1000)

	tiers := []tier{{"small", 0.10, 0.50}, {"mid", 2, 10}, {"frontier", 10, 50}}
	fmt.Println("cost of that call, and of a million like it:")
	for _, t := range tiers {
		c := (input*t.in + output*t.out) / 1e6
		fmt.Printf("  %-9s $%.5f per call   $%9.0f per million calls\n", t.name, c, c*1e6)
	}

	// A conversation resends its history every turn.
	fmt.Println("\ninput tokens sent across a whole conversation (mid tier):")
	fixed, perTurn := 9000.0, 600.0 // instructions+tools+documents; one question plus one answer
	for _, turns := range []int{1, 5, 10, 20, 40} {
		total := 0.0
		for k := 0; k < turns; k++ {
			total += fixed + float64(k)*perTurn
		}
		fmt.Printf("  %2d turns  %8.0f input tokens  $%.3f\n", turns, total, total*2/1e6)
	}
}

Run it. A forty-turn conversation sends about ninety times the input of a one-turn conversation, not forty times. That is why long chats are trimmed or summarised, and why the unchanging start of a prompt is cached — providers charge roughly a tenth of the normal price to reread a prefix they have seen moments ago.

Remember this#

• A model predicts the next token from what is in its context window. Nothing else.
• No memory, a knowledge cutoff, variable output, confident errors: design around all four.
• Tokens are the unit of cost, speed and limits. Output tokens are the expensive, slow ones.
• Small, mid and frontier tiers differ about a hundred-fold in price. Use the smallest that passes.
• Language models are not the answer to prediction on numeric tables.

Try it#

1. Run budget.go. Double the retrieved documents. What happens to the cost per call on each tier, and which line of the breakdown would you cut first?
2. Paste a page of text into any provider’s token counter. How close is “four characters per token”? Try the same with a page in another language.
3. Ask an AI assistant the same factual question about your own company five times in new chats. Record how the answers differ and which are wrong. Which of the seven facts does each failure show?

Check yourself#

1. A chatbot “remembers” your name from earlier in the conversation. Where is that memory?
2. Why does a model not know about a policy your company published last week, and what is the standard fix?
3. Why is one successful demo not evidence that a system works?
4. Which tokens cost more and take longer — input or output — and what does that suggest about answer length?

Sources#

• Claude pricing, OpenAI API pricing, Gemini API pricing — list prices read on 13 October 2026