Retrieval and Knowledge

The idea in one minute#

Retrieval-augmented generation (RAG) is how a model answers from information it was never trained on. When a question arrives, the system first searches your documents or data for the relevant pieces, puts them in the context window, and asks the model to answer using only those pieces, with citations.

It has two halves that are built and owned separately. Ingestion runs in the background: collect documents, clean them, cut them into chunks, index them. Query runs when a user asks: find the best chunks, assemble the prompt, generate, cite.

For an architect the important truth is that RAG is a search and data-quality problem with a model at the end. When a RAG system gives a bad answer, the cause is nearly always that the right passage was never retrieved — or never ingested, or was out of date, or the user should not have been able to see it.

A picture#

flowchart LR
  subgraph ING["Ingestion — background, continuous"]
    direction LR
    SRC[":confluence: <b>Sources</b><br/><small>wikis, drives, tickets, databases</small>"] --> PARSE[":i-file-text: <b>Parse and clean</b><br/><small>PDF, tables, scans</small>"]
    PARSE --> CH[":i-split: <b>Chunk</b><br/><small>passages + metadata</small>"]
    CH --> IDX[(":opensearch: <b>Index</b><br/><small>keywords + vectors + permissions</small>")]
  end
  subgraph QRY["Query — per question"]
    direction LR
    Q[":i-user: <b>Question</b><br/><small>with the user's identity</small>"] --> RET[":i-search: <b>Retrieve</b><br/><small>hybrid search, filter by permission</small>"]
    RET --> RR[":i-funnel: <b>Rerank</b><br/><small>keep the best few</small>"]
    RR --> GEN[":i-brain: <b>Generate</b><br/><small>answer only from these, cite</small>"]
    GEN --> A[":i-check: <b>Answer + sources</b>"]
  end
  IDX --> RET
  class SRC io
  class PARSE,CH,RR queue
  class IDX memory
  class Q,A neutral
  class RET,GEN compute

How it really works#

When retrieval is the right rung#

Use it when answers depend on information that is private, recent, large or must be cited. Do not use it when the whole knowledge base fits comfortably in the context window and rarely changes — a twenty-page handbook can simply be included in every prompt and cached, which is simpler and often more accurate.

The decisions an architect makes#

Beyond basic RAG#

You will hear these names. Each fixes a specific weakness; none is a default.

Where RAG systems fail#

When you review a RAG design, or diagnose a failing one, walk this list in order. The earlier the stage, the more often it is the cause.

Only the last row is a model problem. An architect who knows this stops teams from swapping models to fix what is really a parsing bug.

Measuring it#

Measure the two halves separately, on a set of real questions with known answers.

• Retrieval quality — for each question, is the passage that contains the answer among the chunks retrieved? This is recall. If recall is 70%, no model can get you above 70%.
• Answer quality — given the right passages, is the answer correct, complete and supported by what it cites? This is groundedness or faithfulness.
• Refusal quality — when the answer is not in the sources, does it say so?

A release gate on these numbers is described in Evaluation as the Contract.

What fills the boxes in 2026#

The mechanism of each stage is in Context and Retrieval; a full design at 20 million documents is in An Enterprise RAG Assistant.

Code#

A sizing calculator for the first conversation about a knowledge assistant: how many chunks, what it costs to index, how big the index is, and what each question costs. Every input is a number you can ask for in discovery.

// ragsize.go — size and price a retrieval system from five discovery numbers.
package main

import "fmt"

func main() {
	// Discovery numbers.
	docs := 400_000.0       // documents in scope
	tokensPerDoc := 3_000.0 // average length
	changedPerMonth := 0.05 // share of documents that change each month
	questionsPerDay := 8_000.0
	workdays := 22.0

	// Design choices.
	chunkTokens := 500.0
	overlap := 0.10                // each chunk repeats 10% of its neighbour
	topK := 8.0                    // chunks sent to the model per question
	dims := 1024.0                 // embedding size
	bytesPerDim := 4.0             // float32
	embedPrice := 0.02             // dollars per million tokens, illustrative
	inPrice, outPrice := 2.0, 10.0 // mid-tier model, dollars per million tokens
	fixedPrompt, answer := 1500.0, 350.0

	chunks := docs * tokensPerDoc / (chunkTokens * (1 - overlap))
	indexGB := chunks * dims * bytesPerDim / 1e9
	firstIndex := docs * tokensPerDoc * (1 + overlap) * embedPrice / 1e6
	monthlyReindex := firstIndex * changedPerMonth

	inTokens := fixedPrompt + topK*chunkTokens
	perQuestion := (inTokens*inPrice + answer*outPrice) / 1e6
	monthlyAnswers := perQuestion * questionsPerDay * workdays

	fmt.Printf("chunks               %12.0f\n", chunks)
	fmt.Printf("vector index         %12.1f GB (before index overhead)\n", indexGB)
	fmt.Printf("first embedding run  %12.2f USD\n", firstIndex)
	fmt.Printf("re-embedding / month %12.2f USD\n", monthlyReindex)
	fmt.Printf("input per question   %12.0f tokens\n", inTokens)
	fmt.Printf("cost per question    %12.4f USD\n", perQuestion)
	fmt.Printf("answers / month      %12.0f USD\n", monthlyAnswers)
	fmt.Printf("\nshare of monthly cost that is generation: %.1f%%\n",
		100*monthlyAnswers/(monthlyAnswers+monthlyReindex))
}

Two conclusions hold for almost any inputs. Embedding the corpus is cheap, so do not design around avoiding it. Generation dominates, and its cost is set by how many chunks you send — so reranking down to fewer, better chunks pays for itself.

Remember this#

• RAG is search plus data quality, with a model at the end.
• Two halves: ingestion in the background, query per question. Different owners, different failures.
• Hybrid search by default. Permissions filtered at query time, decided on day one.
• Bad answers are usually retrieval failures. Measure retrieval and generation separately.
• If the corpus is small and stable, put it in the prompt and cache it.

Try it#

1. Run ragsize.go. Change topK from 8 to 3 and to 20. What happens to the monthly cost? What would you need to add to make 3 chunks enough?
2. Take ten real questions people ask about a document set you know. For each, find by hand the passage that answers it. How many needed two passages from different documents?
3. For a knowledge base at your work, write down who owns each source, how often it changes and who is allowed to read it. That table is the first page of a RAG design.

Check yourself#

1. Name the two halves of a RAG system and one failure that belongs to each.
2. Retrieval recall on your test set is 65%. What is the best possible answer accuracy, and what do you fix first?
3. Why is hybrid search the default rather than vector search alone?
4. When is retrieval unnecessary?