Choosing a Model

The idea in one minute#

There is no best model. There is a best model for this task, at this volume, under these constraints, this quarter. Public leaderboards tell you which models are worth testing; they do not tell you which one works on your documents.

Model selection is a three-stage funnel. Hard constraints remove most candidates without any testing — where the data may go, what the model must be able to read, what the contract allows. Your own evaluation on fifty to two hundred real examples ranks the survivors. Cost and speed at your real volume break the ties.

Then design so that the choice is cheap to change, because you will change it. Model generations now arrive every few months and old ones are retired on a schedule. “Which model?” is a configuration value, not a foundation.

A picture#

flowchart LR
  ALL[":i-layers: <b>Every model on the market</b>"] --> HC[":i-funnel: <b>1. Hard constraints</b><br/><small>data location, modality, licence, platform</small>"]
  HC --> SL[":i-list-checks: <b>Shortlist</b><br/><small>three to five</small>"]
  SL --> EV[":i-flask-conical: <b>2. Your evaluation</b><br/><small>50–200 real cases</small>"]
  EV --> PASS[":i-check: <b>Those that pass the bar</b>"]
  PASS --> CS[":i-coins: <b>3. Cost and speed</b><br/><small>at your volume</small>"]
  CS --> PICK[":i-route: <b>Primary + fallback</b><br/><small>behind one gateway</small>"]
  PICK -.->|"new model released, or old one retired"| EV
  class ALL neutral
  class HC,CS queue
  class SL,PASS memory
  class EV compute
  class PICK io

How it really works#

Stage 1 — hard constraints#

These are yes-or-no. A model that fails one is out, however good it is.

Stage 2 — your own evaluation#

Public benchmarks measure general ability on public tasks. Your task is not public. Build a small private test set once and reuse it for every model decision for the life of the system.

1. Collect 50–200 real inputs, including the awkward ones — the long, the ambiguous, the ones that should be refused.
2. Write down the right answer for each, or the rules a good answer must meet.
3. Fix the prompt and settings, then run every shortlisted model on the same set.
4. Score by exact match where possible, by a rubric where not. A stronger model can do the grading if a person has checked a sample of its grades.
5. Set the bar first. Decide the pass mark from the business requirement before looking at results, or you will set it wherever your favourite model landed.

How to build the set and the gate properly is in Evaluation as the Contract. The lesson here is only that this step cannot be skipped or delegated to a leaderboard.

Stage 3 — cost and speed#

Among models that pass, choose on economics. List prices on 13 October 2026, per million tokens, from the providers’ own pricing pages:

Three things make the real bill differ from list price, and each is a design decision.

Speed has two parts: time to the first token, and tokens per second after that. Small models are faster on both. Reasoning modes add seconds of thinking before the first visible word. Measure on your own prompts in your own region.

Open-weight models#

An open-weight model can be downloaded and run on hardware you control. Consider one when a hard constraint demands it — data that may not leave, an offline site — or when volume is so large and steady that owning the capacity is cheaper.

The price per token looks low. The total cost includes GPUs that are paid for whether busy or idle, engineers to run them, and upgrades you now do yourself. A managed endpoint for an open-weight model on a cloud platform is a middle path: their hardware, your choice of model. The arithmetic is in Build, Buy or Partner.

More than one model#

Mature systems rarely use a single model.

• Route by difficulty. Send the many easy calls to a small model and the few hard ones to a larger one. This alone often cuts the bill by more than half.
• Cascade. Try the small model; escalate when a check fails.
• Keep a fallback. A second provider that passed your evaluation, ready for an outage.
• Separate jobs. One model to embed, one to rerank, one to answer, one to grade.

All of this is practical only behind a gateway — one internal endpoint that applications call by a logical name such as “fast” or “smart”, with the real model chosen in configuration. See Model Access and Gateways.

Designing for the next model#

• Never hard-code a model name in application code.
• Keep prompts in version control, separate from code, with the evaluation results they were tested against.
• Rerun the evaluation on every new model release and every deprecation notice. A newer model is usually better and occasionally worse on your task.
• Track the retirement date of every model you depend on as an operational risk.

Code#

The funnel as a program: gate on hard constraints, drop models below the quality bar, then rank by monthly cost. The scores are invented; the method is the point.

// modelpick.go — constraints first, then your evaluation, then cost.
package main

import (
	"fmt"
	"sort"
)

type model struct {
	name       string
	inRegion   bool    // can be processed in the required region
	readsScans bool    // accepts images of documents
	approved   bool    // on the organisation's approved platform
	score      float64 // pass rate on OUR evaluation set, 0–1
	in, out    float64 // dollars per million tokens
	ttftMS     float64 // measured time to first token
}

func main() {
	candidates := []model{
		{"small-a", true, true, true, 0.86, 0.10, 0.50, 350},
		{"small-b", true, false, true, 0.90, 0.30, 2.50, 300},
		{"mid-a", true, true, true, 0.94, 2, 10, 600},
		{"mid-b", false, true, true, 0.96, 2, 10, 550},
		{"mid-c", true, true, true, 0.93, 0.75, 3.75, 450},
		{"frontier-a", true, true, true, 0.97, 10, 50, 1400},
		{"open-weights", true, true, false, 0.91, 0.40, 0.40, 500},
	}

	// The requirement, written before testing.
	const bar, maxTTFT = 0.92, 1000.0
	callsPerMonth, inTok, outTok := 2_000_000.0, 6_000.0, 400.0

	var ok []model
	for _, m := range candidates {
		switch {
		case !m.inRegion:
			fmt.Printf("out  %-13s cannot process in region\n", m.name)
		case !m.readsScans:
			fmt.Printf("out  %-13s cannot read scanned pages\n", m.name)
		case !m.approved:
			fmt.Printf("out  %-13s not on the approved platform\n", m.name)
		case m.score < bar:
			fmt.Printf("out  %-13s %.0f%% is below the %.0f%% bar\n", m.name, m.score*100, bar*100)
		case m.ttftMS > maxTTFT:
			fmt.Printf("out  %-13s first token after %.0f ms\n", m.name, m.ttftMS)
		default:
			ok = append(ok, m)
		}
	}

	cost := func(m model) float64 {
		return callsPerMonth * (inTok*m.in + outTok*m.out) / 1e6
	}
	sort.Slice(ok, func(i, j int) bool { return cost(ok[i]) < cost(ok[j]) })

	fmt.Println("\npassed, cheapest first:")
	for i, m := range ok {
		role := "fallback"
		if i == 0 {
			role = "primary"
		}
		fmt.Printf("  %-13s %.0f%%  $%8.0f / month  %s\n", m.name, m.score*100, cost(m), role)
	}
}

Note what the program refuses to do: it never picks the highest score. The frontier model is best and is rejected for being slow; the winner is the cheapest model that clears a bar set in advance.

Remember this#

• Constraints, then your own evaluation, then cost and speed. In that order.
• Leaderboards build the shortlist. Only your test set makes the decision.
• Three price tiers about a hundred-fold apart; caching and batch change the real bill.
• Use several models behind a gateway: route by difficulty, keep a fallback.
• The model is configuration. Rerun the evaluation whenever it changes.

Try it#

1. Run modelpick.go. Lower the bar to 0.85. Which model becomes primary, and how much does the monthly cost fall? Is seven points of quality worth that to this business?
2. Write the hard-constraint table for a use case at your work. How many of the eight rows can you answer today, and whom must you ask for the rest?
3. Look up the deprecation page of one model provider. Find a model with a retirement date and work out how many months of notice customers were given.

Check yourself#

1. Why set the pass mark before running the evaluation?
2. Which constraints can remove a model without any testing?
3. Give two reasons a system might use more than one model.
4. Why should application code never contain a model name?

Sources#

• Claude pricing, OpenAI API pricing, Gemini API pricing — read on 13 October 2026