Sampling and Structured Output

The idea in one minute#

The forward pass ends with one row of scores per request: a number for every token in the vocabulary. Choosing a token from that row is sampling, and it is where every request’s individual settings — temperature, top-p, penalties, a seed — are finally applied. vLLM does it for the whole batch at once, on the GPU, in a fixed order that determines how the settings interact. Structured output adds one more step at the front: a grammar, running on the CPU in the engine core, computes which tokens are legal next and forbids all the others. That guarantees valid JSON, and it is the one feature whose cost is paid in engine-core CPU time rather than GPU time.

A picture#

flowchart LR
  H[":nvidia: <b>Hidden states</b><br/><small>one row per request</small>"] --> L[":i-layers: <b>Logits</b><br/><small>N × vocab</small>"]
  G[":i-code: <b>Grammar bitmask</b><br/><small>computed on CPU in the engine core<br/>during the forward pass</small>"] -->|"sample_tokens(grammar_output)"| MASK
  L --> MASK["mask illegal tokens"]
  MASK --> P1["allowed IDs, bad words,<br/>min_tokens, logit_bias"]
  P1 --> P2["penalties<br/><small>repetition, frequency, presence</small>"]
  P2 --> GR{"temperature = 0?"}
  GR -- "yes" --> AM["argmax"]
  GR -- "no" --> T["temperature"] --> MP["min_p"] --> K["top_k, top_p"] --> DR["draw"]
  AM --> OUT[":i-zap: <b>token IDs</b>"]
  DR --> OUT
  class H,L compute
  class G queue
  class MASK,P1,P2,T,MP,K neutral
  class GR queue
  class AM,DR compute
  class OUT io

How it really works#

The order is documented, and it matters#

The Sampler class (vllm/v1/sample/sampler.py) states its pipeline in its docstring:

1. If logprobs are requested: keep the raw logprobs (or raw logits) to return.
2. Convert logits to float32.
3. Apply allowed token ids whitelist.
4. Apply bad words exclusion.
5. Apply logit processors which are not argmax-invariant, i.e. that can impact
   greedy sampling.
     a) Min tokens processor
     b) Logit bias processor
6. Apply penalties
     a) Repetition penalty   b) Frequency penalty   c) Presence penalty
7. Sample the next tokens:
     a) If not all_random, perform greedy sampling. If all_greedy, return.
     b) Apply temperature.
     c) Apply logit processors which are argmax-invariant, by default the min_p processor.
     d) Apply top_k and/or top_p.
     e) Sample the next tokens with the probability distribution.
     f) Return random samples where temperature >= epsilon (1e-5), greedy otherwise.
8. Gather the logprobs of the top max_num_logprobs and sampled token (if requested).

The key distinction is argmax-invariant. A transformation is argmax-invariant if it can never change which token has the highest score. Temperature, min_p, top_k and top_p all qualify: they reshape or truncate the distribution but the top token stays on top. Penalties, logit bias and min_tokens do not qualify: they can dethrone the top token.

That split dictates the order. Everything that can change the winner runs before the greedy choice in step 7a, so that greedy decoding respects it. Everything that cannot is skipped entirely for greedy requests.

Consequences worth remembering:

One batch, many settings#

A batch mixes requests with different parameters. The sampler does not loop over requests. Per-request values live in tensors with one row per request (SamplingMetadata), and each step is one vectorised operation:

# Apply temperature.
logits = self.apply_temperature(logits, sampling_metadata.temperature, ...)
...
sampled = torch.where(
    sampling_metadata.temperature < _SAMPLING_EPS,
    greedy_sampled,
    random_sampled,
    out=greedy_sampled,  # Reuse tensor
)

For a mixed batch both the greedy and the random result are computed for every row, and torch.where picks per row. Two flags short-circuit the common cases: if every request in the batch is greedy (all_greedy), the random path is skipped entirely; if every request is random (all_random), the argmax is skipped.

These tensors are part of the persistent batch from Executor, Worker, Model Runner: a request’s temperature is written to its row once, when it joins.

Penalties need history#

Repetition, frequency and presence penalties depend on which tokens a request has already produced (and, for repetition penalty, on its prompt). The sampler therefore needs each request’s token history as a tensor on the GPU. Maintaining it is part of the persistent batch’s job, and it is the reason a server where no request uses penalties is measurably cheaper per token than one where a few do: a flag (no_penalties) skips the whole step.

Seeds and reproducibility#

A request with a seed gets its own random-number generator, stored in the persistent batch and advanced only by that request. The same request with the same seed sees the same random draws regardless of what else is in the batch.

Two caveats keep this from being a guarantee of identical output:

• The logits can differ in their last bits between runs, because batch composition changes the order of floating-point operations and which kernel variant runs. A different last bit can flip a near-tie.
• Reproducibility across vLLM versions, GPUs or batch shapes is not promised.

A newer option, described in the documentation as batch invariance, constrains kernel choices so that a request’s output does not depend on its batch-mates, at some cost in speed.

Logprobs are not free#

When a request asks for logprobs: 5, the sampler must compute a log-softmax over the whole vocabulary for that row, find the top five, and ship them to the API server every step. For one row that is cheap; for a 100,000-token vocabulary across hundreds of requests it becomes a visible share of step time and of the bytes crossing the engine socket. The Model Runner V2 design note describes the improvement it makes: identify the top-k from the logits first, then compute log-probabilities only for those.

prompt_logprobs is more expensive still: it needs a distribution at every prompt position, so the “only the last row” optimisation of Building the Batch no longer applies, and the prefix cache cannot be used.

The sampler in Model Runner V2#

The newer runner reimplements most of this in Triton kernels. Its sampling kernel uses the Gumbel-max trick: adding independent Gumbel noise to each logit and taking the argmax is mathematically the same as sampling from the softmax distribution. This avoids materialising the probabilities at all, and the noise is generated inside the kernel from the request’s seed, so no generator state has to be kept on the CPU.

Custom logits processors#

You can add your own step. A logits processor is a class that receives the batch’s logits and may modify them, declared as argmax-invariant or not so that the sampler can place it correctly. Processors are loaded at startup (--logits-processors) and operate on the whole batch, with a hook that tells them when requests join and leave so they can keep per-request state in tensors. Per-request Python callbacks, which older versions allowed, are gone: a Python function called once per request per token cannot be part of a vectorised pipeline.

Structured output: where the grammar lives#

A request may constrain its output:

{
  "messages": [{"role": "user", "content": "Give me a user record."}],
  "structured_outputs": {"json": {"type": "object", "properties": {"name": {"type": "string"}}}}
}

The variants are json (a JSON schema), regex, choice (one of a list), grammar (a context-free grammar) and structural_tag. The OpenAI response_format field maps onto the same machinery.

The mechanism is constrained decoding. At every step a grammar engine, given everything generated so far, computes the set of tokens that could legally come next. All other tokens' logits are set to minus infinity before sampling. The model cannot produce invalid output because invalid tokens are never candidates.

The work is split across processes:

Compilation is asynchronous. A complex schema can take tens or hundreds of milliseconds to compile. The request does not block the engine while that happens: it is given the status WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR, the scheduler skips it (without blocking those behind it, as described in One Scheduling Step), and it becomes schedulable when its grammar is ready.

The bitmask is one bit per vocabulary token per request, packed into 32-bit integers: about 12.5 kB per request for a 100,000-token vocabulary. It is computed during the forward pass, as The Engine Core Loop showed, and sent with sample_tokens. When more than 128 requests in a batch are constrained, the masks are filled in parallel on a small thread pool, 16 requests per task.

The grammar is the authority. After sampling, accept_tokens feeds the new token to the grammar. If the grammar rejects it — which should be impossible — the request is terminated with an error rather than allowed to continue in an inconsistent state.

Backends#

--structured-outputs-config.backend selects one. The default is auto, which, in the words of the config, “will make opinionated choices based on request contents and what the backend libraries currently support, so the behavior is subject to change in each release.” Pin a backend if you need identical behaviour across upgrades. Regex dialects differ between them: xgrammar, guidance and outlines use Rust-style regular expressions, lm-format-enforcer uses Python’s.

What structured output costs#

• CPU in the engine core, every step, proportional to the number of constrained requests. This is the engine core’s single main thread; heavy structured-output traffic can make it the bottleneck while the GPU has capacity to spare.
• Part of the async-scheduling overlap, because the mask for step N+1 needs the token from step N (Async Scheduling).
• Interaction with speculative decoding: draft tokens must be checked against the grammar too (validate_tokens), and drafts the grammar forbids are discarded.
• Interaction with reasoning models: the constraint must begin only after the model’s reasoning section ends, which is why the structured-output config carries a reasoning_parser setting.

A guarantee of valid output is not a guarantee of good output. A model forced into a schema it does not want to follow will fill it with low-quality content. Describe the schema in the prompt as well; the constraint then merely enforces what the model was already trying to do.

Code#

The sampling pipeline in the documented order, for three requests: free text with every knob turned, and two steps of a JSON-constrained request.

package main

import (
	"fmt"
	"math"
	"math/rand"
	"sort"
)

var vocab = []string{"{", "}", `"name"`, ":", `"Ada"`, ",", "hello", "the", "42", "\n"}

type params struct {
	temperature       float64
	topK              int     // 0 = off
	topP              float64 // 1 = off
	minP              float64 // 0 = off
	repetitionPenalty float64 // 1 = off
}

func softmax(logits []float64) []float64 {
	m := math.Inf(-1)
	for _, l := range logits {
		m = math.Max(m, l)
	}
	sum := 0.0
	p := make([]float64, len(logits))
	for i, l := range logits {
		p[i] = math.Exp(l - m)
		sum += p[i]
	}
	for i := range p {
		p[i] /= sum
	}
	return p
}

func show(stage string, logits []float64) {
	p := softmax(logits)
	fmt.Printf("%-28s", stage)
	for i := range vocab {
		if math.IsInf(logits[i], -1) {
			fmt.Printf("%7s", "-")
		} else {
			fmt.Printf("%7.3f", p[i])
		}
	}
	fmt.Println()
}

// sample follows the order documented in vLLM's Sampler class.
func sample(raw []float64, allowed []bool, history []int, p params, rng *rand.Rand) int {
	logits := append([]float64(nil), raw...)
	show("raw", logits)

	// Structured output: the grammar's bitmask removes tokens before anything else.
	if allowed != nil {
		for i := range logits {
			if !allowed[i] {
				logits[i] = math.Inf(-1)
			}
		}
		show("grammar bitmask", logits)
	}

	// Step 6: penalties. These can change the argmax, so they come before greedy.
	if p.repetitionPenalty != 1 {
		for _, t := range history {
			if logits[t] > 0 {
				logits[t] /= p.repetitionPenalty
			} else {
				logits[t] *= p.repetitionPenalty
			}
		}
		show("repetition penalty", logits)
	}

	// Step 7a: greedy. temperature == 0 stops here.
	if p.temperature < 1e-5 {
		best := 0
		for i := range logits {
			if logits[i] > logits[best] {
				best = i
			}
		}
		return best
	}

	// Step 7b: temperature.
	for i := range logits {
		logits[i] /= p.temperature
	}
	show(fmt.Sprintf("temperature %.1f", p.temperature), logits)

	// Step 7c: min_p. Drop tokens whose probability is below minP x the top probability.
	if p.minP > 0 {
		probs := softmax(logits)
		top := 0.0
		for _, v := range probs {
			top = math.Max(top, v)
		}
		for i := range logits {
			if probs[i] < p.minP*top {
				logits[i] = math.Inf(-1)
			}
		}
		show(fmt.Sprintf("min_p %.2f", p.minP), logits)
	}

	// Step 7d: top_k, then top_p, over the tokens sorted by probability.
	order := make([]int, len(logits))
	for i := range order {
		order[i] = i
	}
	sort.SliceStable(order, func(a, b int) bool { return logits[order[a]] > logits[order[b]] })
	if p.topK > 0 {
		for _, i := range order[min(p.topK, len(order)):] {
			logits[i] = math.Inf(-1)
		}
		show(fmt.Sprintf("top_k %d", p.topK), logits)
	}
	if p.topP < 1 {
		probs := softmax(logits)
		cum := 0.0
		for _, i := range order {
			if cum >= p.topP {
				logits[i] = math.Inf(-1)
			}
			cum += probs[i]
		}
		show(fmt.Sprintf("top_p %.2f", p.topP), logits)
	}

	// Step 7e: draw one token from what is left.
	probs := softmax(logits)
	r, cum := rng.Float64(), 0.0
	for i, v := range probs {
		cum += v
		if r < cum {
			return i
		}
	}
	return len(probs) - 1
}

func main() {
	fmt.Printf("%-28s", "")
	for _, v := range vocab {
		fmt.Printf("%7q", v)
	}
	fmt.Println()

	raw := []float64{1.2, 0.3, 2.0, 0.1, 1.5, 0.2, 3.1, 2.8, 0.9, -1.0}
	rng := rand.New(rand.NewSource(3))

	fmt.Println("\nA. free text: temperature 0.8, repetition penalty 1.3, min_p 0.05, top_k 5, top_p 0.9")
	history := []int{6} // "hello" was already generated
	tok := sample(raw, nil, history, params{0.8, 5, 0.9, 0.05, 1.3}, rng)
	fmt.Printf("sampled: %q\n", vocab[tok])

	fmt.Println("\nB. JSON mode at the start of an object: the grammar allows only \"{\"")
	allowed := make([]bool, len(vocab))
	allowed[0] = true
	tok = sample(raw, allowed, nil, params{temperature: 0.8, topP: 1, repetitionPenalty: 1}, rng)
	fmt.Printf("sampled: %q   (the model preferred %q; the mask overruled it)\n", vocab[tok], vocab[6])

	fmt.Println("\nC. JSON mode after {\"name\": — the grammar allows a string or a number")
	allowed = make([]bool, len(vocab))
	allowed[4], allowed[8] = true, true
	tok = sample(raw, allowed, nil, params{temperature: 0, topP: 1, repetitionPenalty: 1}, rng)
	fmt.Printf("sampled (greedy): %q\n", vocab[tok])
}

In run A, watch "hello" lose first place to "the" at the repetition-penalty row: that could not have happened if penalties ran after the greedy choice. In run B the model’s preference is irrelevant; one token is legal. In run C the temperature is zero, the mask leaves two candidates, and the higher of those two wins, although neither was the model’s overall favourite.

Remember this#

• The sampler runs once per step for the whole batch; per-request settings are rows of tensors.
• Order: masks and bias, then penalties, then greedy; temperature, min_p, top_k, top_p only for random sampling.
• With temperature = 0, top_p/top_k/min_p do nothing; penalties and bias still apply.
• A seed gives a request its own generator, but bit-exact reproducibility is not guaranteed.
• Logprobs, and especially prompt logprobs, add real cost.
• Structured output masks illegal tokens before sampling; the grammar runs on the CPU in the engine core.
• Grammar compilation is asynchronous; the bitmask is computed during the forward pass.
• The structured-output backend defaults to auto and may change between releases.

Try it#

1. In run A, set the repetition penalty to 1.0. Which token is sampled now? Then set temperature to 0 with the penalty at 1.3 and confirm the penalty still decides.
2. In run C, change the temperature to 1.5. Over many seeds, how often is "42" chosen? Is the output still always valid?
3. Against a real server, send the same request with a JSON schema 50 times concurrently and 50 times without one. Compare tokens per second, and watch the engine core’s CPU in top.

Check yourself#

1. What does “argmax-invariant” mean, and why does it decide where a processor runs?
2. A request sets temperature: 0 and top_p: 0.5. What effect does top_p have?
3. Where is the grammar bitmask computed, where is it applied, and why are those different places?

Sources#

Checked on 5 October 2026 against vLLM v0.30.0 and main at commit 0c16eee.

• vllm/v1/sample/sampler.py — the pipeline docstring
• vllm/v1/structured_output/__init__.py — StructuredOutputManager
• vllm/config/structured_outputs.py
• Structured outputs
• Logits processors (design)
• Model Runner V2 design — the Triton sampler