Send the same prompt to an aligned language model a few dozen times. How different will the answers be? The reflexive answer is "depends on the temperature." This paper argues the more fundamental variable is a property of the task itself: consensus density, meaning the degree to which people who know the domain would agree on what counts as a good answer.
What the paper says
The method is clean. Temperature is fixed at 1.0, each prompt gets N=25 completions, every completion is embedded, and dispersion is measured as the mean pairwise cosine distance among those embeddings. There are 20 prompts across five bands: factual (the capital of France), explanatory (how a binary search tree works), judgment (should this startup raise the round), creative (write the opening of a story), and a miscellaneous advice band.
The result is a clean gradient. On factual prompts dispersion is close to zero. Sample twenty-five times and the answers collapse to essentially a single point. The further you move toward the open end, the wider the spread gets, with creative prompts the widest of all.
The more interesting part is predictability. Three held-out judges (GPT-4.1, Gemini 2.5 Flash, Qwen-Max) were shown only the prompt, never a single output, and asked to score how much consensus the task admits. That score predicts the observed dispersion: rho = -0.91 on Claude, rho = -0.84 on GPT-4o. Judge-judge agreement sits between 0.85 and 0.95, and a human expert panel agrees with the judges at 0.74. In other words, how much the outputs will scatter is legible from the task before the model says anything at all.
The base-versus-instruct comparison supplies a mechanism. Qwen2.5-7B and its instruct variant rank the bands in nearly the same order (rho 0.80 vs 0.79), but alignment compresses the high-consensus tasks by roughly 8.7-fold while compressing the creative band only 1.8-fold. Alignment did not create the gradient. It amplified one the pretrained base already carried.
How strong is the evidence
Cross-model replication, a human expert comparison, and a base-versus-instruct ablation are all present. The core correlation is unlikely to be an accident.
The limitation lives in the metric, and the authors are direct about it. The measure captures semantics, not style. Ask a model to "write in the style of Hemingway" and the judges rate consensus as high while embedding distance also reports low dispersion, so metric and judgment happen to line up. The inverse case exists too: two outputs that say the same thing in wildly different registers will look convergent under cosine distance. The claim therefore covers determinism at the content layer, not consistency at the style layer.
The scale deserves an honest read as well. Twenty prompts at twenty-five samples each supports a correlational trend. It does not support threshold-setting. Anyone who wants to turn this into an operational rule like "dispersion above 0.3 routes to a human" needs to recalibrate on their own task distribution first. Token-entropy correlates with the dispersion measure at only 0.66, which says the entropy numbers you already have on hand are not a substitute for actually running this measurement.
What it means in practice
For a contract review workload like Zico AI, where output determinism is a hard requirement rather than a nice-to-have, this gives a design rationale rather than a vague intuition.
The move is to classify subtasks by consensus density first, then pick the interaction shape for each class. Clause and element extraction, amount reconciliation, date consistency checks: these converge naturally, so the system can let the model produce a conclusion directly and have a person spot-check a sample. Risk characterization, negotiation strategy, deciding which clause to concede: these diverge naturally. The right pattern there is for the model to lay out options with reasoning and let a human converge. Forcing a single answer out of a divergent task produces something that looks decisive but is really one draw from a distribution, presented as if it were the distribution.
There is a second, more operational use. Since dispersion is predictable from the prompt alone, a consensus-density estimate can run before a task enters the pipeline and decide whether it takes the automated path or the human-review path. Routing on the consensus structure of the task is more dependable than turning the temperature down across the board. Lowering temperature hides variance. It does not make the answer more deserving of trust, and it does not tell you which answers were shaky to begin with.
A caveat worth carrying into any such deployment: the judges in the paper scored 20 curated prompts, not a messy production stream, and their agreement with human experts was 0.74, which is solid but leaves real disagreement on the table. Treat a predicted consensus score as a routing prior that can be wrong, not as a gate that is right by construction. The failure mode to watch for is a task that scores as high-consensus because the prompt is phrased crisply while the underlying question is genuinely contested.
The broader takeaway is about where variability comes from. It is tempting to treat sampling spread as a knob on the decoder, something you tune away with temperature, top-p, or a longer system prompt. The base-versus-instruct result points elsewhere. The gradient was in the pretrained model, alignment sharpened it unevenly across bands, and the shape of that gradient tracks a fact about the world: some questions have one right answer and some do not. A system built on top of a language model inherits that structure whether or not it acknowledges it, and pretending otherwise mostly relocates the uncertainty into places where nobody is looking for it.