Evals are a strong argument for open‑weight models.

1. The short version

1. Confidence is a zero-latency, zero-cost eval. It’s read from the probabilities the model already computed to write the answer. There’s no judge model to define, prompt or drift: the score is plain arithmetic on the model’s own output.
2. On select tasks, confidence is a strong filter. When the answer is in text you give the model, like a field in a document, confident answers were right 98% of the time. On facts from memory, models were right only 26% of the time overall, but 80% of the time when confident: they mostly know what they don’t know. When the model has to work an answer out, confident mistakes are common: 47% of mistakes on counting, calculating and similar tasks came with high confidence.
3. Repeatability is a second way to measure how sure a model is. Running the same question five times at a temperature above 0 samples from the model’s own probabilities, so agreement across runs is an indirect read of its confidence. Most closed models, including all the labs’ latest reasoning models, won’t show you their logits, but repeatability needs none, so it works on them too. On work-out tasks it was the better read: 93% right against 68%.
4. Output evals calibrate both to your use case. The signals score every answer instantly; grading a sample of outputs shows which scores are actually right for each task and agent. Over time that tunes the thresholds, so the instant signals handle most answers and graded evals focus on the rest.

Confidence = how far ahead the model’s top choice of token was of its second, measured on the tokens of the answer. 95% vs 3% (a gap of 0.92) is sure; 45% vs 40% (0.05) is a coin-toss. We call an answer highly confident when every answer token has a gap of 0.9 or more.

2. What we tested

We tested three kinds of task, each leaning on a different ability: recalling what the model learned in training, finding information in the prompt, and working through steps to compute an answer.

• Answer: the model answers from its own knowledge, e.g. “In what country is Asseek River?”
• Look-up: the answer is in text you give it, e.g. find a value in a list or pull a field from a document.
• Work-out: the answer has to be computed or tracked, e.g. count, calculate or follow a series of swaps.

Open-ended tasks such as summarising or drafting have no single right answer to check, so they weren’t part of this test.

3. When it’s confident, how often is it right?

This is the practical question: if you auto-accept every highly confident answer, how many will be right?

What matters is the gap between each bar and overall accuracy: on recall, confidence lifts it from 26% to 80%, while on counting it barely moves (49% to 52%). On work-out tasks, 26% to 72% of mistakes still came with high confidence, against 10% or fewer on answer and look-up tasks.

4. Two ways to measure how sure a model is

There are two ways to read how sure a model is:

• Confidence score: read straight from the logits, at no extra cost.
• Repeatability: run the same question five times at a temperature above 0 and see whether the answers agree. The runs sample from the model’s own probabilities, so agreement is an indirect read of the same certainty. It costs five extra runs per answer.

Repeatability works on any model; the confidence score needs logits. As of October 2026, OpenAI returns them only for its older non-reasoning models, not for reasoning models such as GPT-5; Google stopped returning them with Gemini 3; Anthropic’s API doesn’t return them.

On answer and look-up tasks, the direct read is as good or better: 80% against 71% on recall, 98-99% against 97-98% on look-ups. On work-out tasks, the indirect read wins by a wide margin: 95% against 52% on counting, 97% against 43% on arithmetic. A likely reason is that on multi-step tasks the uncertainty sits earlier in the reasoning, before the answer tokens the confidence score reads, while separate runs can take different paths and disagree.

Neither read is perfect. On state tracking, repeatability trusts just 13% of answers and a quarter of those are still wrong, so the output itself needs grading.

5. Putting it to work

Real workflows mix these tasks, often within a single request: an agent might pull figures from a document, then add them up. Score each step with the signal that suits it.

1. Calibrate to your use case. How far a signal can be trusted shifts with the model, the prompt and the task: on recall, high confidence meant 67% right for one model and 93% for another. Grade a sample of your own outputs before setting any threshold.
2. Use it as a zero-latency guardrail. Accept answers above the threshold, reject or hold those below it, and send the uncertain middle to a person or a heavier check.
3. Keep refining. Grade a sample on both sides of the threshold. Confident mistakes still get through, 3% to 10% of mistakes even on the best tasks, and rejected answers show whether the bar is too strict. Re-tune as models, prompts and traffic change.

Used this way, the signals don’t replace output evals. They give every response a fast first read, and graded outputs tell you how far that read can be trusted for your use case.

Appendix: Methodology