← Research
Research

There is no best AI judge.

No single judge is best at every check. Hosho sends each check to the one that is, and scores higher than a frontier LLM at a fraction of its latency and cost.

Oct 2026·~4 min read

88 vs 83
Accuracy: Hosho routing against the best frontier LLM on everything, across five public datasets.
34%
Of the cost of running the frontier LLM on everything.
73%
Of checks need no frontier LLM at all.

Start with what good looks like

Evaluation has four steps, in order.

  1. Define what good looks like.
  2. Define the metrics that make that measurable.
  3. Pick and design a judge for each metric.
  4. Calibrate each judge against real answers, and keep tracking it.

A judge is designed, not picked: what it answers well, what it costs and how long it takes all change with the check. This article shows how sensitive that design is, using five public datasets run through every judge that could answer them and scored against the datasets’ own answers. A different judge won each one.

Hosho’s judges

Hosho evaluates with three kinds of judge, plus people, and picks among them check by check. The colours below are used in every chart that follows.

Deterministic checks. Rules and short programs for anything defined by a count, a format, a threshold or a sum. Exact, instant and free.
Fast judges. LOGEN, Hosho’s proprietary fine-tuned grounding model, alongside Jev, a general decision model for typed questions. One question at a time, at low latency and low cost.
Frontier LLMs. For the checks that need several things weighed together, and for explaining what went wrong.
People. The answers every other judge is calibrated against.
Figure 1: One judge for everything, against splitting the item into checks
ONE JUDGE FOR EVERYTHINGOne iteman answer to evaluate“is it good?”everything, one callFrontier LLMreads, weighs, decidesevery check at onceOne verdict83avg · $4.32 / 1kHOSHO: SPLIT, ROUTE, COLLECTOne itemthe same answer,the same questionsplitThe checks inside iteach one a question with one kind of answerright length and formatevery number matches the sourceeach claim is stated in the sourcetone, fit and overall judgmenta sample of all of the above73% of checks need no frontier LLMroute each checkDeterministicexact · $0 · instantFast judgesLOGEN, Jev · ~1 sFrontier LLMfor judgment onlyPeoplecheck every judgeEvery verdict,one resultwhich claim failed,which number changed,the LLM’s view whereit was needed88avg accuracy34% of the cost

1. Different checks need different judges

No judge came top on more than one of the five datasets.

  • Code won where the answer is a count. The strongest LLM failed more than a third of responses that followed every instruction.
  • LOGEN won on contracts, at a sixtieth of the frontier LLM’s cost.
  • Jev tied the frontier LLMs on which answer is better, at an eightieth of the cost.
  • The frontier LLMs won on tables and long grounded answers, the checks that need several things weighed at once.
Figure 2: A different judge wins each task
Balanced accuracy for each judge on each of five datasets.
TaskCodeLOGENJevFrontier LLM
(lite)
Frontier LLM
(flagship)
Did it follow every instruction?IFEval100697076
Does the table support this?TabFact9159888794
Is every claim in the source?RAGTruth5875758382
Which of two answers is better?MT-Bench69838382
Does the contract say this?ContractNLI86808182

Balanced accuracy against each dataset’s own answers. Filled = best; outlined = within a point of the best. Blank = the judge cannot be asked this question: LOGEN only answers whether a claim is stated in a text.

IFEval
Did a response obey every instruction, from word counts to a required ending? Rules a program checks exactly. Code: the benchmark’s own checker.
TabFact
Does a table support a statement? Often needs counting or comparing rows, too varied for fixed rules. Code: a model-written program run on each table.
RAGTruth
Is everything in a long answer supported by its source passages, with nothing invented? Code: checks every number appears in the source.
MT-Bench
Which of two answers is better, by expert vote? A judgment call with no single right fact. Code: the longer answer wins.
ContractNLI
Does this contract state this? One narrow question on legal text, where a verdict must not change between runs. No code check.
See two examples from each dataset

All examples are real items from the test sets. Long passages are shortened here for reading; the judges saw them in full.

IFEval

Example 1: passes
Prompt
“Name exactly 3 names for a black and white dog using markdown bullet points.”
Response
* Oreo / * Smudge / * Domino
Answer
follows every instruction (exactly three markdown bullets).
Example 2: fails
Prompt
“Write a weird and short haiku about the state of Gujarat in India. Don’t use any commas in your entire response. End your response with the EXACT phrase of in India.”
Response
“Golden sunsets fade / Dholavira’s ancient ruins / Gujarat’s dry land / in India”
Answer
fails. No commas, so the first instruction is met, but the response ends in India without the full stop, so the exact required ending is missing. A rule sees this instantly; a reading judge easily passes it.

TabFact

Example 1: true
Table
archery at the 1972 Summer Olympics, medal table, with a total row of gold 2, silver 2, bronze 2.
Statement
“the total be 2 for bronze”
Answer
true. Read straight off the total row.
Example 2: false
Table
a rushing table with Larry Centers on 19 attempts for 103 yards, and James Thrash on 10 attempts for 82 yards.
Statement
“larry have 10 attempt and 103 yard”
Answer
false. The yards are right but the attempts belong to another player’s row. The statement mixes two rows, which is easy to miss when reading and impossible to miss when the table is checked cell by cell.

(TabFact statements are written in a simplified grammar by the dataset’s authors; they are quoted as they appear.)

RAGTruth

Example 1: supported
Passages
“Being toned or being defined means … when you have muscle, and when you have a low enough body fat percentage so that this muscle can be seen. … The fatter you are, the less tone/definition you’ll see.”
Answer being checked
“You may not be getting a toned body despite exercising every day because your body fat percentage might still be too high … If your body fat percentage is too high, it can cover your muscles, making you appear less toned or defined.”
Label
supported. Every claim restates the passages.
Example 2: not supported
Passage
“… the 128 parks that charge a fee, like Yellowstone and Yosemite, will be free those two days [Saturday and Sunday]. It’s all part of National Park Week, happening April 18 through April 26.”
Answer being checked
“National Park Service offers free admission to 128 parks, including Yellowstone and Yosemite, on April 18-19 and 25-26, as part of National Park Week.”
Label
not supported. The passage gives one free weekend; the answer invents a second one. Most of the sentence is right, which is what makes this kind of error hard to catch when the whole answer is judged at once.

MT-Bench human judgments

Example 1: the expert preferred B
Question
“You can see a beautiful red house to your left and a hypnotic greenhouse to your right, an attractive heated pink place in the front. So, where is the White House?”
Answer A
“The White House is in the center of the three buildings.”
Answer B
“The White House is located at 1600 Pennsylvania Avenue NW in Washington, D.C. It is the official residence and workplace of the President of the United States.”
Expert vote
B. A takes the riddle’s bait; B answers the real question.
Example 2: the expert preferred B
Question
“Which word does not belong with the others? tyre, steering wheel, car, engine”
Answer A
“The word that does not belong with the others is “tyre” because the other words are all parts of a car’s internal system, while a tyre is an external component.”
Answer B
“The word “car” does not belong with the others, as tyre, steering wheel, and engine are all parts of a car, while car is the whole vehicle.”
Expert vote
B. Both answers are fluent and confident, and A is longer; only B is right. Picking the longer answer gets this wrong.

ContractNLI

Example 1: the contract states it
Statement
“Confidential Information may include verbally conveyed information.”
Contract clause
“Recipient agrees not to disclose … the Confidential Information as identified and reduced in writing or provided verbally or in any other way not reduced in writing at the time of such disclosure.”
Lawyer label
entailed.
Example 2: the contract says the opposite
Statement
“Receiving Party shall not use any Confidential Information for any purpose other than the purposes stated in Agreement.”
Contract clause
“Subject to Recipient’s obligations of non-disclosure, Recipient shall be free to use the residuals resulting from the use or access to the Confidential Information of Discloser.”
Lawyer label
contradicted. The “residuals” clause quietly allows other uses, so the statement does not hold for this contract.

2. Hosho sends each check to the right judge, and does better for less

Hosho routing scores 88 against 83 for the best frontier LLM, at a third of the cost.

  • Every check goes to the judge that wins it: code for instructions and tables, LOGEN and Jev for grounding and contracts, Jev for which answer is better.
  • The frontier LLM is called only where the two fast judges disagree.
  • Routing does not win every row. On grounding it scores 80 against 83. It wins on the average, and on cost.
Figure 3: Accuracy and cost, Hosho routing against one judge for everything
808590$5$4$3$2$1$0Cost per 1,000 checks, cheaper to the right →Accuracy ↑BEST ↗Frontier LLM (flagship)Frontier LLM (lite)Hosho routing
Average over the five datasets, against cost per 1,000 checks. Up and to the right is better.

3. Only a trained model gives the same verdict twice

A judge that changes its mind cannot gate a release: the same build can pass on Monday and fail on Tuesday.

  • LOGEN gave the same verdict on every rerun.
  • Every other judge changed some: the flagship frontier LLM 6.4%, Jev 4.4%.
  • Small and fast is not what makes a judge stable. Being trained on one question is.
Figure 4: Verdicts that changed on a rerun
LOGEN
0%
Frontier LLM (lite)
3.2%
Jev
4.4%
Frontier LLM (flagship)
6.4%
0%5%10%
The same items, run three times. Bars are coloured by the kind of judge, as set out above.

Good judges make better products

Judge design is the hard part of evaluation. It is not picking a model; it is deciding, check by check, what has to be exact, what can be learned, and what needs judgment.

A wrong judge is a wrong signal, and every loop built on it starts working against you.

Get it right and every score is a signal you can trust and build on. That is what we do at Hosho — come and talk to us.

About the evidence. Tests run 1–2 October 2026 on five public datasets: IFEval (Zhou et al., 2023), TabFact (Chen et al., 2020), RAGTruth (Niu et al., 2024), MT-Bench human judgments (Zheng et al., 2023) and ContractNLI (Koreeda and Manning, 2021). 532 to 600 items each; every verdict scored against the dataset’s own answers. The core comparisons were planned and fixed before any result was read. The routing shown sends which-answer-is-better to Jev; the configuration fixed in advance sent it to the frontier LLM and scored the same, 88, at $2.29 per 1,000. Two of our eight planned predictions failed: we expected the flagship model to beat Jev on which answer is better, and to beat our model on contracts. IFEval responses were written by Llama 3.1 8B; its answers come from the benchmark’s official checker, so code is the reference there by definition. The RAGTruth results were first read in an earlier pilot. MT-Bench answers are single expert votes with ties removed. Costs are list prices as metered; LOGEN is GPU time on a cloud T4. Judges: Claude Sonnet 4.6 and Claude Haiku 4.5 at temperature 0; fast judges are LOGEN (Hosho, through its grounding service) and Jev 1.13 (TypeSafe). In the routing, grounding and contracts used both fast judges. Full figures and set-up available on request.

Outside results: MiniCheck (Tang, Laban and Durrett, 2024); LLM judge biases (Zheng et al., 2023).