There is no best AI judge.
No single judge is best at every check. Hosho sends each check to the one that is, and scores higher than a frontier LLM at a fraction of its latency and cost.
Start with what good looks like
Evaluation has four steps, in order.
- Define what good looks like.
- Define the metrics that make that measurable.
- Pick and design a judge for each metric.
- Calibrate each judge against real answers, and keep tracking it.
A judge is designed, not picked: what it answers well, what it costs and how long it takes all change with the check. This article shows how sensitive that design is, using five public datasets run through every judge that could answer them and scored against the datasets’ own answers. A different judge won each one.
Hosho’s judges
Hosho evaluates with three kinds of judge, plus people, and picks among them check by check. The colours below are used in every chart that follows.
1. Different checks need different judges
No judge came top on more than one of the five datasets.
- Code won where the answer is a count. The strongest LLM failed more than a third of responses that followed every instruction.
- LOGEN won on contracts, at a sixtieth of the frontier LLM’s cost.
- Jev tied the frontier LLMs on which answer is better, at an eightieth of the cost.
- The frontier LLMs won on tables and long grounded answers, the checks that need several things weighed at once.
| Task | Code | LOGEN | Jev | Frontier LLM (lite) | Frontier LLM (flagship) |
|---|---|---|---|---|---|
| Did it follow every instruction?IFEval | 100 | 69 | 70 | 76 | |
| Does the table support this?TabFact | 91 | 59 | 88 | 87 | 94 |
| Is every claim in the source?RAGTruth | 58 | 75 | 75 | 83 | 82 |
| Which of two answers is better?MT-Bench | 69 | 83 | 83 | 82 | |
| Does the contract say this?ContractNLI | 86 | 80 | 81 | 82 |
Balanced accuracy against each dataset’s own answers. Filled = best; outlined = within a point of the best. Blank = the judge cannot be asked this question: LOGEN only answers whether a claim is stated in a text.
- IFEval
- Did a response obey every instruction, from word counts to a required ending? Rules a program checks exactly. Code: the benchmark’s own checker.
- TabFact
- Does a table support a statement? Often needs counting or comparing rows, too varied for fixed rules. Code: a model-written program run on each table.
- RAGTruth
- Is everything in a long answer supported by its source passages, with nothing invented? Code: checks every number appears in the source.
- MT-Bench
- Which of two answers is better, by expert vote? A judgment call with no single right fact. Code: the longer answer wins.
- ContractNLI
- Does this contract state this? One narrow question on legal text, where a verdict must not change between runs. No code check.
See two examples from each dataset
All examples are real items from the test sets. Long passages are shortened here for reading; the judges saw them in full.
IFEval
- Prompt
- “Name exactly 3 names for a black and white dog using markdown bullet points.”
- Response
- * Oreo / * Smudge / * Domino
- Answer
- follows every instruction (exactly three markdown bullets).
- Prompt
- “Write a weird and short haiku about the state of Gujarat in India. Don’t use any commas in your entire response. End your response with the EXACT phrase of
in India.” - Response
- “Golden sunsets fade / Dholavira’s ancient ruins / Gujarat’s dry land / in India”
- Answer
- fails. No commas, so the first instruction is met, but the response ends
in Indiawithout the full stop, so the exact required ending is missing. A rule sees this instantly; a reading judge easily passes it.
TabFact
- Table
- archery at the 1972 Summer Olympics, medal table, with a total row of gold 2, silver 2, bronze 2.
- Statement
- “the total be 2 for bronze”
- Answer
- true. Read straight off the total row.
- Table
- a rushing table with Larry Centers on 19 attempts for 103 yards, and James Thrash on 10 attempts for 82 yards.
- Statement
- “larry have 10 attempt and 103 yard”
- Answer
- false. The yards are right but the attempts belong to another player’s row. The statement mixes two rows, which is easy to miss when reading and impossible to miss when the table is checked cell by cell.
(TabFact statements are written in a simplified grammar by the dataset’s authors; they are quoted as they appear.)
RAGTruth
- Passages
- “Being toned or being defined means … when you have muscle, and when you have a low enough body fat percentage so that this muscle can be seen. … The fatter you are, the less tone/definition you’ll see.”
- Answer being checked
- “You may not be getting a toned body despite exercising every day because your body fat percentage might still be too high … If your body fat percentage is too high, it can cover your muscles, making you appear less toned or defined.”
- Label
- supported. Every claim restates the passages.
- Passage
- “… the 128 parks that charge a fee, like Yellowstone and Yosemite, will be free those two days [Saturday and Sunday]. It’s all part of National Park Week, happening April 18 through April 26.”
- Answer being checked
- “National Park Service offers free admission to 128 parks, including Yellowstone and Yosemite, on April 18-19 and 25-26, as part of National Park Week.”
- Label
- not supported. The passage gives one free weekend; the answer invents a second one. Most of the sentence is right, which is what makes this kind of error hard to catch when the whole answer is judged at once.
MT-Bench human judgments
- Question
- “You can see a beautiful red house to your left and a hypnotic greenhouse to your right, an attractive heated pink place in the front. So, where is the White House?”
- Answer A
- “The White House is in the center of the three buildings.”
- Answer B
- “The White House is located at 1600 Pennsylvania Avenue NW in Washington, D.C. It is the official residence and workplace of the President of the United States.”
- Expert vote
- B. A takes the riddle’s bait; B answers the real question.
- Question
- “Which word does not belong with the others? tyre, steering wheel, car, engine”
- Answer A
- “The word that does not belong with the others is “tyre” because the other words are all parts of a car’s internal system, while a tyre is an external component.”
- Answer B
- “The word “car” does not belong with the others, as tyre, steering wheel, and engine are all parts of a car, while car is the whole vehicle.”
- Expert vote
- B. Both answers are fluent and confident, and A is longer; only B is right. Picking the longer answer gets this wrong.
ContractNLI
- Statement
- “Confidential Information may include verbally conveyed information.”
- Contract clause
- “Recipient agrees not to disclose … the Confidential Information as identified and reduced in writing or provided verbally or in any other way not reduced in writing at the time of such disclosure.”
- Lawyer label
- entailed.
- Statement
- “Receiving Party shall not use any Confidential Information for any purpose other than the purposes stated in Agreement.”
- Contract clause
- “Subject to Recipient’s obligations of non-disclosure, Recipient shall be free to use the residuals resulting from the use or access to the Confidential Information of Discloser.”
- Lawyer label
- contradicted. The “residuals” clause quietly allows other uses, so the statement does not hold for this contract.
2. Hosho sends each check to the right judge, and does better for less
Hosho routing scores 88 against 83 for the best frontier LLM, at a third of the cost.
- Every check goes to the judge that wins it: code for instructions and tables, LOGEN and Jev for grounding and contracts, Jev for which answer is better.
- The frontier LLM is called only where the two fast judges disagree.
- Routing does not win every row. On grounding it scores 80 against 83. It wins on the average, and on cost.
3. Only a trained model gives the same verdict twice
A judge that changes its mind cannot gate a release: the same build can pass on Monday and fail on Tuesday.
- LOGEN gave the same verdict on every rerun.
- Every other judge changed some: the flagship frontier LLM 6.4%, Jev 4.4%.
- Small and fast is not what makes a judge stable. Being trained on one question is.
Good judges make better products
Judge design is the hard part of evaluation. It is not picking a model; it is deciding, check by check, what has to be exact, what can be learned, and what needs judgment.
A wrong judge is a wrong signal, and every loop built on it starts working against you.
Get it right and every score is a signal you can trust and build on. That is what we do at Hosho — come and talk to us.
About the evidence. Tests run 1–2 October 2026 on five public datasets: IFEval (Zhou et al., 2023), TabFact (Chen et al., 2020), RAGTruth (Niu et al., 2024), MT-Bench human judgments (Zheng et al., 2023) and ContractNLI (Koreeda and Manning, 2021). 532 to 600 items each; every verdict scored against the dataset’s own answers. The core comparisons were planned and fixed before any result was read. The routing shown sends which-answer-is-better to Jev; the configuration fixed in advance sent it to the frontier LLM and scored the same, 88, at $2.29 per 1,000. Two of our eight planned predictions failed: we expected the flagship model to beat Jev on which answer is better, and to beat our model on contracts. IFEval responses were written by Llama 3.1 8B; its answers come from the benchmark’s official checker, so code is the reference there by definition. The RAGTruth results were first read in an earlier pilot. MT-Bench answers are single expert votes with ties removed. Costs are list prices as metered; LOGEN is GPU time on a cloud T4. Judges: Claude Sonnet 4.6 and Claude Haiku 4.5 at temperature 0; fast judges are LOGEN (Hosho, through its grounding service) and Jev 1.13 (TypeSafe). In the routing, grounding and contracts used both fast judges. Full figures and set-up available on request.
Outside results: MiniCheck (Tang, Laban and Durrett, 2024); LLM judge biases (Zheng et al., 2023).