Jev is a solid judge when the answer is in the input, not when it has to be worked out.

1. The short version

1. Jev decides in about a second without reasoning, and matches frontier LLMs on look-up checks at a fraction of their cost.
2. It falls behind on counting, maths and inference; TypeSafe and a Carnegie Mellon study report the same split.
3. Trust its confident verdicts on look-up checks; send the rest, and its unsure verdicts, to a stronger judge or a person.

2. What we tested

This is a closer look at one of the judges from There is no best AI judge. Jev, from TypeSafe, is a general decision model: it answers yes/no, pick-one or score questions with a probability, and writes no text. We compared it with:

• Deterministic checks: rules and short programs, such as an official checker or a model-written program.
• LOGEN: Hosho’s proprietary fine-tuned grounding model. It asks one question, “is this claim in the text?”, and gives the same answer every run.
• Frontier LLMs, lite and flagship: LLM judges that reason in text and return a verdict.

17 studies and over 13,000 labelled items, most with the plan fixed before scoring, for under $2 in Jev fees. Details are in the appendix.

3. Where Jev does well, and where it doesn’t

Code and the flagship win by a wide margin on their tasks (IFEval and TabFact). Everywhere else, the judges are within a few points of each other.

Where it does well: checks where the answer is in the input.

• Is this summary faithful? Level with LOGEN, 74 each.
• Is this claim supported by its source? Level with LOGEN on the public RAGTruth set (75 each), and ahead on 279 claims from a production system (76 against 70).
• Paired with LOGEN: where the two disagreed on those production claims (21% of them), problems were twice as common. Send those to review.

Where it falls behind: checks where the answer has to be worked out: counted, inferred or combined.

• Counts and formats: 69, where code scores 100. Use code.
• Errors that take a step of reasoning: it flagged only 13 in 100 summaries that contained one.
• Saying what went wrong: it named the error type 20% of the time, barely above chance (14%).
• Rubrics with several criteria: it matched LLM judges on one graded label, but was near chance on a four-part tone rubric, even split into separate questions.

Ask Jev one question at a time, and only when the answer is in the input.

4. Its confidence tells you which verdicts to trust

Jev returns a probability with every verdict.

• On four of five tasks, confidence separates right from wrong: 89-97% right when most confident, 54-63% when least.
• Unstable verdicts sit at low confidence: on rerun, 17 of 64 low-confidence verdicts changed, against 5 of the other 436.
• It’s well calibrated: when it says 80%, it’s right about 80% of the time.
• The exception is IFEval, where even confident verdicts were right only 68% of the time.

5. On contracts, Jev misses clauses that aren’t spelled out

Contracts often imply a duty rather than state it, and Jev reads literally, so it misses clauses that are there in different words.

• Jev missed 30% of stated clauses. The frontier LLMs missed 22% (flagship) and 33% (lite). LOGEN, trained on exactly this kind of pairing, missed 6%.
• Example: “the NDA grants no licence”. Contracts usually imply this rather than say it. Jev got it right 32% of the time; LOGEN and the flagship frontier LLM 90%.
• Jev scores well on missing clauses (91 against LOGEN’s 75) only because it says “not stated” so often. At matched recall on stated clauses, LOGEN still catches more of the missing or contradicted ones (77% against 63%).
• When LOGEN and Jev agree, on 74% of items, they’re right 94% of the time. When they disagree, even the best judge is right only 62%, so send those to a final referee, ideally a person.

6. How to use it

Before you let Jev pass or fail anything on its own:

• Test a few wordings, then fix one. Rewording the same question changed Jev’s score by 7 points. Try a few on labelled examples and keep the best.
• Set the threshold for each task. By default Jev leans towards “no”. On IFEval it caught 86% of failures but passed only 53% of good answers. Move the cut-off until both are acceptable.
• Look twice at low-confidence verdicts. About 1 in 20 verdicts changes if you ask again, and most of those are the ones Jev was unsure of.
• Use it to flag, not to block. In a medical triage test, adding Jev as a check cut false alarms but also overruled real emergencies, such as seizures, that the other model had caught. Send what it rejects to review; don’t drop it.

Appendix A: How Jev works

Appendix B: Methodology