A better judge won’t fix your evals. A world model will.

How do you build an eval you can trust?

If you’re running agents in production, you probably have some version of an eval loop: judges scoring traces, failures getting triaged, changes checked against a suite before they ship. More and more of that loop runs without a human in it. So the real question isn’t whether you have evals — it’s what it takes for a judge’s verdicts to deserve the authority you’ve given them.

Designing a judge comes down to two foundational choices: which judge you use, and what context you give it. Nearly all the effort goes into the first. Most research asks what the judge is; almost none asks what the judge sees. We built Hosho around the opposite conviction: a judge can only be trusted if it has a world model of the agent it grades — what was asked, what the agent did, and what its sources say.

We didn’t want to assert that; we wanted to measure it. So our first paper isolates the effect of what a judge sees on what it can catch: hold the judge fixed, vary only what it is allowed to know, run it across seven public benchmarks (e.g. RAGTruth, WMT-MQM, IFEval).

Three findings, none of them obvious

1. You cannot buy your way out with a better judge.

A frontier model judging from the output alone reaches 0.79 recall on human-annotated hallucination spans — below the 0.92 a mid-tier model reaches given the source. When your eval misses things, upgrading the model isn’t the fix; widening what the judge sees is.

Figure 1: A better judge does not make up for not seeing the source

2. The failures that matter most leave no trace in the output.

Grammar and formatting, an output-only judge catches fine. Omissions, contradicted claims, ignored constraints — the failures that matter in agentic work — it cannot see, and its miss rate rises in direct proportion to how much of the evidence sits elsewhere. False alarms are the other half: without the world, the judge flagged error-free outputs 2.5x as often — and a judge that flags four in five correct outputs is one your team stops reading. It isn’t more context that helps; it’s the right world: even a correct reference answer beside the real evidence made the judge worse — it started grading against the answer instead of checking the evidence.

3. An output-only judge’s feedback makes your system worse.

We fed each judge’s defect reports back to the model, let it revise, and scored every revision with code — no judge grading the repairs. Acting on the output-only judge’s reports left the system 17 points worse than no feedback at all: every false alarm became an edit, and edits break working behavior. What decided whether feedback helped was not how many real defects a judge found, but the precision of its reports. If your judge feeds a loop — retries, auto-fixes, regression gates — its precision decides whether you’re improving your product or injecting defects into it.

Figure 2: Acting on a blind judge’s reports breaks more than it fixes

Figure 3: Each round of blind feedback leaves the system worse than the last

What an agent world model unlocks

Everything above points to one conclusion: reliable evaluation requires a world model of the agent — what was asked, what the agent did, and what its sources say. With it, a judge can do the two things that make an eval trustworthy: confirm a failure is real, and point toward why it happened. That’s the signal a team trusts enough to wire into its improvement loop; without it, the eval gets ignored — or followed down the wrong path.

This is what the Hosho platform is built to do: assemble a complete world model of each of your agents and judge against it, so you get eval signal you can trust and fixes you can act on.

For the research-minded

The paper is in the process of being published, and will be released alongside the harness and the full prediction ledger — failures included. We’re happy to discuss this work, share the harness, or hear what you’re working on: nitish@hoshoai.com