Introduction

Hosho is the evaluation layer for AI pipelines. It scores every step of your pipeline on real production runs, against your own definition of good — and hands back the runs that failed, with the text that failed them.

One loop, operated from wherever you already work.

AI pipelines fail quietly. A step that confidently produces the wrong thing does not throw an exception. A prompt edit that regresses quality ships green. The block that worked in testing behaves differently on real traffic, and the first alarm is a user complaint.

Monitoring and unit tests were built for deterministic code. They cannot tell you whether a summary was faithful, whether a verifier approved a claim with nothing behind it, or whether last week’s model swap made the output better or worse. In a multi-step LLM system, the behaviour that matters most is a judgment call — and judgment calls need judges.

What Hosho does:

• Grades every block on real runs, not on a synthetic suite that stops resembling production the week after it is written.
• Attributes failures to a specific block— the verifier, the ranker, the composer — never to “the AI” in general.
• Requires evidence. Every signal must quote the exact run text it fired on. A tag that cannot cite the text does not count.
• Scores by counting.“3 of 50 runs failed here”, with those three attached — no opaque aggregate.
• Writes results into your database, where you can query them with plain SQL alongside the rest of your data.

Install

One command. You authorize on your database provider’s own page, so no password or token is ever pasted into a terminal, an email, or a chat.

curl -fsSL https://raw.githubusercontent.com/HOSHO-AI/install/main/install.sh | sh
./hosho install

You pick the project and tick which tables Hosho may read — that list is the fence, and it is recorded as the fence. Nothing is created until you confirm. The installer then creates two tables, hosho.runs and hosho.results, in your own database under a role scoped to them.

Ways in

Every surface is a thin client over the same evaluation API, with the same permissions and the same audit trail.

MCP

Hosho ships as an MCP server, so its verbs are tools wherever MCP runs — your IDE, your coding agent, or your own product’s agents. Ask in plain language; get scores with quoted evidence back.

GitHub

Regression results and prompt reviews arrive as comments on the pull request, so a change is measured before it merges rather than after it ships.

HTTP API

The same paths every other surface uses. Push runs, trigger evaluations, poll long jobs, and pull results with their evidence.

Slack & web

Ask for evaluations and diagnoses in conversation, with flagged sessions posted to a channel you choose — or read runs, scores and evidence on the dashboard.

Core concepts

Evaluators

Judges bound to one block each. 22 built-in types covering conversation, content, visual and rule-based checks, plus a prompt-anchored type for anything bespoke. Hosho recommends them; you configure and own them.

Signals

A named behaviour an evaluator watches for — a failure mode or a strength, from a versioned taxonomy. Counts roll up to a per-block health percentage that drills straight through to the runs behind it.

Baselines

Freeze a scored input set, then re-run the changed pipeline against it for a defensible before and after. Golden sets pin the scoring machinery itself, so a number from last month and one from today are comparable.

Experiments

Tournaments, regression verdicts and benchmarks over your real outputs. Every pair is judged in both presentation orders, so position bias is measured rather than assumed away.

Where your data lives

• Results are written into your own database — the two tables the installer creates. Hosho keeps a run record of what ran and when.
• Hosho writes nothing else. It reads the tables you grant and writes only those two. It does not modify your data, your schema, or your pipeline.
• Credentials transit once. The scoped database login is served on an authenticated call, held in memory, and never logged or persisted. The role it grants is confined to the tables you ticked — it cannot reach the rest of your project.
• Changes reach you as pull requests you review. Nothing merges itself.
• SOC 2. Hosho does not hold a report today. The platform is built against the criteria, and we will walk through where we stand control by control — tell us which ones your auditor is testing.

Running today

Chat-transcript analysis, ingestion-pipeline grading, workflow-builder checks, and website and content evaluation — all running as configuration on the same engine, across OpenAI, Anthropic, Google and xAI models.

This page describes what is running today. It grows as the platform does — sections are added as capabilities ship, rather than listed ahead of them.