See your AI the way your customers do — and make it better.

The independent evaluation and fix layer for enterprise AI.

Request access Read the research

aligned · live

v1 is live — 2 customers · 3 products · 100,000+ outputs scored.

What our customers say

“Without a doubt, anybody who’s serious about agents really needs this.”

Abhishek Nayak

Co-founder & CEO, AppSmith

Leading low-code platform · 100K+ developers · Kite, their AI marketer + website-building agent

36%

Reduction in issue incidence — experience quality

~5×

Increase in win rate vs competition — design & visual distinctiveness

100,000+

Signals scored online — every live session, customer and agent

Read the AppSmith story →

Products

Three products, one engine.

Hosho Agent Optimizer

Continuous evaluation and improvement of AI output — metrics, calibrated judges, root-cause diagnosis, fixes, delivered in Slack.

Kills: “Is the output actually good enough to ship?”

LangfusePostgresPJR

Explorer23 sessions · 18 tasks in scope · 9 clean · 11 warnings · 3 failing7 days30 days90 daysAll history

The lenses · the five things every session is judged on

Lens 01

Behaves truthfully

Is every claim backed?

0% pass

84 of 221 flagged

Lens 02

Does what was asked

Did it do the task?

0% pass

97 of 231 flagged

Lens 03

Runs its team well

Are handoffs clean?

0% pass

41 of 172 flagged

Lens 04

Good to work with

Is it a good session?

0% pass

9 of 188 flagged

Lens 05

Produces good work

Is the work any good — vs the best alternative?

0% pass

9 of 38 flagged

Primary drivers of gaps

result_groundinglens 138 units

policy_compliancelens 121 units

handoff_timinglens 317 units

+ Add cardMetric · Lens · Module

Recent evaluation runs

RunUnitsClean shareDateReport

nightly-regression3155% (17 of 31)30 Augfull report →

Weekly benchmark2236% (8 of 22)28 Augfull report →

probe-2026-08-141471% (10 of 14)14 Augfull report →

probe-2026-08-07580% (4 of 5)7 Augfull report →

probe-2026-07-31967% (6 of 9)31 Julfull report →

Ask Hosho

Hosho Signals

How customers actually experience your AI — frustration, dead-ends, promoter/detractor sessions, issues triaged and fixed.

Kills: “Are customers quietly struggling — and would we even know?”

HOSHO / SIGNALSInspectorLangfuse

JourneyAppsSignalsControlDatabaseSupport agentDocs agentSoon

— 128 customers · 96 agents · since 9 Jul · updated 4m ago

Discovery

stopped before anything was built

1,204 reached

dropped off · 0 still active

What they hit before leaving

latency_spike968%

repeated_ask615%

slow_first_reply444%

Setup

stopped while configuring

987 reached

dropped off · 0 still active

What they hit before leaving

rage_quit14214%

dead_end889%

context_drop525%

First run

stopped mid-task

681 reached

dropped off · 0 still active

What they hit before leaving

retry_loop7411%

vague_answer528%

clarification_loop315%

Habit

stopped coming back

531 reached

dropped off · 0 still active

What they hit before leaving

stalled_task397%

unresolved_handoff183%

silent_abandon122%

Hosho Prompt & Model Intelligence

Reads how models actually interpret your prompts — prompt demand vs model capability, cheapest model that clears the bar, prompt optimization in your workflow.

Kills: “Are we paying for tokens we don’t need?”

← new analysis

› Your promptReady

01 The model’s readOnly on Hosho

Your prompt · 405 tok

① How the model’s attention splits

Product listing copy30% of effort

title and description from specs

Role and task framing25% of effort

content operations context · migration scope

↕the model moves between these

Url inventory and routing23% of effort

retiring and live URL sets · match logic

Other work22% of effort

two smaller workstreams

② How demanding it is

See rationale per axis›

① The model reads this as four primary jobs, plus an output contract

– Its attention clusters into four jobs — the three largest: product listing copy 30%, role and task framing 25%, url inventory and routing 23%.

– Four jobs, all drawing on product specs, retiring URLs, live URLs.

② 44/100 — moderate for a model to answer

– Reasoning and format drive this; parallel and knowledge are light.

– Two distinct outputs require separate inference: copy synthesis from specs, and URL matching logic.

02 Two ways to run it

Option Aas-is

Run as one call

Nothing changes in your pipeline.

ClosedOpen

Base★cheapest that clears itGemini 3.5 FlashKimi K2.6

Saferclears it with marginGPT-5.2Kimi K2.6

Cheapermay be inconsistentGPT-5 miniKimi K2.5

Copy model recommendations

See capability-load match for all models›

Option Bstructural change

Split into two calls

Split into two independent jobs

Call 1Write product listing copyone title and one description, each meeting its character and register bounds

Call 2Build redirect mapone row per retiring URL: source, destination, and 301 or 410 code

Copy proposed structure

See the call structure›

↳ Difficulty, axis by axisclose ×

Prompt difficulty

0/100Moderate

Reasoning loadi

How much careful, step-by-step thinking your prompt needs before it can answer.

Moderate multi-step reasoning. The prompt chains 8 steps from product data through copy constraints to redirect logic.

Format loadi

How strict and specific the output format has to be.

Some output structure is expected. The probe reads 3 explicit formatting rules in your text.

Parallel demandsi

How many separate things the model has to juggle in a single answer.

There's little to track in parallel. 5 independent task clusters sit in one request.

Knowledgei

How much niche or hard-to-recall knowledge your prompt assumes.

Little external knowledge required. All facts needed — product specs, URLs, constraints — are supplied in the prompt itself.

02 Two ways to run it

Option Aas-is

Run as one call

Nothing changes in your pipeline.

Option Bstructural change

Split into two calls

Split into two independent jobs

↳ Option A · Recommended models for your current structureclose ×

Optimising forModelInOutAbility vs demand

Basecheapest that clears itability ≥ demand

closedGemini 3.5 FlashfrontierDTS pick$1.50$9.00+0.21

closedGemini 3.1 Profrontier$2.00$12.00+0.21

closedo3-promid$20.00$80.00+0.38

openKimi K2.6highDTS pick$0.95$4.00+1.20

Saferclears it with marginability ≥ demand + 0.5

closedGPT-5.2midDTS pick$1.75$14.00+1.21

closedClaude Opus 4.8frontier$5.00$25.00+1.03

closedGPT-5.5frontier$5.00$30.00+1.59

closedClaude Fable 5frontier$10.00$50.00+1.90

closedGPT-5.5 Profrontier$30.00$180.00+1.90

Cheapercould work, may be inconsistentability < demand

closedGPT-5 minilightDTS picknear$0.25$2.00−0.07

closedClaude Opus 4.6frontier$5.00$25.00−0.01

closedClaude Sonnet 4.6mid$3.00$15.00−0.15

openKimi K2.5midDTS picknear$0.60$3.00−0.27

openDeepSeek V4 Flashhigh$0.14$0.28−0.38

Does not meetmodel ability significantly below prompt load

closedClaude Haiku 4.5light$1.00$5.00−1.11

closedGemini 2.5 Flashmid$0.30$2.50−1.14

closedgpt-4olight$2.50$10.00−1.52

openQwen3 14Blight$0.14$0.57−2.10

closedGPT-5.4 nanolight$0.20$1.25−2.40

Assessed across 34 models · 12 clear this prompt

Recommendations based on text-only analysis — verify the picked model supports other modalities if used.

Talk to us about which fits your stack →

How it works

From raw traces to a merged pull request.

01Your AI

Your AI in production

02Ingest

Traces · messages · tool-call logs · approvals · reactions

03Agents

Orchestrator sequences:

Onboarding — writes the rubricEvaluator — calibrates judgesTriage — finds root causeFixer — opens the PR

04Engines

Evaluation engine — deterministic evaluators + LLM judges

Triage engine — deterministic chassis + evidence agent

05Outcomes

Verdicts per evaluator per record · diagnosis with root cause + confidence

Fix PR → human merge — never applied silently

Use Cases

What you can do with Hosho.

Get pinged when quality drops

Issues land in Slack with the evidence attached — not in a dashboard you forget to check.

When Hosho picks up a regression it posts straight to your team channel — what broke, how confident it is, and how many incidents it has seen — then follows up with the root cause and the fix PR. The evidence lands where your team already works.

Hosho

Channels

#general

#engineering

#ai-quality3

#deploys

#incidents

Direct Messages

Jamie L.

Alex T.

#ai-qualityHosho workspace

6 members

HoshoAPP9:14 AM

Hey Jamie — picking up a quality regression on refunds. The assistant's been giving incorrect refund guidance, looks like it's mishandling refund-policy questions when context is thin.

Confidence: high4 incidents today

Jamie L.9:17 AM

Yeah, couple of customers flagged this too. Can you dig in and find the root cause?

HoshoAPP9:17 AM

On it. The prompt's falling back to a generic handler when refund context is missing — and retrieval order isn't surfacing the right policy docs first. I'll open a PR with the fix.

PR opening on GitHub

Message #ai-quality

Stop losing customers to poor AI performance

See every user session and the signals that predict the outcome.

Hosho surfaces every user session with the signals that predict the outcome — a vague answer, dropped context, or a repeat question. See the friction before your users complain about it.

Session	Signals	Verdict

Replace vibe checks with real scale evaluations

One rubric, applied to everything.

Define a rubric once and Hosho applies it to every response, at any volume. You stop spot-checking and start seeing the full picture — which responses meet your standard, and which don't.

Tone · Rubric at scale

5Warm, acknowledges the issue

4Friendly, polite

3Neutral

2Curt or dismissive

1Rude or defensive

12,480 responses graded on tone

avg 4.3/5 · 6% scored ≤2, flagged for review

Never ship a regression

Every version scored across your rubric.

After every deployment, Hosho re-scores your real conversations across every rubric dimension. A version-by-version matrix shows exactly where a score dropped, so you catch the regression before it reaches your users.

Regression Matrix

Dimension	v1	v2	v3	v4
Accuracy	74	✓ 88	86	87
Tone	68	72	✓ 81	80
Context fidelity	61	65	✓ 79	63regression
Resolution	70	75	78	✓ 84

See how you stack up vs competition

Dimension by dimension, with the why.

Run the same conversations through your system and your competitors’. Hosho scores everyone on your rubric, dimension by dimension, and tells you the behavioural reason for each gap.

Benchmark Comparison

Dimension	You	Comp A	Comp B	Comp C
Accuracy	✓ 95	88	82	79
Tone	78 ↓	74	✓ 91	72
Context fidelity	71 ↓	68	✓ 84	76
Resolution	✓ 89	80	77	73

Key Gap

You trail Comp B on tone & context fidelity — its replies are friendlier (more ‘you’ language) and acknowledge the problem before resolving.

Higher-quality signal for improvement loops

Calibrated verdicts your team can act on directly.

Every session gets a calibrated judgment — promoter, neutral, detractor — with the evidence attached. Improvement loops stop arguing about whether the signal is real and start acting on it.

Session verdicts

Refund request, thin context	Detractor
Pricing question	Neutral
Onboarding walkthrough	Promoter

Reduce your team’s workload

A prebuilt platform, evaluation best practices embedded.

No harness to build and no eval infrastructure to maintain. Hosho ships with calibrated judges, rubric workflows, and fix pipelines already wired together — your team reviews outcomes instead of building tooling.

Out of the box

✓Calibrated judges, tuned to your rubric

✓Session monitoring with verdicts and signals

✓Root-cause triage with evidence attached

✓Fix PRs opened for human review

“I think everyone building in AI needs this…”

“This is my #1 issue right now and Hosho is directly making my product better.”

Team

A team that’s been evaluating AI and human outputs for over a decade.

Hosho’s early backers include investors, founders and executives who see the AI trust problem from inside global enterprises and AI labs every day.

Prateik Pothuneedi

Co-founder

200+ evaluators built. 2+ yrs building and testing agentic pipelines. 10 yrs at Bain, including growing the business into new countries. CS engineer by training.

Greg Bandak

Co-founder

10 yrs at Bain measuring and improving customer experience, 2.5 yrs at Salesforce on enterprise agentic AI.

Nitish K

Founding AI Scientist

12 yrs in generative AI, cognitive agents and NLP — including AI research for space applications.

Research

The research behind the platform.

Evaluating and improving agentic systems is an open problem. These systems fail in ways that are hard to observe, hard to attribute, and harder still to fix — and the field’s methods for measuring them are younger than the systems themselves.

Hosho Research works on this problem across its parts, from how evaluators should be built to how their findings should drive repair, and what we learn becomes the platform.

CompanyWhy we exist CustomerAppSmith × Hosho CustomerGhost × Hosho ResearchResearch: The Anatomy of a Blind Spot.

We’re happy to discuss this work, share the harness, or hear what you’re working on: nitish@hoshoai.com

All research →

Stop relying on vibe checks.

Get independent, human-grade evaluation into your AI pipeline today. v1 is live with paying customers.

Request access

Built with Kite