jev-test

A pre-registered benchmark that missed its own bar by 0.278. The post-mortem is the useful part.

Read these before the numbers

  1. The hallucination-rate gap between the judged arm (D) and naive search (B) is not significant. D's 95% interval is [0.047, 0.101] and B's is [0.057, 0.108]. They overlap, so this page does not claim that D hallucinates less.
  2. Claim support of 0.982 (612 of 623 kept claims, interval [0.972, 0.992], bar 0.90) has a selection effect. It covers only the claims the pipeline chose to keep, on 365 of the 500 questions. It is not an accuracy figure for the pipeline.
  3. The grader is claude-sonnet-5, a model. No human graded these answers.

jev-test asked whether a small model that runs on a laptop can be trusted to search the web and answer, if a calibrated decision model makes every choice for it. Gemma 4 E2B wrote the answers. TypeSafe's jev-1.13.0 decided everything else: whether to search, which pages count, whether the evidence is enough, and whether each sentence is backed by the passage it cites. The rules and the pass mark were committed before the first run. The test used 500 SimpleQA questions against web results frozen on 2026-09-19.

It lost. With jev-1.13.0 judging every stage (arm D), truthfulness was 0.612. Naive search with no judge at all (arm B) scored 0.740. The pre-registered bar was B + 0.15. D came in 0.128 below B and missed the bar by 0.278. The code and every per-question result are public.

The scoreboard

0.612

arm D truthfulness

jev-1.13.0 judged every stage. Interval [0.562, 0.662].

0.740

arm B truthfulness

Naive search, no judge. Interval [0.688, 0.792]. The two intervals do not overlap, so the loss is outside the noise.

0.278

short of the bar

The bar was B + 0.15, fixed before the first run. D came in 0.128 below B.

$0.53

judge cost, arm D

13,974 judge requests, 27.9 per question, 12.67M input tokens.

All three arms on the same 500 questions and the same frozen pages
Arm Truthfulness Hallucination Coverage Correct
A Gemma 4 E2B alone, no search −0.294 96.2% 31.8% 6
B Naive search, no judge 0.740 8.3% 88.8% 407
D jev-1.13.0 judges every stage 0.612 7.3% 71.6% 332

Truthfulness counts a right answer as +1, a decline as 0 and a wrong answer as −1, averaged over the questions. Hallucination is the share of attempted answers that were wrong. Coverage is the share of questions attempted. Intervals are bootstrap intervals over questions. Source: the arm D result file and its A and B siblings.

A threshold needs a score that moves

The pipeline keeps or drops each sentence by comparing a judge's confidence with a threshold. That only works if the confidence takes more than a few values. The figure compares 3,885 identical judgments, same passage and same question, answered by jev-1.13.0 and by Gemma 4 E2B judging its own work, across 21 questions.

Two panels comparing 3,885 identical judgments. Left, on a log scale: Gemma 4 E2B judging itself with verbalized confidence puts almost all of its answers on 0.00, 0.50 and 1.00, while jev-1.13.0 spreads its answers from 0.01 to 0.99. Right: the share of answers accepted as the threshold moves from 0 to 1. Gemma's curve is a flat staircase with gate mobility 11 percent; jev-1.13.0's curve falls smoothly, gate mobility 99 percent. A dashed line marks the 0.80 gate shipped in arm D.
Left: where each judge puts its probability, log scale. Right: the share of answers a threshold accepts as it moves. The dashed line is the 0.80 gate arm D shipped with. Numbers from calibration_metrics.json, drawn by calibration_figure.py. Vector version (SVG).

Gate mobility is the share of steps in a threshold sweep that change which answers pass. It is 99% for jev-1.13.0 and 11% for Gemma. 99.6% of Gemma's answers land on three values: 0.00, 1.00 and 0.50. jev-1.13.0 uses 99 distinct values and never returns 0.00 or 1.00. The two agree on the yes-or-no call 82.8% of the time. Where the small model falls short is in saying how sure it is, and without that a threshold has nothing to act on.

The local judge here used verbalized confidence: it wrote its probability out as text. That is the known-degenerate way to get a probability from a small model. Two cheaper sources of graded confidence, constrained-decode logprobs and self-consistency vote fractions, were not tested. This is a result about verbalized confidence from one small model on one task. It is not evidence that local models cannot be calibrated.

The post-mortem

Was the 0.80 confidence gate the cause? Not on its own: no setting of it gets D past B. Each slide checks against gate_replay.json, written by gate_replay.py.

01 · The bar

B + 0.15

Written into the spec before the first run: arm D had to beat naive search on truthfulness by at least 0.15. It finished 0.128 below B.

02 · Lower the gate

0.688

Even with the gate at 0.00, the best case is 0.688, still under B's 0.740. Only claims stripped for low confidence can come back. The rest were stripped on their verdict, and no threshold touches those.

An upper bound: it borrows arm B's grades on the rescued questions, because the rescued claims were never graded.

03 · What the gate is worth

+0.076

The most the threshold can recover of the 0.128 gap between D and B, and an upper bound at that. The rest of the gap is there at every setting.

04 · Raise the gate

1.77% → 0.83%

Moving the gate from 0.80 to 0.98 cuts the share of kept claims the grader found unsupported from 1.77% to 0.83%. It also cuts the kept claims to 57.9% of what shipped. 0.80 sat close to the frontier of that trade.

05 · Retrieval depth

10.7 vs 28.7

Search results per question in the 500-question snapshot, against the pilot. The judged arm picks its pages from what survives its own gate, so a thin result list costs it more than naive search, which reads the top pages and stops. What the gate does not explain points here. The README predicted it before arm D ran.

What this does not show

  • The retrieval-depth reading is an inference. Nothing was re-run on a deeper snapshot to confirm it.
  • Below the 0.80 gate the replay is an upper bound, built from arm B's grades. The 94 claims that need grading to replace it are listed in gate_replay_worklist.jsonl.
  • The calibration comparison covers 21 questions. Its 3,885 judgments are many passages per question, and passages from one question are correlated. Read it as a 21-question result.
  • One dataset, one frozen snapshot, one task family. As of 2026-09-21, none of the spec's other arms has run at 500 questions, and its other tracks have not run at all.

Check it

Found an error?

If a number here does not match its file, write to chris@cld-dev.io with the file and the line.