Jev Tested on 415 Documents: What Held Up

Field notes, 21 September 2026. Every figure comes from a logged run with the dataset hashed and the model version pinned. I’m not publishing the harness or the data. Figures from other people link to their source. If calibration, AUC or threshold fitting are new to you, how Jev works and how to test it explains them first.

TypeSafe released Jev on 15 September 2026. It does not write text. You give it a block of state and a set of typed questions, and it returns a probability for each question in about 400 milliseconds. Input costs $0.042 per million tokens; output is free. I spent a day and a half testing it against public disaster reports and a bilingual club website, for about $3 in API spend.

The first thing I measured did not test the judgement I cared about. Working out why took most of the first day, and that lesson transfers to almost any Jev project.

First result

I built a scope gate. Given 415 documents reporting on 3 disasters, decide which ones cover the August 2026 Nepal flash flood rather than the Colombia or Indonesia earthquakes. The labels came from the collection each document was filed under, so they were not my judgement.

Jev scored 96.6%, with a calibration error of 0.076 and a median latency of 362 milliseconds, for 2.5 cents across the whole set. I nearly wrote that up.

Then I wrote the simplest baseline I could think of and ran it on the same 415 documents. grep 'nepal' scored 97.3%.

A one-word text search beat the model. The task I had designed did not need semantic judgement at all. The out-of-scope documents were about other countries, so detecting the country name was enough. My 96.6% measured country detection and nothing else.

This is not unusual. An independent analysis of Jev on a phishing dataset found a two-line regular expression scoring 91.8%, against Jev’s 89.4%, on a single question.

It then happened to me again, on the task I had been saving as the hard one. The collection contains a genuine trap: a flood on the same river, in the same district, 13 months earlier, with the same death toll at first report. I built 110 documents by date substitution to test whether a model could tell the 2 apart. Jev scored 94.5%. Counting which year string appears more often in the document scored 97.3%, because the documents state their own dates.

Across three tasks, a few lines of text matching won or tied twice. You cannot tell by reading a problem whether it needs a model. Write the simple baseline and measure it.

Removing shortcuts

A fairer test had to make keywords useless. I rewrote every out-of-scope document, replacing its place names with Nepali ones. Colombia became Nepal, Quindío became Rasuwa, Palu became Timure. A Copernicus product describing an earthquake now carried the headline “Earthquake in Nepal”. I then went the other way and stripped the place names out of the genuinely in-scope Nepal documents, replacing them with generic phrases such as “the upstream district”.

The keyword baseline fell from 97.3% to 18.1%. Jev scored 96.9%, and at the default 0.5 threshold it rejected all 195 of the relabelled earthquakes. Told that a document was about Nepal, about Rasuwa, about the Bhote Koshi, it still read the hazard description and concluded that an earthquake is not a glacial-collapse flood.

That is a real result and the first experiment could not have produced it.

Comparing models

I ran the same set through 2 general models. The first comparison I produced was unfair to both of them.

In that first run, at the default 0.5 threshold, Jev scored 96.9%, Claude Haiku 4.5 scored 66.3% and Gemini 2.5 Flash Lite scored 65.3%. A 30-point gap looks decisive and it does not survive checking.

The fair version scores every model on the same documents with the same method. The table below uses the 398 documents that every model answered, fits each threshold on one half of them, and scores on the other half, 201 documents:

Getting to that table took four corrections, and every one of them had been flattering Jev.

Fitting the threshold rather than defaulting to 0.5 gained Haiku about 21 points. A more tolerant output parser gained Gemini 14 more. A single Python idiom, p or 0.5, treated a probability of exactly 0.0 as a missing value, because zero is falsy in Python; Gemini returns 0.0 often, so its most confident correct answers were being rewritten into the middle of the range, at a cost of about 32 points. And Haiku lost 14 of 415 calls to rate limits, which I was scoring as 0.5, dropping its AUC from 0.993 to 0.965. The fair table simply leaves those 14 documents out for every model.

Jev’s accuracy lead is 1.5 points, not 30. Anyone publishing the uncorrected version of that table is publishing their own configuration.

AUC deserves a note on precision. At 3 decimal places, Jev’s 0.9999 rounds to 1.000 and becomes indistinguishable from a perfect score. On the full set a handful of the 42,900 pairs are ranked the wrong way or tied, so the separation is very good and it is not perfect. The genuinely perfect 1.000000 belongs to the easier first task, which is the one grep won.

What does hold up is the ranking. AUC does not depend on where the threshold goes, which is why it survives the criticism that sank the accuracy comparison. The cost and latency differences are the larger part of the practical case: about 33 times cheaper per correct decision than Haiku, and about 8 times faster.

Recommendation

Then I wrote the baseline I should have written on the first day. TF-IDF and logistic regression, about 40 lines, no dependencies, 5-fold cross-validation grouped by source document so that no rewritten variant could appear in both training and test.

On the adversarial set, where grep manages 18.1%:

Both are scored on all 415 documents, and both fitted figures use a threshold chosen on the same predictions they are scored on, so read those as upper bounds. At the default threshold the classifier is ahead by 0.4 of a point. With a fitted threshold Jev is ahead by half a point. Either way, a model that runs in microseconds on a laptop and makes no network call is within half a point of the decision model.

The classifier needed 415 labels. Jev needed none. That is the whole difference, and it belongs in the recommendation rather than in a footnote: use Jev while you have no labels, and replace it once you do.

I think I know why the simple classifier does so well. The place-name swap removed the location cue and left the hazard vocabulary untouched. Words such as “seismic”, “magnitude” and “aftershock” against “debris flow” and “glacial” are enough, and bag-of-words picks that up directly. So my adversarial set was harder than the first task and still not hard. That is 3 attempts and 3 shortcuts: country names, then year strings, then hazard words.

What the comparison shows

Before the corrections, the split-out numbers told a clean story. On in-scope documents with the place names stripped, and at the default 0.5 threshold, Jev recognised 94.1% while Haiku managed 43.6% and Gemini 45.0%. I wrote it up as a mechanism: the general models need place names to confirm an event, and Jev reads the hazard.

At fitted thresholds those figures are 100%, 96.4% and 92.7%. The mechanism was the 0.5 threshold, not the models. I had found a property of my own configuration and given it a story.

What survives is smaller. On the relabelled earthquakes, at fitted thresholds, every model did well: Jev rejected 99.0%, Gemini 97.4% and Haiku 92.3%. None of them fell for the place-name swap, which is the opposite of what I predicted before running it.

Calibration

Calibrated probabilities are the main claim for this model. TypeSafe train for it with a method they call Reinforcement Learning for Calibrated Decisions, described in their launch post, and the promise is that answers given 90% confidence are right about 90% of the time.

Across 7 of my own runs, at base rates from 10% to 53%, the fitted temperature landed between 0.47 and 0.83. An independent study on entirely different data reports 0.66 for the same kind of yes-or-no question. A temperature below 1 means the model is hedging, so Jev is under-confident, and this is the most reproducible thing I measured.

The reliability curve on the first task says the same more plainly. Every confidence bucket between 0.1 and 0.9 had an observed accuracy of 1.000, so everything the model hedged on turned out to be correct. A 0.35 did not mean “probably not”. It meant “yes, and I am hedging”.

What I could not establish is why. I tested the 3 available explanations and separated none of them. Base rate runs from 10% to 53% with no consistent relationship to the temperature. Difficulty runs from AUC 0.75 to 1.00 with none either. And the explanation by question type, that yes-or-no questions hedge while choice and score over-commit, did not reproduce. My yes-or-no result matched that independent study almost exactly, at 0.64 against 0.66, while my choice came back at 1.02 against their 3.29, and my score pointed the other way.

So the claim is narrow. The effect holds and the explanation does not. That is workable, because the practical advice does not depend on the explanation.

Reuse calibration

The miscalibration belongs to the question rather than to the corpus. A temperature fitted on the easy task lowers the error on the adversarial one, and a temperature fitted on the adversarial task lowers the error on the easy one. It works in both directions.

That is the difference between a one-time calibration step and a permanent maintenance cost. Fit it once per question, reuse it, and re-check it when the model version changes.

Two-parameter calibration

Jev is sharp when it rejects and under-confident when it accepts:

A temperature sharpens both tails equally, so it cannot correct a one-sided error. Adding a bias term can, and on held-out data it takes the error from 0.1430 raw, and 0.0876 with a temperature alone, down to 0.0116.

One warning for anyone repeating this. The task is nearly separable, with 6 errors in 208 training documents, so the fitting problem is badly constrained and the fitted bias runs to whatever limits it. Mine moved from 4 to 8 to 14 to 19 as I widened the search grid. Someone fitting by gradient descent saw the same drift with the number of steps. In both cases the grid width and the stopping point were acting as an accidental prior.

So fit 2 parameters with an explicit penalty, choose the penalty by cross-validation, and report the plain temperature as the comparable summary. Do not quote an unregularised bias, including mine.

Decomposing questions

Published work on a phishing dataset reports a single question at 89.4%, rising to 95.0% when the judgement is split into 5 questions and their answers are weighted by a logistic regression fitted on 1,000 labels. My first attempt at decomposition, with no fitted weights, showed the opposite: accuracy fell from 96.6% to 89.9%.

The first explanation came from the way I combined the answers. I had combined 4 sub-answers with min(), which is the obvious conjunctive choice and the wrong one, and 1 of my 4 questions asked about a date, which Jev reads as text rather than as an ordered quantity. Re-scoring every rule on the identical stored sub-answers gave a spread from 88.0% to 99.8%, with min near the bottom.

Then I re-ran the whole thing on the adversarial set and the ranking inverted. max, which had been best, came last. That looked interesting. It was a broken label.

Two of the 4 sub-questions are invalid by construction. The adversarial transform rewrites place names to Nepali ones, so asking whether a document reports on events in Nepal is scored against an inverted label on every swapped document:

The country question answers correctly and is marked wrong. The places question carries no information either way, because the stripped documents have no place names and the swapped ones have the wrong ones.

Drop the 2 invalid questions and decomposition wins clearly:

mean(hazard, timeframe)   99.52% at 0.5, 99.76% fitted, AUC 0.99995
single question           96.87% at 0.5, 99.52% fitted, AUC 0.99992

The mean at an unfitted 0.5 already matches the fitted single-question number. So decomposition helps. What does not carry across tasks is the combiner: max wins on the easy task and mean wins here. Fit the combiner on the data it will run on, and do not default to min().

Citation checking

I had made 5 attempts at a task with no shortcut and all of them failed. The 6th worked.

It starts from a citation guard of the kind many RAG pipelines have: it checks that a cited figure appears in the cited document. That is a string check, so I built a sample entirely from its blind spot. 415 claim and passage pairs where the figure is present in the cited document, drawn from different source documents, so the only open question is whether the number means the same thing. I labelled 30 of them blind, before seeing any model output.

An AUC of 0.500 means the figure-present check carries no information at all on these pairs, which is what a well-built sample should do to a shortcut. After 5 attempts, this was the first task where the cheap baselines failed.

The finding matters more than the model comparison. 13 of the 30 labelled pairs, 43%, cite a figure that is genuinely present in the passage but means something else there, and a figure-presence check passes every one of them. Here are 4 of the 13:

11 of the 13 are unit mismatches like these. The number matches and the thing being counted does not, so a check that only looks for the number cannot see the error.

I think this is the most useful finding in the whole exercise, and it is also the least finished. It needs more work before anyone should treat 43% as a rate. The test was not comprehensive:

• 30 pairs, 1 reviewer. At this size the 43% could plausibly sit anywhere from about a quarter to about 60%. A second reviewer would also catch judgement calls I would miss.
• The pairs were built, not observed. I matched claims to passages from other documents that happened to share a number. Real summaries cite real sources, so the error rate in genuine output is probably lower than in this sample, and it could be much lower.
• Some mismatches are too easy. No real summary would plausibly cite a count of missing people to a passage listing bottles of water treatment solution. Cases like that inflate the rate.
• 1 event, 1 language, 1 kind of document. Every pair came from reports on one flood.
• No fix was tested. A narrow unit check did well in this test (below), but I have not tried it on real output.

The next steps are clear. Sample citations from real generated summaries rather than constructed pairs, label a few hundred of them with a second reviewer, and then test whether a unit check catches the mismatches without flagging correct citations. Until then, treat this as evidence that the error class exists and is easy to miss, not as a measurement of how common it is.

Check units first

One narrow question, asked of the same 30 pairs, came close to separating them completely: does the number in the passage count the same kind of thing as the number in the claim? It scored 96.7% with AUC 0.991.

The broad question, does the passage support the claim, scored 86.7% at the default threshold on my first run. That is worth acting on whatever you conclude about the model, because the common defect here is not a subtle failure of meaning. It is a unit mismatch, and a narrow check catches it.

Model comparison limits

At 30 labelled pairs, Jev is not significantly better than TF-IDF cosine: +0.064 AUC with a 95% interval from -0.027 to +0.196. That comparison is preliminary for the same reasons as the finding above.

Question wording

My first run asked the claim-level question: does the passage assert what the claim asserts? My labelling rule turned out to be narrower. The passage must assert the specific figure cited, and the rest of the claim needs its own sources. Re-running with the question matched to the rule moved accuracy from 86.7% to 96.7% and the calibration error from 0.158 to 0.111.

AUC did not move at all. The ranking was right the whole time, and 10 points of accuracy came from the wording alone.

Guard questions

Jev cannot abstain. There is no “not applicable” in its output. Asked a question, it must return a probability.

I asked a set of documents whether the figures in each one described our event. Some of those documents contained no figures at all. Asked whether absent figures matched, the model returned a low probability, which looks exactly like “the figures are wrong”. A thematic advisory with no numbers in it was flagged as carrying another event’s data.

The fix is to split the question. Ask whether figures exist, ask whether they match, and combine the 2 answers in code. It is the same principle TypeSafe’s own documentation gives for dates: extraction is a judgement, so give it to the model, and the logic stays in code.

I applied that in a 7-step publishing gate for a bilingual website, together with a second change described in the next section. With both applied, false positives on the held-out clean pages went from 2 of 5 to 0. I made the two changes at once, so I cannot credit either one on its own.

Any question that presupposes something needs a guard question in front of it. That is obvious in hindsight and it caught me twice.

Keep exact logic in code

Jev is not a calculator, and dates are text to it. In the same 7-step gate I gave it 2 steps that code does exactly: matching link counts and comparing number sets. The link-matching step produced every false positive in that configuration, and the number-comparison step never fired. Moving both back into Python was the second of the two changes above.

It also produces no rationale. A probability is not an audit trail, so it is not suitable for any decision that has to be explained to someone who can contest it.

How to use Jev

Write the simplest baseline and measure it. Much of what looks like a judgement problem is a string-matching problem, and you will not know which you have until you check. It cost me 3 tasks to learn that.

Then write the second baseline. Fit TF-IDF and logistic regression on whatever labels you have or can produce, before you price a model. Mine came within half a point on the hardest task I could build, for nothing. A candidate task has to survive both baselines, not 1.

If both baselines lose, you may have a real use for the model. Split the judgement into narrow questions and ask them in a single call, since questions run in parallel and a tenth one costs almost nothing. Fit the rule that combines them, because the best combiner on 1 task was the worst on another. Put a guard in front of any question that presupposes something, because the model cannot answer “not applicable” and a low probability looks the same as a real finding. Keep counting, dates and schema validation in code, where they are free and exact.

Fit your threshold on your own labelled data. The default of 0.5 cost 1 comparison model about 21 points, and it would have given me the wrong answer about which model to use. Fit the calibration once per question and reuse it, since it transfers across task difficulty.

And if you rewrite documents to build a harder test, check every question on every variant before you believe any number computed on them.

Notes on method

The model was pinned to jev-1.13-20260917. Thresholds were fitted on a training split and scored on held-out data, and the small-sample results carry bootstrap intervals.

Validate transformed data

Rewriting documents to remove a shortcut breaks ground-truth labels silently. It happened 4 times here: 2 of 3 arms in 1 experiment, and 2 of 4 sub-questions in another. Each time the overall label looked fine while 1 specific question was being scored against its opposite, and each time I found it by reading rather than by testing.

There is a mechanical check, and it has 3 parts. Each part is justified by a failure the other 2 miss.

• Score every sub-question on each variant separately. One arm passed at the whole-experiment level, at 83.4% against a 59.5% baseline, and failed only once the rewritten documents were scored on their own, at 65.1%.
• Compare against the majority-class rate, not 50%. A useless question still beats chance on an imbalanced slice by answering the majority every time. Both broken arms sat at 69.7% and 65.1% against an 86.2% baseline, which is above chance and below useful.
• Fall back to 50% where a variant contains only 1 class, since the majority rate is then 100% by construction and would flag everything.

Below the reference means the label is inverted, which is often repairable. Level with the reference means the question carries no information, which is not: drop it. The whole check runs on stored answers with no API calls, and it catches all 4 known failures with no false positives.

Limits

The adversarial set was built by mechanical place-name substitution, so it is harder than the original task and still not hard enough, since a bag-of-words classifier matched the model on it. The real trap for this collection is the July 2025 flood in the same district on the same river, and I have 4 real documents of it rather than a sample. A measured 10-point accuracy drop on Nepali documents is confounded with document type. I have not put any of this into use yet.

The largest limitation is that I wrote the mutations, the questions, the labels and the transforms. The labels that every accuracy figure rests on came from folder membership, are known wrong at least 5 times, and every known error was in the model’s favour. An audit of 99 documents is prepared but not yet done, and it will not be independent either, because I wrote the scope definition it is judged against. Until then, every number here carries a bias of unknown size in 1 known direction.

Additional findings

One published analysis of Jev found that the same question gets a different answer depending on the format it is asked in. I tested the milder version of that: wording. Four phrasings of the same question on the adversarial set produced a swing of 1.4 points, from 96.1% to 97.6%. The best of the 4 was a deliberate double negative, which TypeSafe’s own documentation says is answered less reliably. Straightforward rewording is not a major risk here.

Input length matters even less. Truncating documents from 6,000 characters to 400 cost 1.0 point and saved 60% of the bill, with the ranking almost unchanged. For a gate running over thousands of documents, sending a title and a first paragraph is the cheapest saving available.

Three repeat runs at 415 documents on a pinned build returned identical accuracy to 6 decimal places. AUC moved at the 5th decimal and the calibration error at the 4th, so accuracy is stable at this scale and finer quantities are not.