# How Jev Works, and How to Test a Decision Model > Jev answers typed questions with probabilities instead of text. What that means, and how to test a model like it: baselines first, fitted thresholds, AUC, calibration, guard questions, and checking data you have rewritten. - Author: Fawad Hussain Syed, software engineer, Geneva (https://fawadhs.dev/about) - Published: 2026-09-21 - Category: Automation - Canonical: https://fawadhs.dev/blog/how-jev-works-and-how-to-test-it --- *Field notes, 21 September 2026. Written for a reader who has not met calibration, AUC or threshold fitting before. Every figure comes from a logged run with the dataset hashed and the model version pinned. I'm not publishing the harness or the data.* A model that cannot write a sentence sounds like a step backwards. It is not. The useful distinction is not whether the model can talk, but whether your software needs prose at all. TypeSafe released Jev on 15 September 2026. You give it a block of state and a set of typed questions, and it returns a probability for each one, all at the same time, in about 400 milliseconds. It writes no text at all. ```python ask(state={"document": text}, questions={"is_urgent": noul("This message conveys urgency")}) # -> {"is_urgent": {"noul": 0.94}} ``` That 0.94 is not a word the model wrote; it is a number your code can branch on. There is no parsing, no JSON mode, and no handling the run where the model replies “Sure!” before getting to the answer. I spent a day and a half measuring it against public disaster reports and a bilingual website, for about $3 in API spend. Most of what I learned was not about Jev. It was about how easy it is to run a careful-looking experiment that measures the wrong thing. I did that three times before I got it right. ## 1. Start with a baseline My first test looked rigorous. 415 real documents from 3 collections of public disaster reports, deciding which ones report on the August 2026 Nepal flood rather than the Colombia or Indonesia earthquakes. The labels came from which folder each document was filed in, so they were not my opinion. Jev scored 96.6%. Then I wrote the simplest baseline I could think of. ```python "nepal" in document.lower() ``` That scored 97.3%. One line of Python beat the model. The task looked semantic and was not. Every out-of-scope document was about another country, so detecting the country name was enough. I had measured country detection and was about to call it judgement. This is not unusual. [An independent analysis of Jev on a phishing dataset](https://xenospectrum.com/en/jev-typesafe-bert-classifier-decomposition/) found a 2-line regular expression scoring 91.8% against the model's 89.4% on a single question. It then happened twice more. I built a harder test by rewriting every out-of-scope document so that its place names said Nepal, which drops the keyword baseline to 18.1% while Jev holds at 96.9%. That worked. But on a third task, 110 documents built by date substitution to separate 2 floods on the same river 13 months apart, Jev scored 94.5% and counting which year string appears more often scored 97.3%, because the documents state their own dates. Across three tasks, a few lines of text matching won or tied on two of them. You cannot tell by reading a problem whether it needs semantics. Measure the 5-line version before you price anything. ## 2. Add a classifier baseline The other baseline is a classifier you train yourself. I used TF-IDF and logistic regression, about 40 lines, no dependencies. TF-IDF means: count the words, weight down the ones that appear everywhere such as "the" and "report", and weight up the distinctive ones. Each document becomes a vector of numbers. Logistic regression then learns one weight per word and adds them up. On the hard task, where keyword matching manages 18.1%: | | at 0.5 | threshold fitted | cost | latency | |---|---|---|---|---| | Jev | 96.9% | 99.5% | $0.0258 per 415 documents | 360 ms | | TF-IDF and logistic regression | 97.3% | 99.0% | nothing | microseconds | Both fitted figures use a threshold chosen on the same predictions they are scored on, so read them as upper bounds. At the default threshold the classifier is ahead by 0.4 of a point; with a fitted threshold Jev is ahead by half a point. The two are within half a point either way. The classifier needed 415 labels and Jev needed none. That is the whole difference, and it is the recommendation: **use the zero-shot model while you have no labels, and replace it once you do.** Why did bag-of-words do so well on a task built to be hard? Because the test still leaked. Removing place names left the hazard vocabulary untouched, so "seismic", "magnitude" and "aftershock" against "debris flow" and "glacial" were enough. That is 3 attempts at a hard task and 3 leaks. ## 3. Choose a threshold There are two ways to score a model that outputs probabilities, and confusing them cost me a wrong conclusion. **Accuracy** needs a cut-off. Pick 0.5, call anything above it yes, and count how often you were right. The problem is that 0.5 is a convention rather than a property of the model. **AUC** ignores the cut-off. Take every pairing of one true-yes document with one true-no document, and measure the fraction where the model scored the yes higher. 1.0 means every pair was ordered correctly. 0.5 means the ordering is no better than a coin flip. The difference was decisive. Scored on the same held-out half of the documents, 201 of them, with the threshold fitted on the other half: ``` scored at 0.5 with a fitted cut-off Claude Haiku 4.5 76.1% 97.5% ``` Haiku gained about 21 points from changing 1 number. My original comparison made it look hopeless, when it was being scored at a threshold that happened to suit it badly. The model hands you an ordering, and choosing the cut-off is your job. Fit it on a training split and score it on held-out data. Jev's best cut-off on my task was 0.09, nowhere near 0.5. ## 4. Calibrate probabilities A model is calibrated when, across all the times it says 70%, it is right about 70% of the time. Most language models are poor at this: ask for a confidence and almost everything comes back as 95%. Jev's claim, per [TypeSafe's launch post](https://typesafe.ai/blog/introducing-system-one-models-and-jev), is that its numbers mean something. The measurement is ECE, expected calibration error. Bucket the predictions by stated confidence, compare each bucket's confidence against its actual hit rate, and average the gaps, weighted by bucket size. Zero is perfect. Here is what the buckets looked like on the first task: | confidence bucket | n | stated | actual | |---|---|---|---| | 0.1 to 0.2 | 5 | 0.162 | 1.000 | | 0.5 to 0.6 | 2 | 0.590 | 1.000 | | 0.7 to 0.8 | 19 | 0.768 | 1.000 | | 0.8 to 0.9 | 62 | 0.870 | 1.000 | Every bucket between 0.1 and 0.9 was right every time. The model is not wrong, it is **under-confident**. A 0.35 did not mean "probably not". It meant "yes, and I am hedging". One caution about ECE itself. A perfectly calibrated model measured on a finite sample still shows a non-zero error, and how large depends on the shape of the predictions and the sample size. On this task the floor was 0.031 for Jev and 0.026 for one of the comparison models, so raw ECE figures from different studies are not directly comparable. Report the ratio to your own floor alongside the raw number. The standard fix is temperature scaling. Convert the probability to a logit, divide by a constant, and convert back. A constant below 1 sharpens the distribution. Across 7 of my own runs, at base rates from 10% to 53%, mine landed between 0.47 and 0.83. [An independent study on entirely different data](https://github.com/scienthoon/jev-ood-calibration) reported 0.66 for the same kind of yes-or-no question. The useful part is that a temperature fitted on one task lowered the error on a different task, in both directions. So it is a one-time step per question rather than a maintenance cost. One warning if you try this. My task was nearly separable, meaning almost no errors, which leaves the fitting problem badly constrained. When I added a second parameter, the fitted value drifted with whatever limited it: the bias moved from 4 to 8 to 14 to 19 as I widened the search. Someone fitting by gradient descent saw the same drift with the number of steps. Use an explicit penalty, choose it by cross-validation, and report the plain temperature as the number others can compare against. ## 5. Add guard questions Jev cannot answer "not applicable". Given a question, it must return a probability. I asked a set of documents whether the figures they contained described our event. Some of those documents had no figures at all. Asked whether absent figures matched, the model returned a low number, which looks the same as "the figures are wrong". A thematic advisory containing no statistics was flagged as carrying another event's data. The fix is to split the question and combine the answers in code. ```python questions = { "has_figures": noul("The document contains numeric figures"), "figures_match": noul("The figures describe the event in scope"), } flag = a["figures_match"] < T if a["has_figures"] >= 0.5 else False ``` I applied this in a 7-step publishing gate for a bilingual website, together with a second change that moved counting and link-matching back into ordinary code. With both applied, false positives on the held-out clean pages went from 2 of 5 to 0. I made the two changes at once, so I cannot say how much each one contributed. The general rule still holds: any question which presupposes something needs a guard question in front of it. ## 6. Make the evaluation auditable Four evaluation choices inflated Jev's apparent lead over the competing models, and every one pointed in the same direction. | Evaluation choice | Effect on the comparison | |---|---| | Scoring everything at 0.5 | Haiku about 21 points | | Output parser too strict | Gemini 14 points | | `p or 0.5` in Python | Gemini 32 points | | Scoring failed calls as 0.5 | Haiku's AUC 0.993 down to 0.965 | The third one is worth a closer look, because it is the kind of thing that passes review. ```python >>> p = 0.0 >>> p or 0.5 0.5 ``` In Python, `0.0` is falsy. Gemini answers 0.0 frequently, meaning "definitely not", and that idiom quietly rewrote its most confident correct answers into the middle of the range. Jev's accuracy lead over the closest competitor went from 31 points to 1.5. None of these choices was unusual; that is why they are worth making visible. If each correction moves the result in the same direction, pause and audit the evaluation before drawing a conclusion. ## 7. Check transformed data To build harder tests I rewrote documents. That corrupted ground truth 4 times without any error appearing. Here is the shape of it. I swapped place names so that Colombian earthquake documents said Nepal. One of my sub-questions asked whether the document reported on events in Nepal. The document now says Nepal, so the model correctly answered yes, and my label, still derived from the original folder, marked it wrong. That question scored 31.8%, which is what being marked wrong for correct answers looks like. The question was sound. My transform had inverted its label. A check for this needs 3 parts, and each part is justified by a failure the other 2 miss. 1. **Score every question on each variant separately.** One broken case passed at the whole-experiment level, at 83.4% against a 59.5% baseline, and failed only when the rewritten documents were scored on their own, at 65.1%. 2. **Compare against the majority-class rate, not 50%.** If 86% of a slice is "no", then answering "no" every time scores 86%. A question at 69.7% is above chance and still useless. 3. **Fall back to 50% where a slice contains only 1 class**, because the majority rate is then 100% by construction and would flag everything. Below the reference means the label is inverted, which is often repairable. Level with the reference means the question carries no information, which is not repairable: drop it. ## 8. Build a meaningful test Steps 1 and 2 say to measure the cheap baselines. That sounds easy. It took me 6 tries to build a task where the string check carried no information at all and a lexical baseline fell clearly behind on accuracy. The design that worked starts from a citation guard of the kind many RAG pipelines ship: it checks that a cited figure appears in the cited document. That is a string check, so I built a sample entirely from its blind spot: claim and passage pairs where the figure **is** present, drawn from different source documents, leaving only 1 question open. Does the number mean the same thing? I labelled 30 pairs blind, before seeing any model output. | | accuracy | AUC | |---|---|---| | figure present, the string check | 56.7% | 0.500 | | lexical cosine | 46.7% | 0.919 | | the model | 96.7% | 0.982 | An AUC of 0.500 means the string check carries no information at all on these pairs, which is exactly what a well-constructed sample should do to a shortcut. After 5 failures, that was the first clean one. The finding matters more than the model comparison. **13 of the 30 labelled pairs, 43%, cite a figure that is genuinely present in the passage but means something else there**, and a figure-presence check passes every one of them. Here are 4 of the 13: | The claim says | The cited passage actually says | |---|---| | 10,000 **households** affected | 10,000 **people** (2,000 households) | | supplies for 19,000 **people** | 19,000 **security personnel** mobilised | | 4,000 **people** still missing | 4,000 **bottles** of water treatment solution | | 600 **students** unaccounted for | 600 **teachers** | 11 of the 13 are unit mismatches like these. The number matches and the thing being counted does not, so a check that only looks for the number cannot see the error. I think this is the most useful finding in the whole exercise, and it is also the least finished. It needs more work before anyone should treat 43% as a rate. The test was not comprehensive: - **30 pairs, 1 reviewer.** At this size the 43% could plausibly sit anywhere from about a quarter to about 60%. A second reviewer would also catch judgement calls I would miss. - **The pairs were built, not observed.** I matched claims to passages from other documents that happened to share a number. Real summaries cite real sources, so the error rate in genuine output is probably lower than in this sample, and it could be much lower. - **Some mismatches are too easy.** No real summary would plausibly cite a count of missing people to a passage listing bottles of water treatment solution. Cases like that inflate the rate. - **1 event, 1 language, 1 kind of document.** Every pair came from reports on one flood. - **No fix was tested.** A narrow unit check did well in this test (below), but I have not tried it on real output. The next steps are clear. Sample citations from real generated summaries rather than constructed pairs, label a few hundred of them with a second reviewer, and then test whether a unit check catches the mismatches without flagging correct citations. Until then, treat this as evidence that the error class exists and is easy to miss, not as a measurement of how common it is. ### Use narrow questions Asked *does the passage support the claim*, the model scored 86.7%. Asked *does the number count the same kind of thing*, it scored 96.7% and reached AUC 0.991. 11 of the 13 defects were unit errors, so the narrow question is not only more accurate, it matches the shape of the actual problem. That finding costs nothing and holds whatever you decide about the model. ### Match questions to labels My first run asked the claim-level question. My labelling rule was narrower: the passage must assert the specific figure cited, and the rest of the claim needs its own sources. Re-running with the question matched to the rule moved accuracy from 86.7% to 96.7% and the calibration error from 0.158 to 0.111. AUC stayed at 0.982. The ordering was right the whole time, and 10 points came from the wording alone. That is section 3 restated as a measurement: fit the threshold, and check that your question means what your labels mean. ### Know what the sample can show At 30 pairs the model is **not** significantly better than TF-IDF cosine: +0.064 AUC with a 95% interval from -0.027 to +0.196. Like the 43%, that result is preliminary and says so. ## Where Jev fits Bounded judgements, at volume, where you have no labels, where latency matters, and where being wrong is recoverable. Guardrails in front of a risky action. Filtering retrieved passages before they reach an expensive model. Anything inside a loop that a 3-second call cannot fit into. It is not for counting, dates or arithmetic. It is not for anything needing an explanation you could show a regulator, because a probability is not a rationale. It is not for anything a regular expression already solves. And it is not for a stable task where you already hold labels, because a classifier you train will be cheaper and entirely predictable. ## Current limits Every figure above rests on labels that came from folder membership. Those labels are known wrong at least 5 times, and every known error was in the model's favour. An audit of 99 documents, stratified by the model's own confidence, is prepared but not yet done. It will not be independent either, because I wrote the scope definition it is judged against. Until then, the relative results are the ones to rely on, because every model was scored against the same imperfect labels: keyword baseline against Jev, classifier against Jev, and the cost and latency figures, which were measured from the billing payload rather than estimated. The absolute accuracy numbers deserve less weight than the comparisons between them. That distinction is worth carrying into any evaluation you read, including the ones vendors publish.