How to Benchmark a Decision Model: Lessons From Testing Jev

Field notes, 21 September 2026. This is the procedure, and it is the part that will still be true when the next model is released. Every figure comes from a logged run with the dataset hashed and the model version pinned. I’m not publishing the harness or the data. The findings are in Jev tested on 415 documents, and the concepts in how Jev works and how to test it.

Model evaluations measure the test design as much as the model. In my testing, six design choices changed the result in a day and a half, and each moved it in the same direction.

I spent a day and a half and about $3 benchmarking a decision model against a corpus of public disaster reports. The verdict matters to a small number of people and will expire with the next release. The workflow is more durable: it shows which measurements are useful, which inputs Jev needs, and which checks should stay in code.

The workflow

Before you spend anything on a model:
  1. grep              is the signal a literal string?
  2. TF-IDF + logreg   is the signal a bag of words?
  3. embeddings        is the signal simple semantic similarity?
  Only what survives all 3 is worth a model.

Before you believe any number you produce:
  4. fit the threshold, on train, scored on held-out data
  5. count the imputed values
  6. re-score every question on every variant of the data
  7. check whether evaluation choices all point the same way

Everything below is why each line is there.

1. Test for literal signals

My first test looked careful. 415 real documents, 3 collections, labels taken from folder membership rather than from my judgement. The model scored 96.6%.

Then I wrote "nepal" in document.lower() and it scored 97.3%.

The task looked semantic and was not. Out-of-scope documents were about other countries, so detecting a country name was enough. I had measured country detection.

It happened twice more. Rewriting documents to remove place names produced a task where grep scores 18.1%, which was better. But a third task, 110 documents built by date substitution to separate 2 floods on the same river 13 months apart, was beaten again by counting which year string appeared more often, at 97.3% against 94.5%.

Across three tasks, two were won or tied by a few lines of text matching. You cannot tell by reading a problem whether it needs semantics.

2. Test a local classifier

The first baseline is easy to remember. The second is the one that most often changes the deployment decision.

TF-IDF plus logistic regression, about 40 lines of standard library, 5-fold cross-validation grouped by source document so that no variant could appear in both training and test. On the task where grep manages 18.1%:

Both fitted figures use a threshold chosen on the same predictions they are scored on, so they are upper bounds. At the default threshold the classifier is ahead; with a fitted threshold the model is ahead; the gap is under half a point either way. The classifier needed 415 labels and the model needed none, which is a real difference, but it is a difference in deployment rather than in capability.

The third baseline I could not run. The gateway I was using hosts no embedding models, and the retrieval system I had access to exposes no embedding endpoint. So my honest claim is that no cheap lexical method solves these tasks. I cannot claim that no cheap semantic one does, and that distinction belongs in the limits section rather than being quietly dropped.

3. Fit the threshold

I ran a comparison across 3 models at the conventional 0.5 cut-off. One of them looked hopeless. Scored on the same held-out half of the documents, 201 of them, with the threshold fitted on the other half:

                  at 0.5    with a fitted cut-off
Claude Haiku 4.5   76.1%          97.5%

That is about 21 points from changing 1 number. The best cut-offs across the 3 models were 0.09, 0.15 and 0.05, and none of them was near 0.5.

Fit it on a training split and score on held-out data. If you publish a comparison at a default threshold, you are publishing your own configuration rather than a property of the models.

Report AUC alongside it. AUC ignores the cut-off entirely and measures only whether the ordering is right, so it survives the criticism that sinks an accuracy table. In my work the ordering was consistently good while the raw probabilities needed rescaling.

One detail on reporting AUC. Print it to 5 decimal places. At 3 places a score of 0.99992 rounds to 1.000 and becomes indistinguishable from perfect separation, which is how the phrase “perfect separation” spread through my own notes. When I counted the pairs, a handful of the 42,900 were ranked the wrong way or tied.

4. Track missing outputs

My harness was quietly losing data in 2 of the arms.

A brittle output parser rejected valid answers and defaulted them to 0.5, costing one model 14 points. Rate limiting swallowed 14 of 415 calls in another arm, and I scored those as 0.5 too, which dropped its AUC from 0.993 to 0.965. A later run lost 74 of 246 calls the same way, and I caught it only because I checked the null count before scoring.

None of these raised an exception. Each produced a plausible number.

So count the imputed values and report them. If more than a few percent of your calls did not return an answer, you are measuring your harness rather than the model.

Be careful with this idiom:

p = response.get("probability") or 0.5

In Python, 0.0 is falsy. A model answering 0.0, meaning “definitely not”, has its most confident correct answers rewritten into the middle of the range. That single line cost one arm 32 points.

5. Validate transformed data

To remove a shortcut I rewrote documents, swapping place names so that out-of-scope material carried in-scope names. That was reasonable and it worked, since the keyword baseline collapsed.

It also inverted my ground truth 4 times without any error appearing.

One sub-question asked whether a document reported on events in a particular country. After the swap the document said that country throughout, so the model answered correctly and my label, still derived from the original folder, marked it wrong. That question scored 31.8%.

A check for this needs 3 parts, and each part is justified by a failure the other 2 miss.

• Score every question on each variant separately. One broken case passed at the whole-experiment level, at 83.4% against a 59.5% baseline, and failed only when the rewritten subset was scored alone, at 65.1%.
• Compare against the majority-class rate, not 50%. A useless question still beats chance on an imbalanced slice by always answering the majority. Both broken arms sat at 69.7% and 65.1% against an 86.2% baseline, which is above chance and below useful.
• Fall back to 50% where a slice has only 1 class, since the majority rate is then 100% by construction and would flag everything.

Below the reference means the label is inverted, which is often repairable. Level with the reference means the question carries no information, which is not: drop it.

The whole check runs on stored answers with no API calls. Mine catches 4 of 4 known failures with no false positives, and it would have caught all of them at the moment they were created rather than 3 experiments later.

6. Check evaluation bias

Over the course of the evaluation I found four evaluation choices that each inflated one model’s apparent lead: the default threshold, the brittle parser, the falsy zero, and imputed rate-limit failures. The lead went from 31 points to 1.5.

Each choice was ordinary, but all four moved the result in the same direction. That pattern is a signal to audit the evaluation before trusting the headline.

The practical version is to audit hardest when a result pleases you, and to publish the count of corrections. The tally tells readers how much of the result comes from the model and how much comes from the test.

7. Build a meaningful test

Five of my task designs leaked. Country names, year strings, hazard vocabulary, and twice a label rule that became the feature solving the task.

That last one is the subtlest. Trying to avoid hand labelling, I generated entailment labels with a rule: a passage supports a claim if the claim was extracted from it, and does not if the passage lacks the claim’s figures. Both halves of that rule are trivially detectable, so the first tier scored 100%. I had built the answer into the label definition.

The design that finally worked started from a citation guard of the kind many RAG pipelines have: it checks that a cited figure appears in the cited document. So I built the sample entirely from that check’s blind spot: pairs where the figure is present, leaving only the question of whether it means the same thing.

The figure-presence check scores AUC 0.500 on that sample, exactly at the base rate, which is what a properly constructed task does to a shortcut.

If you cannot find a task where the cheap baselines fail, that is a finding. It means the problem you were about to spend money on is a string-matching problem, and you should say so rather than keep hunting for a framing that flatters the model.

8. Report useful comparisons

Every accuracy figure I produced rests on labels derived from folder membership. Those labels are known wrong at least 5 times, and every known error favoured the model.

So the numbers I would stand behind are the comparisons, because every arm was scored against the same imperfect labels: keyword against model, classifier against model, and cost and latency measured from the billing payload rather than estimated. The absolute accuracy figures deserve less weight than the gaps between them.

That distinction is worth carrying into any evaluation you read. When a vendor publishes a benchmark, the first question is not how high the number is. It is what the labels are, who made them, and whether a baseline built in an afternoon would have scored the same.

Practical checklist

1. Write the grep version. Measure it.
2. Write the TF-IDF version. Measure it.
3. Write the embedding version. Measure it.
4. Fit thresholds on train, score on held-out data, and report AUC alongside accuracy, to 5 decimal places.
5. Count every imputed or defaulted value and publish the count.
6. Re-score every question on every variant after any transformation.
7. Tally your corrections and check whether they point one way.
8. Report comparisons more confidently than absolutes, and name who made the labels.

None of it requires a budget. Most of it runs on results you already have.