# Smaller Models, Better Answers

> A half-billion-parameter model can beat a frontier model on a narrow task, and the reason is not what the headline suggests. How to pick a model for your own business data, and why the evaluation matters more than the choice.

- Author: Fawad Hussain Syed, software engineer, Geneva (https://fawadhs.dev/about)
- Published: 2026-09-21
- Category: Data & Dashboards
- Canonical: https://fawadhs.dev/blog/smaller-models-for-your-own-data

---
A fine-tuned model with half a billion parameters beat GPT-5.4 and Claude Sonnet 4.6 on a language task this year.

The temptation is to read that as small models catching up. The paper says something more useful, and the authors say it themselves: the result "does not imply that SLMs are intrinsically stronger; rather, it shows that targeted task adaptation enables superior performance".

That is a statement about your task rather than about the models, and it is the most useful finding in this year's model research, because targeted task adaptation is something a small team can actually do.

## Who This Is For

- **Analytics teams** classifying, extracting or routing business data at volume
- **Founders** looking at a per-token bill that scales with success
- **Engineers** whose data cannot leave the building for contractual reasons
- **Anyone** who assumed the answer was always the largest available model

## What You Will Need

- A task that repeats: the same shape of question over the same shape of data
- A few hundred examples where you know the right answer
- A way to measure accuracy that you agree with before you start

That last one is the hard part and the one people skip. Without it you are not choosing a model, you are choosing a vibe.

## The Pattern

Model selection is not a ladder from small to large. It is a decision about how narrow the task is and where the data is allowed to go.

```mermaid
flowchart TD
  T[Task] --> N{Narrow and repeating?}
  N -->|No| F[Frontier model]
  N -->|Yes| D{Labelled examples?}
  D -->|No| C[Frontier model, curated context]
  D -->|Yes| L{Data may leave?}
  L -->|Yes| H[Hosted small model]
  L -->|No| O[Open-weight, run locally]
  H --> E[Evaluate on your data]
  O --> E
  C --> E
  F --> E
```

Every path ends at the same box. The choice is reversible; the evaluation is what makes it a decision rather than a guess.

## What the Sub-Billion Result Actually Says

The study fine-tuned Qwen2.5-0.5B on pooled general-domain data and tested relation extraction, which means pulling structured relationships out of unstructured text. It scored 0.83. GPT-5.4 scored 0.69 zero-shot, Claude Sonnet 4.6 scored 0.66.

Three things follow, and only two of them are good news.

**The task was narrow and the small model was adapted to it.** The frontier models were asked cold. That is a fair comparison of a realistic choice, adapt a small model or prompt a large one, and not a claim that the small model is better in general.

**Narrow tasks are most of what businesses actually run.** Classifying support tickets, extracting fields from invoices, routing enquiries, tagging products, normalising supplier names. None of these need a model that can also write a sonnet.

**It says nothing about your data.** A benchmark result transfers to your problem only in the sense that it tells you the approach is viable. Whether it works on your invoices is an empirical question about your invoices.

The open-weight families worth evaluating in 2026 are Qwen, GLM, DeepSeek, Llama, Mistral and Gemma. I am deliberately not ranking them: the head-to-head claims circulating are comparison-blog material without reproducible harnesses, and the ranking that matters is the one you produce on your own task anyway.

## Where a Small Model Wins, and Where It Does Not

| | Small, adapted model | Frontier model |
|---|---|---|
| Narrow repeating task | Usually wins after adaptation | Good without setup |
| Open-ended or novel work | Poor | The reason it exists |
| Cost at volume | Flat, predictable | Per token, scales with success |
| Data residency | Runs where you put it | Wherever the provider is |
| Latency | Low, local | Network-bound |
| Setup cost | Labelling and evaluation | A prompt |
| Failure mode | Confidently wrong in a familiar way | Confidently wrong in a novel way |

The setup row is the honest one. A small adapted model is cheaper to run and more expensive to start. If the task will not run ten thousand times, the arithmetic does not work and the frontier model is the right answer.

## Test-Time Compute

There is a third option people miss between "small model" and "big model": let the small model work harder on the hard cases.

Test-time compute means spending more effort at the moment of answering: sampling several attempts and taking the consensus, reasoning through intermediate steps, or checking its own answer before returning it. It buys accuracy with latency and cost.

This changes the shape of the decision. A small model with a self-check on the 5% of inputs it is unsure about can reach an accuracy a much larger model gets in one pass, at a fraction of the cost, because the expensive path only runs on the hard cases.

```python
# Spend compute where it is needed, not uniformly.
def classify(text: str) -> Label:
    votes = [small_model(text, temperature=0.3) for _ in range(3)]

    if len(set(votes)) == 1:          # unanimous: ~90% of real traffic
        return votes[0]

    # Disagreement is the signal. Escalate only these.
    return frontier_model(text, context=retrieve_similar(text, k=5))
```

Route on disagreement, not on a confidence score the model reports about itself. Self-reported confidence is one of the least reliable signals a model produces; three independent samples disagreeing is an actual observation.

## Context Engineering Beats Model Size

Most quality problems on business data are not model problems. The model answered reasonably given what it was shown, and it was shown the wrong rows.

This is the same lesson that shows up in [text-to-SQL and conversational analytics](/blog/text-to-sql-dashboards-conversational-analytics), where the accuracy gains came from improving the semantic model rather than the prompt or the model. What reaches the model matters more than which model receives it.

```python
# Most of the quality lives in this search call, not in the model below it.
def answer(question: str) -> str:
    rows = search(
        question,
        # Narrow before you rank. Filtering beats re-ranking on cost and quality.
        filters={"entity": resolve_entity(question), "active": True},
        k=8,
    )
    if not rows:
        return "No matching records. Try naming the supplier or the period."

    return model(
        system=CONVENTIONS,          # how to answer, what never to guess
        context=render_table(rows),  # the evidence, structured
        question=question,
    )
```

The early return matters more than it looks. A system that says "no matching records" is more useful than one that answers anyway from a general knowledge of how invoices usually work. The second is what you get by default if you let it happen.

## Evaluating It On Your Own Data

Everything above is a hypothesis until you measure it on your data. The harness does not need to be sophisticated; it needs to exist and be run on every change.

```python
# evals/run.py : 40 lines that outrank any benchmark you read.
import json, statistics

CASES = [json.loads(l) for l in open("evals/cases.jsonl")]

def score(candidate, expected) -> float:
    return 1.0 if candidate.strip().lower() == expected.strip().lower() else 0.0

def evaluate(model_fn, name: str) -> None:
    results = [(c, model_fn(c["input"])) for c in CASES]
    scores = [score(out, c["expected"]) for c, out in results]

    print(f"{name}: {statistics.mean(scores):.3f} over {len(scores)} cases")

    # The failures are the deliverable. The mean is just how you notice.
    for c, out in results:
        if score(out, c["expected"]) < 1.0:
            print(f"  MISS  {c['input'][:60]!r}\n"
                  f"    expected {c['expected']!r}\n"
                  f"    got      {out!r}")

for name, fn in [("small-adapted", small), ("frontier", frontier)]:
    evaluate(fn, name)
```

Build the case file from real failures as they occur, not from examples you invent. Fifty cases drawn from things that actually went wrong beat five hundred synthetic ones, because the synthetic set encodes your assumptions and the real one encodes your users'.

If you want to run any of this on your own hardware, the practical setup is covered in [running an open LLM locally](/blog/run-open-llm-locally-mac) and is not repeated here.

## Failure Modes to Avoid

- **Choosing a model before defining the measurement.** Then there is no way to know if the change helped, and every subsequent decision is taste.
- **Reading a benchmark as a prediction about your data.** It tells you an approach is viable. Nothing more.
- **Fine-tuning to fix a retrieval problem.** If the right rows never reached the model, adaptation teaches it to guess more confidently.
- **Trusting self-reported confidence.** Sample three times and route on disagreement instead.
- **Comparing open-weight models on someone else's leaderboard.** The versions move monthly and the harnesses are rarely reproducible. Run your own thirty cases.
- **Assuming local means private by itself.** Where the weights run is one part of it; what the surrounding system logs, caches and sends is the rest.
- **Adapting a model for a task that runs a hundred times a month.** The setup cost never pays back. Use the frontier model and move on.

## What to Build First

1. **Write thirty evaluation cases** from real examples where the current approach got it wrong.
2. **Score your current setup** against them. This number is the baseline everything else is measured against.
3. **Fix retrieval first.** Filters, freshness, the fields you actually pass. Re-score.
4. **Only then try a smaller model**, and score it on the same cases.
5. **Add a disagreement route** so hard cases escalate and easy ones stay cheap.
6. **Re-run the evaluation on every change**, including model version bumps you did not choose.

## Final Take

The interesting shift in model research this year is not that small models got good. It is that the question changed from "which model is best" to "which model is best for this task, measured on my data". The second question has an answer a small team can compute in an afternoon.

That reframing costs the industry its favourite argument and gives you something better: a number you produced, on data you own, about a task you actually run. The half-billion-parameter model beating the frontier system is not a story about model size. It is a reminder that a well-specified task with real examples behind it beats general capability, and that the specification and the examples are the parts nobody can sell you.