The Harness Matters More Than the Agent

Give two teams the same model and the same task and you will get results far enough apart that the model is clearly not the variable.

What differs is everything around it: what the agent could see before it started, what it was allowed to touch, and whether anything told it that it was wrong before it had built four more things on top.

That scaffolding has a name now. People call it the harness, and it is where the engineering actually is.

Who This Is For

• Engineers whose agent produces good work in one repository and nonsense in another
• Platform teams asked to make agent use safe rather than merely permitted
• Tech leads deciding what an agent is allowed to run without a human present
• Anyone who has watched a long agent run go confidently in the wrong direction

What You Will Need

• A containerised way to run your test suite, which you probably have for CI
• Write access to your repository’s conventions, wherever they live
• One task with a cheap, fast verifier, meaning a test command that runs in seconds

The Pattern

Thoughtworks splits agent controls into two families in its v34 Radar, and the split is the clearest structural idea in this whole subject. Feedforward controls constrain the work before it happens: spec-driven development, curated instructions, Agent Skills. Feedback controls detect afterwards, and mutation testing is the one they name.

A harness is both, arranged in a loop.

flowchart TD
  I[Curated instructions] --> A[Agent]
  S[Spec / acceptance criteria] --> A
  A --> X[Sandboxed execution]
  X --> V[Verifier: tests, lint, build]
  V -->|Fails| A
  V -->|Passes| H[Human review of intent]
  P[Scoped permissions] --> X

Everything left of the agent is feedforward. Everything right of it is feedback. Most teams build neither and then blame the model.

Feedforward: What the Agent Sees Before It Starts

Most agent failures are not reasoning failures. They are an agent doing something entirely reasonable given what it was shown, where what it was shown did not include the thing that mattered.

Curated instructions fix more of this than any prompt technique. Keep them in the repository, version them, and write them as constraints rather than encouragement:

# Conventions this repository enforces

## Database
- All queries go through `src/db/client.py`. Never import `psycopg` directly.
- Parameters are bound, never interpolated. No f-strings in SQL, ever.
- Migrations are additive. We do not drop columns in the same release that
  stops writing them.

## Testing
- `pytest -q` must pass before any commit.
- Never edit an assertion to make a test pass. If an assertion is wrong,
  stop and say so.

## Out of scope without asking
- Anything under `infra/`
- Dependency version changes
- Reformatting files the task did not otherwise touch

The “out of scope without asking” section earns its place repeatedly. Unbounded, an agent tidies. Tidying inflates the diff, and an inflated diff does not get reviewed. It gets approved, which is the failure mode covered in verification is the bottleneck now.

Context engineering is the same principle applied per task: deciding which files, examples and prohibitions go in front of the agent for this piece of work. More context is not better. The relevant three files beat the whole repository, because everything irrelevant competes for attention with everything that matters.

The Sandbox

An agent with your credentials is not a colleague with your credentials. A colleague has a reputation, a manager and a contract. An agent has a prompt, and a prompt can be wrong, or can be manipulated by content the agent reads while working.

Thoughtworks calls sandboxed execution and defence in depth “non-negotiable table stakes”. In practice that means a bad step should cost you a discarded container, not an incident.

# Run the agent against a copy, with no network and no credentials.
docker run --rm -it \
  --network none \
  --read-only \
  --tmpfs /tmp \
  --mount type=bind,src="$PWD",dst=/work \
  --workdir /work \
  --user "$(id -u):$(id -g)" \
  --memory 4g --pids-limit 512 \
  agent-runtime:2026-09 \
  run-task ./specs/db-migration.md

Three of those flags do most of the work. --network none means an agent cannot reach your production database because it cannot reach anything. --read-only with a tmpfs means the only writable path is the one you mounted deliberately. --user means nothing runs as root inside the container.

Where the task genuinely needs the network, for installing packages or calling an API, allow-list those hosts rather than opening it. And keep the credentials task-scoped: a read-only token for a reconnaissance task cannot be misused into a write, whatever goes wrong upstream of it.

Feedback: The Loop That Says No

Feedforward controls reduce the number of wrong turns. They do not eliminate them, so the second half of the harness decides how expensive a wrong turn is.

The property that matters is who sees the failure. An agent that runs the tests and reads the output corrects itself in seconds. An agent that reports back at the end has spent the intervening hour building on its first mistake, and you are now reviewing a plausible edifice with a rotten foundation.

# This loop is the part that matters. Everything else is configuration.
MAX_ATTEMPTS = 6

def run_task(spec: str) -> Result:
    for attempt in range(MAX_ATTEMPTS):
        patch = agent.step(spec, feedback=last_failure)
        apply(patch)

        checks = [
            run("pytest -q"),               # does it do what we said
            run("ruff check ."),            # house style
            run("semgrep --config=p/security-audit --error"),
            run("git diff --stat | tail -1"),  # is it still in scope
        ]

        failed = [c for c in checks if c.returncode != 0]
        if not failed:
            return Result.ok(patch, attempts=attempt + 1)

        # The agent reads this. That is the entire point.
        last_failure = "\n".join(c.stdout[-2000:] for c in failed)
        checkpoint(f"attempt {attempt + 1}: {len(failed)} check(s) failing")

    return Result.exhausted(last_failure)

Two details are load-bearing. The attempt cap turns an infinite loop into a task that ends and asks for help. And the checkpoint on every attempt means a wrong turn at attempt two is a state you can return to, rather than an archaeology exercise.

The verifier has to be fast. A loop gated on a twelve-minute suite will be run once and disabled. If your tests are slow, the highest-value harness work is not the harness at all. It is carving out a fast subset that gates the loop and leaving the full run to CI.

Before and After

Failure Modes to Avoid

• A prompt where a permission should be. “Do not touch production” is a request. A credential that cannot reach production is a control.
• Feeding the whole repository as context. Everything irrelevant competes with everything that matters. Curate per task.
• A verifier the agent cannot see. If failures only reach you, you have built a reporting pipeline, not a feedback loop.
• No attempt cap. An agent that cannot succeed and cannot stop will keep producing changes until someone notices the bill.
• Instructions that encourage rather than constrain. “Write clean, idiomatic code” changes nothing. “Never interpolate into SQL” changes something.
• Sandboxing the agent and then reviewing its output carelessly. The sandbox limits what a mistake can reach, not whether a mistake merges.
• Letting the agent hold the same permissions across tasks. Scope them to the brief, the same way you would scope a deploy key.

What to Build First

1. Write the conventions file, from the three corrections you find yourself repeating.
2. Add an out-of-scope section to it. This one section prevents most diff inflation.
3. Put the agent in a container with --network none for any task that does not need the network, which is most of them.
4. Wire your fastest meaningful check into a loop the agent can read the output of.
5. Cap the attempts and checkpoint each one, so exhaustion is a stopping condition rather than a surprise.
6. Scope one credential down this week. Start with whichever token currently has more access than the task needs.

Final Take

Building a harness is unglamorous, and it is most of the job. Models improve on a schedule you do not control. The scaffolding is yours, it compounds, and it survives the next model release.

There is a useful way to think about the whole arrangement: the harness is the same set of controls you would put around a capable new colleague who is fast, tireless, literal, occasionally confident about things that are not true, and who has read no institutional context whatsoever. Sandbox, scoped access, written conventions, fast feedback, and someone reading the result. None of that is distrust of the agent. It is what the work has always required, applied to a collaborator who never learns unless you build the loop that teaches them.