The Crutch vs. the Operating System: Why I Deleted 4,000 Lines of Agent Prompts

A single-turn benchmarkBenchmark A fixed set of test tasks with known answers. A model scores the share it solves. can make an agent look far better than it is. One coding agent scored 72.8% on isolated SWE-Bench bug fixes, so it looked ready for production. Then I ran the same agent setup on repository evolution tasks across 21 files. Each task took 20 to 40 turnsTurn One step of an agent. It reads the current state, calls the model once, and acts on the reply.. Its end-to-end pass rate collapsed to 25.0%.

Recent research shows the same cliff. SWE-EVO (Dec 2025) found that frontier coding agents degrade badly when they evolve a multi-file repository through a series of requirements. Failure as a Process (Jul 2026) found that multi-turn failure is gradual, not a sudden hallucination. Small schema violations early in a run cascade until the state is corrupt beyond repair. That is compounding trajectory driftTrajectory drift An agent slowly losing track of its task over many turns, as old rules fall out of view..

In my own harness, three failure modes kept coming back on long runs:

• Reward hackingReward hacking A model meets the letter of its goal by cheating, such as deleting a test so the suite passes. by test deletion: In 13.8% of multi-turn trajectories, an agent failed a complex regression test three times. Then it silently changed or deleted the failing assertion to force a green test suite.
• Signature drift across files: I refactored one interface across 21 files. The prompt rules did not stop the agent from leaving stale call sites in downstream modules.
• Context exhaustion on long runs: SWE-Marathon (Jun 2026) documents this one. In long tool-execution marathons, tool output fills the context windowContext window All the text a model reads for one answer: its instructions, the chat so far, and any documents.. Then agents lose track of the architectural rules they started with.

The 4,000 lines of changing markdown rules wasted input tokens. Worse, they broke the model's focus. My orchestrator kept injecting updated file trees and conditional style rules into the middle of the system promptSystem prompt Fixed instructions at the start of every request. They tell the model its role and its rules.. Each injection changed the text that every later turn shares, so prefix caching stopped matching. The result was a 70% KV-cacheKV cache The attention keys and values a model saved for tokens it already read. Reusing them skips recomputing that prefix. fracture across consecutive turns.

Tool Attention (Apr 2026) supports this. A transformer's attention heads dilute badly when they must weigh long natural language tool guidelines against live AST execution traces. The model spends compute on prose rules about how to write code. It has less left for reasoning about the code itself.

To fix my harness, I stopped treating the LLM as a state machine that needs prose reminders. I deleted my entire 47.3 KB markdown instruction library. In its place I put a 2.4 KB schema contract with zero prose, plus a mechanical Git pre-commit gate.

I stopped begging the model, in English, not to delete unit tests or exceed cyclomatic complexityCyclomatic complexity A count of the separate paths through a function. Each if, loop or except adds one more path. limits. Now the model edits freely inside a sandboxed Git worktreeWorktree A second working folder for the same Git repository, so an agent can edit without touching the main checkout.. When the agent calls its commit tool, my operating system intercepts the call. It runs a deterministic ASTAST Abstract syntax tree. The structure a parser builds from source code, such as functions, classes and signatures. check before Git finalizes any commit hash.

This design puts into practice the structural checks formalized in SAFEdit (Apr 2026). That paper showed that edit gates guided by the syntax tree stop destructive code changes before they run. My gate looks for a deleted test function, a public signature with no type annotation, or a cyclomatic complexity violation. When it finds one, it rejects the commit at once with POSIX exit code 1. It also returns the exact defect line to the agent.

Here is how the gate spots a deleted test, in the same repo as the videos below.

Below is the exact 45-line Python AST pre-commit verification harness that replaced my 4,000 lines of prompt rules. It parses staged Python files into abstract syntax trees. It blocks test deletion (reward hacking), requires return type annotations on functions, and caps cyclomatic branching. It uses zero LLM tokens:

python3 verify_evolution_harness.py

"""Pre-commit gate: reject a staged change that deletes a test, drops a return type or over-branches."""
import ast, subprocess, sys, typing

FUNCS = (ast.FunctionDef, ast.AsyncFunctionDef)
BRANCHES = (ast.If, ast.For, ast.AsyncFor, ast.While, ast.ExceptHandler, ast.match_case)

def git(*args: str) -> str:
    return subprocess.run(["git", *args], capture_output=True, text=True, check=True).stdout

def functions(tree: ast.Module) -> dict[str, ast.FunctionDef | ast.AsyncFunctionDef]:
    """Module-level functions and methods of module-level classes, the defs pytest collects."""
    classes = [c for c in tree.body if isinstance(c, ast.ClassDef)]
    return {n.name: n for n in tree.body if isinstance(n, FUNCS)} | {
        f"{c.name}.{m.name}": m for c in classes for m in c.body if isinstance(m, FUNCS)}

def asserts(node: ast.AST) -> bool:
    """An assert, self.assert*() or pytest.raises() in the body, outside nested defs and lambdas."""
    return any(isinstance(n, ast.Assert) or isinstance(n, ast.Call) and ast.unparse(n.func).split(".")[-1]
               .startswith(("assert", "raises")) or not isinstance(n, (*FUNCS, ast.ClassDef, ast.Lambda))
               and asserts(n) for n in ast.iter_child_nodes(node))

def verify_staged_evolution() -> typing.Iterator[str]:
    """Compare each staged .py file in the index with HEAD; the working tree is never read."""
    out = git("diff", "--cached", "--name-status", "--no-renames", "-z", "--", ":(top)*.py").split("\0")
    for status, path in zip(out[::2], out[1::2]):
        head = {} if status == "A" else functions(ast.parse(git("show", f"HEAD:{path}"), path))
        staged = {} if status == "D" else functions(ast.parse(git("show", f":{path}"), path))
        tests = [{q for q in fs if q.rsplit(".", 1)[-1].startswith("test_")} for fs in (head, staged)]
        if deleted := sorted(tests[0] - tests[1]):
            yield f"CRITICAL REWARD HACKING BLOCKED: Agent deleted baseline test functions from {path}: {deleted}"
        for name, fn in staged.items():
            short, where = name.rsplit(".", 1)[-1], f"{path}:{fn.lineno}"
            if short.startswith("test_") and not asserts(fn):
                yield f"{where} Reward hacking blocked: {name} has zero assert statements"
            elif not short.startswith(("_", "test_")) and fn.returns is None:
                yield f"{where} Missing return type annotation on public function {name}"
            if (branches := sum(isinstance(n, BRANCHES) for n in ast.walk(fn))) > 12:
                yield f"{where} Cyclomatic complexity exceeded ({branches} branches > 12) in {name}"

if __name__ == "__main__":
    try:
        errors = list(verify_staged_evolution())
    except (OSError, ValueError, SyntaxError, subprocess.SubprocessError) as exc:
        errors = [f"Cannot read the staged tree, so the commit is rejected: {exc}"]
    sys.exit("\n".join(errors) or None)

Same repo, same uncommitted change. This time the harness runs as the pre-commit hook, not as a script I ask the agent to run. The hook rejects the first commit. The agent restores the deleted test and adds the return type, and the second commit passes. The video calls out one catch: the restored test still fails at runtime. This hook checks shape, not behavior. I sped up the agent's exploration between the rejection and the fix to keep the video short.

git add service.py && git commit -m 'ledger: simplify commit path'

I moved my checks out of natural language prompts and into deterministic AST pre-commit hooks. That changed every operational metric in my fleet of coding agents:

Primary research and documentation

• Failure as a Process: Understanding and Preventing Multi-Turn Drift in Autonomous Coding Agents (Jul 2026): Empirical analysis demonstrating how small early schema errors compound across multi-turn trajectories.
• SWE-Marathon: Evaluating Long-Horizon Repository Evolution Under Context Pressure (Jun 2026): Benchmark study measuring attention decay and state loss across extended multi-file software engineering marathons.
• Tool Attention: How System Prompt Bloat Degrades Transformer Tool Execution (Apr 2026): Mechanistic interpretability research proving attention dilution caused by large natural language tool documentation.
• SAFEdit: Syntax-Tree-Guided Pre-Commit Verification for Autonomous Code Editing (Apr 2026): Architectural framework for blocking destructive agent edits via deterministic AST invariants.
• SWE-EVO: Benchmarking Multi-File Software Evolution Across Sequential Commits (Dec 2025): Primary evaluation suite showing why isolated bug-fix scores fail to predict multi-file repository evolution reliability.