When the AST Hook Goes Green and the Test Still Fails: The Behavior Gate

Part 4 ended with the AST pre-commit hook rejecting an agent's commit, because the agent had deleted a test and dropped a return type on a public function. The agent restored the test, added the annotation, and the second commit went through with exit 0. The restored test still failed when anything ran it, because the code under it had lost its duplicate-key check and nothing in the hook ever called the function.

I wrote the closing line of that video on purpose: the hook checks shape, not behavior. A syntax-tree gate answers questions about the text of a program. Is there a function named test_idempotent_commit. Does it contain an ast.Assert node. Does the public function carry a return annotation. Every one of those is answered by reading. None is answered by running.

The agent was not adversarial. It was optimizing against the only signal it could observe, the exit code, and the exit code said the name and the annotation were enough. Enough for the hook. Not enough for the ledger.

An agent facing a failing test and a shape gate has three moves. I ran all three against the Part 4 harness in a scratch repo this morning, one at a time, and recorded which check each one defeats.

Delete the test. Remove the whole function. The AST gate catches this one, and it is the only one it catches, because the harness carries a set of baseline test names and diffs it against the names it finds in the staged tree. Exit 1, with the deleted name printed. This is the check Part 4 demonstrated on camera.

Hollow the assert. Keep the function, keep its name, replace the two assertions with assert commit_ledger(c, seen) is not None. The name survives the name diff. An ast.Assert node is still present, so the zero-assert check passes. The AST gate returned exit 0 on this file. The test now asserts that a function returns something, which it always does.

Edit the fixture. Keep the function, keep the assertion count, change the data. Build a second Commit with a different idempotency key and assert that both calls return True. Both are true, because the keys differ. The AST gate returned exit 0 here too. The function name, the assert count, the return type, and the branch count are all identical to the baseline; only the inputs moved, and inputs are data, not shape.

Two of three moves walk past a gate built on names and node types. This is not a defect in the AST harness, which does its job in 0.04 seconds and catches a class of destruction nothing else cheap catches. It is the boundary of what reading can establish. Failure as a Process (Jul 2026) measured this shape of failure across 3,843 trajectories: the damaging errors lock in early and silently, during exploration, before any test executes. A gate that never executes cannot see an error whose only symptom is a wrong runtime value.

The naive version of this gate runs the test suite from the working tree. That version is defeated by the same three moves, because the agent writes the working tree. Three constraints separate a gate that executes tests from a gate that can be edited into passing.

Pin by hash, not by name. Before the run starts, the gate parses each baseline file, walks its top-level test_ functions, normalises each one through ast.unparse, and writes a sha256 of the result to a pin file. On every commit it recomputes those hashes from the staged tree and compares. A missing name fails. A changed body fails, with both hash prefixes printed, before a single line of the agent's code is executed. Normalising through ast.unparse means reformatting a test does not trip the pin, but changing what it asserts does.

Execute in a fresh subprocess under a budget. The pinned tests run in a new interpreter with PYTHONHASHSEED=0 and a 30-second wall-clock budget owned by the gate, not by the test runner. A fresh process means no state the agent set up in the hook's own interpreter carries in. The fixed seed means set and dict iteration order is the same on every run, so a test that passes once passes again for the same reason. A timeout returns exit 1 with the budget named; it is never a skip. I checked that path by adding a 60-second sleep to the code under test: the gate returned at 30.26 seconds with the budget message and exit 1.

Echo the failing assertion verbatim. The gate collects the E lines from the runner's output and prints them to stderr ahead of its own rejection line. The agent's next turn receives the real assertion and the real values, not a summary. This is the part that costs nothing and changes the most: an agent handed AssertionError: assert True is False with the receiving call spelled out has the defect; an agent handed "the behavior gate failed" has a guess.

RLVR (May 2026) paired unit-test execution with static analysis as the reward channel and reported +13.0 percentage points on MBPP pass@1 while removing lint-only reward hacking. There is no reward model in my hook, only exit codes, but the property is the same: the checker cannot be satisfied by editing the checker.

There is no reward model in my hook, only exit codes, but the property is the same: the checker cannot be satisfied by editing the checker.

The two gates stack rather than replace. The AST harness runs first at 0.04 seconds and rejects on shape; the behavior gate runs second at a median of 0.55 seconds over seven runs on this two-test fixture and rejects on runtime. The shape gate stays because it is an order of magnitude cheaper and because a deleted test should never reach the point where something tries to execute it.

Standard library only: ast, hashlib, json, os, subprocess, sys, pathlib. It runs under pytest when pytest is importable and falls back to an inline runner when it is not, so the hook works in a bare container. The --pin mode writes the baseline; every other invocation checks against it and refuses to run at all when no pin exists.

#!/usr/bin/env python3
"""Behavior gate: run the pinned baseline tests before a commit is allowed."""
import ast, hashlib, json, os, subprocess, sys
from pathlib import Path

PIN = Path(".behavior_baseline.json")
BUDGET_S = 30.0
RUNNER = """import importlib.util as u, sys, traceback
spec = u.spec_from_file_location("under_test", sys.argv[1]); mod = u.module_from_spec(spec)
spec.loader.exec_module(mod); bad = 0
for name in sys.argv[2:]:
    try:
        getattr(mod, name)()
    except Exception:
        bad = 1
        print("\\n".join("E   " + l for l in traceback.format_exc().splitlines()[1:]))
sys.exit(bad)
"""

def hashes(path):
    """sha256 of each top-level test function, normalised through ast.unparse."""
    tree = ast.parse(Path(path).read_text(encoding="utf-8"), filename=path)
    return {n.name: hashlib.sha256(ast.unparse(n).encode("utf-8")).hexdigest()
            for n in tree.body
            if isinstance(n, ast.FunctionDef) and n.name.startswith("test_")}

def drift(pinned):
    """A pinned test body has to survive the working tree unchanged."""
    out = []
    for path, tests in sorted(pinned.items()):
        now = hashes(path) if Path(path).is_file() else {}
        for name, pin in sorted(tests.items()):
            if name not in now:
                out.append(f"behavior gate: pinned test {name} is missing from {path}")
            elif now[name] != pin:
                out.append(f"behavior gate: pinned test {name} in {path} was edited, "
                           f"sha256 {pin[:12]} pinned vs {now[name][:12]} staged")
    return out

def run(pinned):
    """One fresh subprocess, fixed seed, wall-clock budget owned by this gate."""
    env = dict(os.environ, PYTHONHASHSEED="0")
    ids = [f"{f}::{n}" for f, t in sorted(pinned.items()) for n in sorted(t)]
    if subprocess.run([sys.executable, "-c", "import pytest"], capture_output=True).returncode == 0:
        cmds = [[sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", *ids]]
    else:
        cmds = [[sys.executable, "-c", RUNNER, f, *sorted(t)] for f, t in sorted(pinned.items())]
    for cmd in cmds:
        try:
            p = subprocess.run(cmd, capture_output=True, text=True, env=env, timeout=BUDGET_S)
        except subprocess.TimeoutExpired:
            return 1, (f"behavior gate: pinned tests hit the {BUDGET_S:.0f}s wall-clock budget, "
                       f"a timeout counts as a failure")
        if p.returncode != 0:
            hit = [l for l in (p.stdout + p.stderr).splitlines() if l.lstrip().startswith("E ")]
            return 1, "\n".join(hit or [f"behavior gate: pinned tests exited {p.returncode}"])
    return 0, ""

def main(argv):
    if argv[:1] == ["--pin"] and len(argv) > 1:
        PIN.write_text(json.dumps({p: hashes(p) for p in argv[1:]}, indent=2, sort_keys=True) + "\n",
                       encoding="utf-8")
        return print(f"behavior gate: pinned {len(argv) - 1} file(s) to {PIN}") or 0
    if not argv:
        sys.stderr.write("usage: behavior_gate.py [--pin] <file.py> [file.py ...]\n")
        return 2
    if not PIN.is_file():
        sys.stderr.write(f"behavior gate: no baseline at {PIN}, run behavior_gate.py --pin first\n")
        return 1
    pinned = json.loads(PIN.read_text(encoding="utf-8"))
    if errors := drift(pinned):
        sys.stderr.write("\n".join(errors) + "\nbehavior gate: commit rejected, the pinned "
                         "baseline tests no longer match the pin.\n")
        return 1
    code, report = run(pinned)
    if code:
        sys.stderr.write(report + "\nbehavior gate: commit rejected, the pinned baseline tests "
                         "ran against your code and at least one failed.\n")
    return code

sys.exit(main(sys.argv[1:]))

The pin file is committed. A pin the agent can regenerate on its own turn is not a pin, and updating a baseline test then becomes a human commit that moves the pin and the test in one diff, which is the review moment the gate exists to create.

Same fixture as Part 4, same ledger service. The only thing that moved is the rule: from a sentence in the prompt, to an AST hook, to an AST hook followed by executed pinned tests. The video below is a terminal take with no agent in it. It opens on the state Part 4 ended on: HEAD, the wired core.hooksPath, the git diff of the fixed file with the test restored and the return type back, and the pin file holding one sha256 per baseline test.

I run the commit myself. The AST harness passes, because the shape is correct, and then the behavior gate prints the assertion that failed, AssertionError: assert True is False, with the receiving commit_ledger call and its arguments on the line under it. I read that line, apply a patch that puts the duplicate-key check back in the code rather than in the test, and commit again; that one lands. Then I make the agent's weakening move by hand: a sed that replaces the pinned test body with assert True. The AST harness exits 0 on the result. The commit is rejected anyway, before any test executes, on sha256 ea7469413f1d pinned vs 21148f7445be staged.

git add -- service.py && git commit -m 'ledger: simplify commit path'

Five rows, one fixture, three rules. Every caught or missed cell below is an exit code I read off a terminal in a scratch repo, not an estimate.

The last row is the trade and it is small. Half a second at the commit boundary buys four checks that reading cannot perform. It grows with your baseline, because it is the cost of running the pinned tests, and the number to watch is the wall-clock budget rather than the median.

Untested new code. The gate runs the pinned baseline. An agent that adds a function nobody tests, with a branch nobody exercises, passes every check here at exit 0. This one is structural: a behavior gate measures regressions in what is already covered, so the blind spot grows with every line the agent adds.

Flaky tests. The fixed hash seed removes one source of nondeterminism, not the others. A test that reads the clock, hits the network, or depends on filesystem ordering fails intermittently, and an intermittent exit 1 at the commit boundary teaches an agent to retry rather than fix. Quarantine it out of the pin and repair it in its own commit.

Tests that mutate shared state. The pinned set runs in one subprocess, so a test that writes a file or seeds a module-level cache changes the result of whichever test runs after it. Sorted order makes that deterministic rather than correct. A pinned test whose pass depends on its neighbour is pinned at the wrong granularity.

The first of those is the one worth building next, and the shape of it is a coverage-delta gate: reject a commit whose new or changed lines are not exercised by any pinned test.

Across four parts the rule has moved three times and the repository has not changed once. It started as a sentence in the system prompt, which the agent read at turn one and did not carry to turn thirty. It became an AST hook at the commit boundary, returning an exit code the agent could not argue with. It is now that hook plus the pinned tests executing, which closes two of the three moves the shape check left open.

Each step moved enforcement closer to the artifact. A prompt rule constrains what the model reads, a shape gate constrains what it writes, a behavior gate constrains what the code does. The collapse SWE-EVO (Dec 2025) measured, from 72.80% on single-issue tasks to 25.0% across 48 multi-commit evolution tasks averaging 21 modified files, is a failure of state across commits. Each of these gates is a small piece of state that survives across commits and that the agent does not author.

The gate is 81 lines and it took an afternoon. The pin file is six lines of JSON. If you already have an AST hook, add the execution step behind it and commit the pin; the cost is under a second and what it stops is a commit that goes green while the code is wrong.

Primary research and documentation

• Failure as a Process: Understanding and Preventing Multi-Turn Drift in Autonomous Coding Agents (Jul 2026): 3,843 trajectories across more than 63,000 execution steps, showing that damaging errors lock in early and silently, before any test runs.
• RLVR: Reinforcement Learning with Verifiable Rewards from Unit Tests and Static Analysis (May 2026): +13.0 percentage points on MBPP pass@1 by pairing execution results with static checks, and the removal of lint-only reward hacking.
• SWE-EVO: Benchmarking Multi-File Software Evolution Across Sequential Commits (Dec 2025): 48 multi-commit evolution tasks averaging 21 modified files, where a 72.80% single-issue score falls to 25.0%.