When the AST Hook Goes Green and the Test Still Fails: The Behavior Gate

Part 4 ended with the AST hook rejecting an agent's commit. The agent had deleted a test and dropped a return type on a public function. It restored the test and added the annotation, and the second commit went through with exit 0. The restored test still failed when anything ran it. The code under it had lost its duplicate-key check, and nothing in the hook ever called the function.

I wrote the closing line of that video on purpose: the hook checks shape, not behavior. A syntax-tree gate answers questions about the text of a program. Is there a function named test_idempotent_commit? Does it contain an ast.Assert node? Does the public function carry a return annotation? Reading answers every one of those. Running answers none of them.

The agent was not adversarial. It optimized against the only signal it could see, the exit code. The exit code said the name and the annotation were enough. They were enough for the hook. They were not enough for the ledger.

Say an agent faces a failing test and a shape gate. It has three moves. This morning I ran all three against the Part 4 harnessHarness The code around an agent that hands it the task, runs its tools and checks its work. in a scratch repo, one at a time. I recorded which check each one defeats.

Delete the test. Remove the whole function. The AST gate catches this one, and only this one. The harness compares the test names at HEADHEAD Git's name for the latest commit on the current branch. with the names in the staged tree. It exits 1 and prints the deleted name. This is the check Part 4 showed on camera.

Hollow the assert. Keep the function and its name. Replace its two assertionsAssertion The line in a test that states what must be true. If it is false, the test fails. with assert commit_ledger(c, seen) is not None. The name survives the name check. An ast.Assert node is still there, so the zero-assert check passes. The AST gate returned exit 0 on this file. The test now asserts that a function returns something, and it always does.

Edit the fixtureFixture A fixed, scripted input for a test or demo. It plays out the same way every time.. Keep the function and the assertion count, and change the data. Build a second Commit with a different idempotency keyIdempotency key A unique ID sent with a request, so a retry of the same request takes effect only once., and assert that both calls return True. Both are true, because the keys differ. The AST gate returned exit 0 here too. The function name, the assert count, the return type and the branch count all match the baseline. Only the inputs moved, and inputs are data, not shape.

So two of the three moves walk past a gate built on names and node types. The AST harness has no defect here. It does its job in 0.04 seconds, and it catches a kind of damage nothing else this cheap catches. This is the limit of what reading can establish. Failure as a Process (Jul 2026) measured this shape of failure across 3,843 trajectoriesTrajectory One full agent run on one task, from its first turn to its last.. The damaging errors lock in early and silently, during exploration, before any test runs. A gate that never runs code cannot see an error whose only symptom is a wrong value at runtime.

The naive version of this gate runs the test suite from the working tree. The same three moves defeat it, because the agent writes the working tree. Three choices separate a gate that runs tests from a gate the agent can edit into passing.

Pin by hash, not by name. Before the run starts, the gate parses each baseline file and finds its top-level test_ functions. It normalises each one through ast.unparse, together with any other top-level line that names it. Then it writes a sha256sha256 A hash: a 64 character fingerprint of some text. Change one character and the whole fingerprint changes. of the result to a pin file. On every commit, it reads the pin from HEAD, recomputes those hashes from the staged tree, and compares. A missing name fails. A changed body fails, with both hash prefixes printed, before a single line of the agent's code runs. Because of ast.unparse, reformatting a test does not trip the pin, but changing what it asserts does.

Execute in a fresh subprocess under a budget. The pinned tests run in a new interpreterInterpreter The program that reads and runs Python code. A new one starts with no state from the last., on a copy of the staged tree in a temporary directory. The gate, not the test runner, owns a 30-second wall-clock budget, and it sets PYTHONHASHSEED=0. The repo's conftest.py, plugins and pytestpytest A common Python test runner. It exits 1 when a test fails and 0 when all tests pass. settings stay off. A pinned test counts only when the report shows it passed, so a skip is a failure. A fresh process means no state the agent set up in the hook's own interpreter carries over. The fixed hash seedHash seed A number Python uses to scramble hashes. Fixing it makes set and dict order the same on every run. keeps set and dict order the same on every run, so a test that passes once passes again for the same reason. A timeout returns exit 1 and names the budget; it is never a skip. I checked that path. I added a 60-second sleep to the code under test, and the gate returned at 30.26 seconds with the budget message and exit 1.

Echo the failing assertion word for word. The gate collects the E lines from the runner's output. It prints them to stderrstderr Standard error. The output stream a program uses for error messages., ahead of its own rejection line. So the agent's next turn gets the real assertion and the real values, not a summary. This part costs nothing and changes the most. Hand an agent AssertionError: assert True is False, with the failing call spelled out, and it has the defect. Hand it "the behavior gate failed", and it has a guess.

RLVR (May 2026) paired unit-testUnit test A small automatic test that runs one function with a known input and checks the output. runs with static analysis as the reward channel. It reported gains of up to 13 percentage points on MBPP pass@1, and it removed lint-only reward hackingReward hacking A model meets the letter of its goal by cheating, such as deleting a test so the suite passes.. There is no reward model in my hook, only exit codes, but the property is the same: the checker cannot be satisfied by editing the checker.

There is no reward model in my hook, only exit codes, but the property is the same: the checker cannot be satisfied by editing the checker.

The two gates stack; one does not replace the other. The AST harness runs first, in 0.04 seconds, and rejects on shape. The behavior gate runs second and rejects on runtime. Over seven runs on this two-test fixture, its median was 0.55 seconds. The shape gate stays for two reasons. It is an order of magnitude cheaper. And a deleted test should never reach the point where something tries to run it.

It uses the standard library only: ast, hashlib, importlib, json, os, subprocess, sys, tempfile, xml.etree, pathlib. It runs under pytest when pytest is importable. When it is not, it falls back to an inline runner, so the hook works in a bare container. The --pin mode writes the baseline. Every other call checks against it, and refuses to run at all when no pin exists.

#!/usr/bin/env python3
"""Behavior gate: run the pinned baseline tests on the staged tree before a commit is allowed."""
import ast, hashlib, importlib.util, json, os, subprocess, sys, tempfile
import xml.etree.ElementTree as ET
from pathlib import Path

PIN = ".behavior_baseline.json"
BUDGET_S = 30.0
OK = "behavior gate runner: every pinned test passed"
RUNNER = f"""import importlib.util as u, sys, traceback
spec = u.spec_from_file_location("under_test", sys.argv[1]); mod = u.module_from_spec(spec)
spec.loader.exec_module(mod); bad = 0
for name in sys.argv[2:]:
    try:
        getattr(mod, name)()
    except BaseException:
        bad = 1
        print("\\n".join("E   " + l for l in traceback.format_exc().splitlines()[1:]))
if bad:
    sys.exit(1)
print({OK!r})
"""

def git(*args):
    return subprocess.run(["git", *args], capture_output=True, text=True, check=True,
                          timeout=BUDGET_S).stdout

def words(node):
    """Every identifier and string constant inside one top-level statement."""
    for n in ast.walk(node):
        yield from (getattr(n, field, None) for field in ("id", "name", "asname", "attr"))
        if isinstance(n, ast.Constant):
            yield n.value

def hashes(source, path):
    """sha256 per top-level test function: its def plus any other top-level statement
    that names it, such as `test_x = lambda: None`, normalised through ast.unparse."""
    body = ast.parse(source, filename=path).body
    tests = {n.name for n in body if isinstance(n, ast.FunctionDef) and n.name.startswith("test_")}
    return {t: hashlib.sha256("\n".join(ast.unparse(n) for n in body if t in words(n))
                              .encode("utf-8")).hexdigest() for t in sorted(tests)}

def staged_tree(root):
    """Check the index out under root and return the pin as committed at HEAD."""
    pin = git("show", f"HEAD:{PIN}")
    git("checkout-index", "--all", f"--prefix={root}/")
    if not (root / PIN).is_file() or (root / PIN).read_text(encoding="utf-8") != pin:
        raise ValueError(f"the staged {PIN} differs from HEAD, and moving the pin is a human commit")
    return json.loads(pin)

def drift(pinned, root):
    """A pinned test has to survive the staged tree unchanged."""
    out = []
    for path, tests in sorted(pinned.items()):
        f = root / path
        now = hashes(f.read_text(encoding="utf-8"), path) if f.is_file() else {}
        for name, pin in sorted(tests.items()):
            if name not in now:
                out.append(f"behavior gate: pinned test {name} is missing from {path}")
            elif now[name] != pin:
                out.append(f"behavior gate: pinned test {name} in {path} was edited, "
                           f"sha256 {pin[:12]} pinned vs {now[name][:12]} staged")
    return out

def passed(report):
    """(classname, name) of every JUnit testcase with no failure, error or skip."""
    try:
        cases = ET.parse(report).getroot().iter("testcase")
    except (OSError, ET.ParseError):
        return set()
    return {(c.get("classname"), c.get("name")) for c in cases
            if not any(child.tag in ("failure", "error", "skipped") for child in c)}

def run(pinned, root):
    """One fresh subprocess on the staged tree, fixed seed, wall-clock budget owned by this gate."""
    env = dict(os.environ, PYTHONHASHSEED="0", PYTEST_DISABLE_PLUGIN_AUTOLOAD="1")
    report = root.parent / "report.xml"
    want = {(f.removesuffix(".py").replace("/", "."), n) for f, t in pinned.items() for n in t}
    ids = [f"{f}::{n}" for f, t in sorted(pinned.items()) for n in sorted(t)]
    if importlib.util.find_spec("pytest"):  # no conftest, plugins or ini from the repo
        cmds = [[sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", "--noconftest",
                 "-c", os.devnull, f"--rootdir={root}", f"--junitxml={report}", *ids]]
    else:
        cmds = [[sys.executable, "-c", RUNNER, f, *sorted(t)] for f, t in sorted(pinned.items())]
    for cmd in cmds:
        try:
            p = subprocess.run(cmd, cwd=root, capture_output=True, text=True, env=env, timeout=BUDGET_S)
        except subprocess.TimeoutExpired:
            return 1, (f"behavior gate: pinned tests hit the {BUDGET_S:.0f}s wall-clock budget, "
                       f"a timeout counts as a failure")
        ok = OK in p.stdout if cmd[2] == RUNNER else passed(report) >= want
        if p.returncode != 0 or not ok:
            hit = [l for l in (p.stdout + p.stderr).splitlines() if l.lstrip().startswith("E ")]
            return 1, "\n".join(hit or [f"behavior gate: pinned tests exited {p.returncode} "
                                        f"without every pinned test passing"])
    return 0, ""

def main(argv):
    if argv[:1] == ["--pin"] and len(argv) > 1:
        pins = {p: hashes(Path(p).read_text(encoding="utf-8"), p) for p in argv[1:]}
        Path(PIN).write_text(json.dumps(pins, indent=2, sort_keys=True) + "\n", encoding="utf-8")
        return print(f"behavior gate: pinned {len(argv) - 1} file(s) to {PIN}") or 0
    if not argv:
        sys.stderr.write("usage: behavior_gate.py [--pin] <file.py> [file.py ...]\n")
        return 2
    with tempfile.TemporaryDirectory() as tmp:
        root = Path(tmp).resolve() / "tree"
        try:
            pinned = staged_tree(root)
            errors = drift(pinned, root)
        except (OSError, ValueError, SyntaxError, subprocess.SubprocessError) as exc:
            errors = [f"behavior gate: cannot check the staged tree against the HEAD pin: {exc}"]
        if errors:
            sys.stderr.write("\n".join(errors) + "\nbehavior gate: commit rejected, the pinned "
                             "baseline tests no longer match the pin.\n")
            return 1
        code, report = run(pinned, root)
    if code:
        sys.stderr.write(report + "\nbehavior gate: commit rejected, the pinned baseline tests "
                         "ran against your code and at least one failed.\n")
    return code

sys.exit(main(sys.argv[1:]))

The pin file is committed. The gate reads it from HEAD and rejects a commit that stages a different one. If the agent can make a new pin on its own turn, the pin protects nothing. So updating a baseline test becomes a human commit. A person makes it with --no-verify, and it moves the pin and the test in one diff. That commit is the review moment the gate exists to create.

This is the same fixture as Part 4, with the same ledger service. Only the rule moved. It started as a sentence in the prompt, became an AST hook, and is now an AST hook followed by pinned tests that run. The video below is one agent turn in the Antigravity app. The agent works through a steps.md list I wrote, and runs each command as written. It opens on the state Part 4 ended on: HEAD, the wired core.hooksPath, and the git diff of the fixed file, with the test restored and the return type back. The pin file holds one sha256 per baseline test.

The agent runs the commit. The AST harness passes, because the shape is correct. Then the behavior gate prints the assertion that failed, AssertionError: assert True is False. The line under it shows the failing commit_ledger call and its arguments. The next steps apply a patch that puts the duplicate-key check back in the code, not in the test, and commit again. That commit lands. Then the list makes the agent's weakening move. A small helper script replaces the pinned test body with assert True. The AST harness exits 0 on the result. The gate rejects the commit anyway, before any test runs, on sha256 ea7469413f1d pinned vs 21148f7445be staged.

git add -- service.py && git commit -m 'ledger: simplify commit path'

Five rows, one fixture, three rules. Every caught or missed cell below is an exit code I read off a terminal in a scratch repo. None is an estimate.

The last row is the trade, and it is small. Half a second at the commit boundary buys four checks that reading cannot do. The cost grows with the baseline, because it is the time to run the pinned tests. So the number to watch is the wall-clock budget, not the median.

Untested new code. The gate runs the pinned baseline. Say an agent adds a function nobody tests, with a branch nobody runs. It passes every check here at exit 0. This blind spot is structural. A behavior gate measures regressionsRegression test A test that checks an old feature still works after a change. When it fails, something broke. in what tests already cover, so the blind spot grows with every line the agent adds.

Flaky testsFlaky test A test that sometimes passes and sometimes fails on the same code.. The fixed hash seed removes one source of nondeterminismNondeterminism When the same code, run twice on the same input, can give different results., not the others. A test that reads the clock, calls the network or depends on file order fails now and then. An exit 1 that comes and goes at the commit boundary teaches an agent to retry, not to fix. Take that test out of the pin and repair it in its own commit.

Tests that change shared state. The pinned set runs in one subprocess. So a test that writes a file or fills a module-level cache changes the result of whichever test runs after it. Sorted order makes that result repeatable, not correct. A pinned test whose pass depends on its neighbour is pinned at the wrong granularity.

The first of those is the one worth building next. Its shape is a coverage-delta gate: reject a commit whose new or changed lines no pinned test runs.

Across four parts, the rule has moved three times, and the repository has not changed once. It started as a sentence in the system promptSystem prompt Fixed instructions at the start of every request. They tell the model its role and its rules.. The agent read it at turn one and did not carry it to turn thirty. It became an AST hook at the commit boundary, with an exit code the agent could not argue with. It is now that hook plus the pinned tests, run on every commit. That closes two of the three moves the shape check left open.

Each step moved the check closer to the code itself. A prompt rule limits what the model reads. A shape gate limits what it writes. A behavior gate limits what the code does. SWE-EVO (Dec 2025) measured a collapse from 72.80% on single-issue tasks to 25.0% across 48 multi-commit evolution tasks, which average 21 modified files. That is a failure of state across commits. Each of these gates is a small piece of state that survives across commits, and the agent does not write it.

The gate is 123 lines, and it took an afternoon. The pin file is six lines of JSON. If you already have an AST hook, add the run step behind it and commit the pin. It costs under a second. It stops a commit that goes green while the code is wrong.

Primary research and documentation

• Failure as a Process: Understanding and Preventing Multi-Turn Drift in Autonomous Coding Agents (Jul 2026): 3,843 trajectories across more than 63,000 execution steps, showing that damaging errors lock in early and silently, before any test runs.
• Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback (May 2026): up to 13 percentage points on MBPP pass@1 by pairing execution results with static checks, and the removal of lint-only reward hacking.
• SWE-EVO: Benchmarking Multi-File Software Evolution Across Sequential Commits (Dec 2025): 48 multi-commit evolution tasks averaging 21 modified files, where a 72.80% single-issue score falls to 25.0%.