An agent fixing a bug has two cheap ways to look done: write the regression test after the fix so it is green on its first run, and reshape a public function to satisfy the one caller in the diff. Part 5 of this series, the behavior gate, is a pre-commit hook that records a content hash of every existing test body, re-runs those tests in a fresh subprocess at the commit boundary, and rejects the commit when a hashed body changed or a hashed test fails. It sees neither move. This part adds two read-only rules in front of the same hook.
Two moves the behavior gate does not see
The Friday commit that did not land
Friday afternoon I asked the agent to stamp a theme tag into every PNG the social-card renderer writes. Its first patch added a theme keyword to render_html_to_png and to build_carousel, two functions with five callers outside the diff. Every test in the diff passed. The pre-commit hook printed one line per function and exited 1. In a two-file scratch repo the line reads:
[R2-fence] render.py: 'render_html_to_png' signature changed (html,output_path)d0 -> (html,output_path,theme)d1
exit=1The name, the shape at HEAD, the shape now. The agent read it, kept the signature, and moved the theme into module state (the patch that passes the fence, below).
What the behavior gate does not see
The test that was green from the start. The agent fixes the bug, then writes the regression test, green on its first run. Nothing shows it failed before the fix, and a new test is not in the pinned baseline the behavior gate hashes.
The fix by reshaping. To satisfy one call site the agent adds a keyword, renames a method, or inlines a public helper. The tests in the diff pass; the callers outside it break at call time.
The rules
R1: the test must be red first
repro_fence.py red --cmd "python3 -m pytest tests/test_x.py::test_bug -q" runs the reproducer on the current tree with shell=False and a hard timeout, and passes only when the command fails. Exit codes 0, 2, 5, 124 and 127 are rejected: a pass, a usage error, a collection error, a timeout, and a missing binary:
REJECTED_EXIT_REASONS: dict[int, str] = {
0: "exited 0 on the current tree; a passing reproducer proves nothing",
2: "exited 2 (usage error); the reproducer command itself is wrong",
5: "exited 5 (import or collection error); the test never ran",
124: "timed out; a hang is not a reproduction",
127: "binary not found or not executable",
}
def check_reproducer(repo: Path, argv: list[str], timeout: int) -> list[Violation]:
"""R1: the reproducer must fail in the failing range on the current tree."""
if not argv:
return [Violation("R1-repro", "no reproducer command supplied")]
code = run_argv(argv, repo, timeout)
reason = REJECTED_EXIT_REASONS.get(code)
if reason is None:
return []
if code == 124:
reason = f"timed out after {timeout}s; a hang is not a reproduction"
return [Violation("R1-repro", f"reproducer {shlex.join(argv)} {reason}")]I gamed the first version within a day. pytest test_x.py || true exits 0 and is rejected, but sh -c "pytest test_x.py; exit 1" exits 1 and sails through. So R1 has a second half, R1-shape: the reproducer may not carry ||, &&, ;, exit, true, or false, including inside a sh -c string. A command that picks its own exit code cannot prove the bug.
R2: the public surface keeps its shape
repro_fence.py fence --rev HEAD --file a.py --file b.py reads each file at rev with git show, parses both versions with ast, and compares public symbols on a normalized signature rather than source text, so a docstring edit passes and a changed parameter list does not:
def signature_of(node: ast.FunctionDef | ast.AsyncFunctionDef) -> str:
"""Normalized parameter shape: `(name,/,name,*va,name,**kw)dN`."""
spec = node.args
names = [arg.arg for arg in spec.posonlyargs]
if spec.posonlyargs:
names.append("/")
names.extend(arg.arg for arg in spec.args)
if spec.vararg is not None:
names.append("*" + spec.vararg.arg)
elif spec.kwonlyargs:
names.append("*")
names.extend(arg.arg for arg in spec.kwonlyargs)
if spec.kwarg is not None:
names.append("**" + spec.kwarg.arg)
defaults = len(spec.defaults) + sum(1 for d in spec.kw_defaults if d is not None)
return f"({','.join(names)})d{defaults}"The dN suffix counts positional and keyword-only defaults together, so appending extra=None changes both the name list and the count, d0 to d1. A name at HEAD that is absent now is a removed symbol; a name in both with a different shape is a changed signature. Leading underscores are skipped, new public symbols are free, and a file the parser does not know prints R2-skip.
def diff_symbols(rel: str, before: dict[str, str], after: dict[str, str]) -> list[Violation]:
"""Public symbols that vanished or changed shape between two sources."""
out: list[Violation] = []
for name, sig in sorted(before.items()):
if name not in after:
out.append(Violation("R2-fence", f"{rel}: public symbol '{name}' was removed"))
elif after[name] != sig:
out.append(
Violation("R2-fence", f"{rel}: '{name}' signature changed {sig} -> {after[name]}")
)
return outIn the pre-commit hook the fence reads the staged copy of each file, so an unstaged edit cannot make a staged one look safe. Above 30 staged files it prints a notice and fails open. On the three-file scratch repo the run is 0.08 s.
The fix
The patch that passes the fence
The renderer needs the theme and the signature cannot change. A module-level setter satisfies both: one public function added, zero changed.
_RENDER_THEME = ["paper"]
def set_render_theme(theme: str) -> None:
"""Records the theme that later render_html_to_png calls stamp into the PNG."""
_RENDER_THEME[0] = theme
def render_html_to_png(html, output_path):
"""Writes html to output_path as a PNG."""
...
stamp_png(output_path, _RENDER_THEME[0]) # new line, same signature
return output_pathEach command handler calls set_render_theme(theme) before it renders. The public signature is byte-identical to HEAD, the behavior changed, and the five callers outside the diff were not touched. The commit lands.
Four gotchas from two weeks behind the rule. A keyword default is still a signature change; add a public name_with(..., *, extra=None) and have the old name delegate. A top-level def test_* is public, so renaming a test is rejected. Inlining a public helper is a removed symbol. And one honest false positive: a deliberate CLI change, audit_social_bundle()d0 -> (slug)d0. The shape that passed read sys.argv[1] inside the unchanged body. That is the cost of the rule.
Show the shell command (reproduce locally)
git add -- render.py && git commit -m 'render: stamp the theme'What each rule can see
Six moves, one scratch repo, three rules. Every cell is an exit code read off the terminal.
| Move | Behavior gate (part 5) | R1 red test | R2 fence |
|---|---|---|---|
| Regression test written green after the fix | Missed, not in the pinned baseline | Caught, exit 1 on code 0 | n/a |
| Reproducer that hangs | n/a | Caught, timed out after 2 s | n/a |
|| true appended to the reproducer | n/a | Caught, R1-shape, before anything runs | n/a |
| Public keyword added to satisfy one caller | Missed while pinned tests pass | n/a | Caught, d0 -> d1 |
| Public helper inlined away | Missed until a pinned test imports it | n/a | Caught, removed symbol |
Private _helper reshaped | Missed | n/a | Allowed by design, exit 0 |
The last row is deliberate. A fence on private names would turn every refactor into an override, and an override typed on every commit stops being a gate. The underscore is the contract.
The boundary
What these gates cannot see
R2 sees Python only. A .ts or .sh file prints R2-skip and passes.
R2 reads names and arity, not types or semantics. A function that keeps its parameter list and changes its return contract passes. That is the behavior gate's job.
R1 proves the test fails now, not that it fails for the right reason. A test that fails on its own typo is red. Exit code 5 catches the import errors; the rest is on the author.
Three papers this year measured the problem from outside. TDAD (Mar 2026) cut regressions on SWE-bench Verified from 6.08% to 1.82% with a code-to-test map at commit time; test-first prompt instructions alone pushed them to 9.94%. How Coding Agents Fail Their Users (May 2026) read 20,574 real sessions; 91.49% of resolutions needed a user correction. DEPBENCH (Aug 2026) set 203 upgrade tasks with hidden signature changes; the best configuration solved 104. The fence is that last problem run backwards.
Same rule, different reader
Part 5, the behavior gate, put the pinned tests at the commit boundary. This part adds the contract on either side of the fix: red before it, same shape after it. The two rules are 361 lines of standard library Python with 26 tests.
The rule behind the Friday rejection had been a sentence in the system prompt for months. The agent read it at turn one. The hook read the staged files at commit time and printed the diff. Same rule, different reader, and only one returns an exit code.
Primary research and documentation
- TDAD: Test-Driven Agentic Development for Regression-Free Code Repair (Mar 2026): regressions on SWE-bench Verified fell from 6.08% to 1.82% with a code-to-test map at commit time; TDD instructions alone pushed them to 9.94%.
- How Coding Agents Fail Their Users (May 2026): 20,574 sessions across 1,639 repositories; 91.49% of visible resolutions needed explicit user correction.
- DEPBENCH: Update from Hell (Aug 2026): 203 dependency-upgrade tasks with hidden signature and API changes; the best agent configuration solved 104 of 203, 51.2%.
- repro_fence.py, the two rules and their tests, on GitHub Gist: the standard-library script this post quotes, with the README and the unittest file.