An AI Agent A program that calls a language model in a loop and lets it use tools, such as search or code. that fixes a bug has two cheap ways to look done. First, it can write the Regression test A test that checks an old feature still works after a change. When it fails, something broke. after the fix. Then the test passes on its first run, and nothing shows that it ever failed without the fix. Second, it can reshape a Public function A function other files may call. In Python, its name does not start with an underscore. to suit the one Caller Code that calls a function. If the function changes shape, the caller can break. in the Diff The lines a change adds, removes or edits, compared with the code before it.. Then the callers outside the diff break when they run. In both cases, every test in the diff passes. So does the behavior gate from Part 5. That gate re-runs the tests that existed before the change. It rejects the commit when one of them was edited or fails. In this part I add two read-only rules to the same Pre-commit hook A script Git runs before it saves a commit. If the script fails, Git refuses the commit., one for each move.
Two moves the behavior gate does not see
The Friday commit that did not land
On a Friday afternoon, I asked the agent to stamp a theme tag into every PNG that my social-card renderer writes. Its first patch added a theme keyword to render_html_to_png and to build_carousel. Those two functions have five callers outside the diff. Every test in the diff passed. Then the pre-commit hook printed one line per function and Exit code The number a program returns when it ends. 0 means success; any other number, such as 1, means failure., which means it refused the commit. In a two-file scratch repo, the line reads:
[R2-fence] render.py: 'render_html_to_png' signature changed (html,output_path)d0 -> (html,output_path,theme=)d1
exit=1The line gives three things: the function's name, its shape at HEAD Git's name for the latest commit on the current branch., and its shape now. The agent read the line and kept the Signature A function's name and its list of parameters. A call must match it.. It moved the theme into module state instead, and that passed. The patch I kept, below, uses a Delegate A function that only calls another function and returns its result..
What the behavior gate does not see
The test that was green from the start. The agent fixes the bug first. Then it writes the regression test, and the test is Green A test that passes. Test tools print a pass in green. on its first run. Nothing shows that it failed before the fix. The behavior gate keeps a hash of the old tests, its pinned baseline, and a new test is not in it.
The fix by reshaping. To satisfy one call site, the agent adds a keyword, renames a method, or inlines a public helper. The tests in the diff pass. The callers outside the diff break when they call the function.
Get the next essay by email
One email per new essay: a production failure, taken apart, and the code that fixed it. Reply to any of them to unsubscribe.
The rules
R1: the test must be red first
R1 says the test must be Red A test that fails. Test tools print a failure in red. first: it must fail before the fix. It checks the Reproducer A test that shows the bug. It fails while the bug is there and passes once it is fixed., the test that shows the bug. I run it like this: repro_fence.py red --cmd "python3 -m pytest tests/test_x.py::test_bug -q". The script runs the reproducer on the current tree, with shell=False and a hard timeout. It passes only when two things are true: pytest A common Python test runner. It exits 1 when a test fails and 0 when all tests pass. exits 1, and its JUnit report An XML file a test runner writes. It lists each test and whether it passed or failed. shows that each named test failed. Every other exit code is rejected with a reason. Among them: 0 means the test passed, 2 means a collection error, 4 means a test id that does not exist, 124 means a timeout, and 127 means a missing binary:
REJECTED_EXIT_REASONS: dict[int, str] = {
0: "exited 0 on the current tree; a passing reproducer proves nothing",
2: "exited 2 (interrupted or collection error); the test never ran",
3: "exited 3 (pytest internal error); the test never ran",
4: "exited 4 (usage error or unknown test id); the test never ran",
5: "exited 5 (no tests collected); the test never ran",
124: "timed out; a hang is not a reproduction",
127: "binary not found or not executable",
}
def check_reproducer(repo: Path, argv: list[str], timeout: int) -> list[Violation]:
"""R1: pytest exits 1 on the current tree and every named test failed."""
with tempfile.TemporaryDirectory() as tmp:
report_xml = Path(tmp, "r1.xml")
extra = [f"--junitxml={report_xml}", f"--rootdir={repo}", "-p", "no:cacheprovider"]
code = run_argv([*argv, *extra], repo, timeout)
cases = junit_cases(report_xml)
if code != 1:
reason = REJECTED_EXIT_REASONS.get(code, f"exited {code}; only exit 1 (tests failed) is red")
if code == 124:
reason = f"timed out after {timeout}s; a hang is not a reproduction"
return [Violation("R1-repro", f"reproducer {shlex.join(argv)} {reason}")]
return [Violation("R1-repro", f"reproducer exited 1 but {test_id} {why}")
for test_id in _test_ids(argv) if (why := _not_failed(test_id, cases))]I gamed the first version within a day. pytest test_x.py || true exits 0, so it is rejected. But sh -c "pytest test_x.py; exit 1" exits 1 and sails through. A list of banned tokens loses the same way, to bash -lc, a pipe into grep, or python3 -c "assert 0". So R1 has a second half, R1-shape, and it is an Allowlist A list of the only things allowed. Anything not on the list is rejected.. The command must start with pytest or python3 -m pytest. After that, it may hold only path::test ids and the flags -q, -v and -x. A command that picks its own exit code cannot prove the bug.
R2: the public surface keeps its shape
R2 says the public surface keeps its shape. The command is repro_fence.py fence --rev HEAD --file a.py --file b.py. For each file, it reads two versions with git show: the one at rev and the one in the Index Git's staging area. It holds the exact file versions the next commit will save.. It parses both with Python's AST Abstract syntax tree. The structure a parser builds from source code, such as functions, classes and signatures. module. Then it compares the public symbols by a normalized signature, not by their source text. So a docstring edit passes, and a changed parameter list does not:
def signature_of(node: ast.FunctionDef | ast.AsyncFunctionDef) -> str:
"""Normalized shape: `[@decorator ][async ](name,name=,/,*va,name,**kw)dN`.
`=` marks a parameter with a default and N counts them, so appending
`extra=None` or moving a default changes the shape. So do `async` and
decorators such as `@property`, which change how callers call it.
"""
spec = node.args
positional = spec.posonlyargs + spec.args
first_default = len(positional) - len(spec.defaults)
names = [a.arg + ("=" if i >= first_default else "") for i, a in enumerate(positional)]
if spec.posonlyargs:
names.insert(len(spec.posonlyargs), "/")
if spec.vararg is not None:
names.append("*" + spec.vararg.arg)
elif spec.kwonlyargs:
names.append("*")
names.extend(a.arg + ("" if d is None else "=") for a, d in zip(spec.kwonlyargs, spec.kw_defaults))
if spec.kwarg is not None:
names.append("**" + spec.kwarg.arg)
defaults = len(spec.defaults) + sum(1 for d in spec.kw_defaults if d is not None)
prefix = "".join(f"@{ast.unparse(d)} " for d in node.decorator_list)
if isinstance(node, ast.AsyncFunctionDef):
prefix += "async "
return f"{prefix}({','.join(names)})d{defaults}"An = marks each parameter that has a Default The value a parameter takes when the caller leaves it out, as in extra=None., and dN counts them. So appending extra=None changes both the name list and the count, from d0 to d1. Moving a default to another parameter is a change too. A name that is at HEAD but absent now is a removed symbol. A name in both versions with a different shape is a changed signature. Names with a leading underscore are private in Python, so the fence skips them, except dunders such as __init__. New public symbols are free. A file the parser does not know prints R2-skip.
def diff_symbols(rel: str, before: dict[str, str], after: dict[str, str]) -> list[Violation]:
"""Public symbols that vanished or changed shape between two sources."""
out: list[Violation] = []
for name, sig in sorted(before.items()):
if name not in after:
out.append(Violation("R2-fence", f"{rel}: public symbol '{name}' was removed"))
elif after[name] != sig:
out.append(
Violation("R2-fence", f"{rel}: '{name}' signature changed {sig} -> {after[name]}")
)
return outThe fence reads the staged copy of each file. So an unstaged edit cannot make a staged one look safe. The hook also hands it deleted and renamed files. A rev that is not a commit, a git error, or a file found on neither side makes it reject. On the three-file scratch repo, the run takes 0.15 s.
The fix
The patch that passes the fence
The renderer needs the theme, and the signature cannot change. The agent's first answer, the one in the video below, was a module-level setter. It passes the fence. But two handlers that render at once Race condition A bug where the result depends on which of two processes happens to finish first. on that shared theme. And a handler that skips the setter inherits the last caller's theme. The patch I kept adds a new public function and has the old one delegate to it: one public function added, zero changed.
def render_html_to_png_with(html, output_path, *, theme="paper"):
"""Writes html to output_path as a PNG stamped with theme."""
...
stamp_png(output_path, theme)
return output_path
def render_html_to_png(html, output_path):
"""Writes html to output_path as a PNG."""
return render_html_to_png_with(html, output_path) # same signature as HEADA handler that needs a theme calls render_html_to_png_with(html, path, theme=theme). The old signature is byte-identical to HEAD. The theme travels with each call. The five callers outside the diff were not touched. The commit lands.
Two weeks behind the rule taught me four gotchas. First, a keyword default is still a signature change, even a keyword-only one. The delegate above is the way through. Second, a top-level def test_* is public, so renaming a test is rejected. Third, inlining a public helper counts as a removed symbol. Fourth, one honest False positive A check flags something as wrong when it is in fact fine.: a deliberate CLI change, audit_social_bundle()d0 -> (slug)d0. The shape that passed read sys.argv[1] inside the unchanged body. That is the cost of the rule.
Show the shell command (reproduce locally)
git add -- render.py && git commit -m 'render: stamp the theme'What each rule can see
Here are six moves in one scratch repo, against three rules. Every cell is an exit code read off the terminal.
| Move | Behavior gate (part 5) | R1: red first | R2: same shape |
|---|---|---|---|
| Regression test written green after the fix | Missed, not in the pinned baseline | Caught, exit 1 on code 0 | Not its job |
| Reproducer that hangs | Not its job | Caught, timed out after 2 s | Not its job |
|| true appended to the reproducer | Not its job | Caught, R1-shape, before anything runs | Not its job |
| Public keyword added to satisfy one caller | Missed while pinned tests pass | Not its job | Caught, d0 -> d1 |
| Public helper inlined away | Missed until a pinned test imports it | Not its job | Caught, removed symbol |
Private _helper reshaped | Missed | Not its job | Allowed by design, exit 0 |
The last row is on purpose. A fence on private names would turn every refactor into an override. And an override I type on every commit stops being a gate. The underscore is the contract.
The boundary
What these gates cannot see
R2 sees Python only. A .ts or .sh file prints R2-skip and passes.
R2 reads names and arity, not types or semantics. Arity is the number of parameters. A function that keeps its parameter list but changes what it promises to return still passes. Catching that is the behavior gate's job.
R1 proves the test fails now, not that it fails for the right reason. A test that fails because of its own typo is still red. A failed import or a missing test id is rejected. The rest is on the author.
Three papers this year measured the problem from outside. TDAD (Mar 2026) cut regressions on SWE-bench Verified A benchmark of real GitHub issues from Python projects. A model must write a patch that fixes each one. from 6.08% to 1.82%. It did this with a code-to-test map at commit time. Test-first prompt instructions alone pushed regressions to 9.94%. How Coding Agents Fail Their Users (May 2026) read 20,574 real sessions. In them, 91.49% of resolutions needed a user correction. DEPBENCH (Aug 2026) set 203 upgrade tasks with hidden signature changes. The best configuration solved 104. The fence is that last problem run backwards.
Same rule, different reader
Part 5, the behavior gate, put the pinned tests at the commit boundary. This part adds the contract on either side of the fix: red before it, same shape after it. The two rules are 430 lines of Standard library The modules that ship with Python itself, so nothing extra needs installing. Python, with 33 tests.
The rule behind the Friday rejection had been a sentence in the System prompt Fixed instructions at the start of every request. They tell the model its role and its rules. for months. The agent read it at turn one. The hook read the staged files at commit time and printed the diff. Same rule, different reader, and only one returns an exit code.
Primary research and documentation
- TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis (Mar 2026): regressions on SWE-bench Verified fell from 6.08% to 1.82% with a code-to-test map at commit time; TDD instructions alone pushed them to 9.94%.
- How Coding Agents Fail Their Users (May 2026): 20,574 sessions across 1,639 repositories; 91.49% of visible resolutions needed explicit user correction.
- DEPBENCH: Update from Hell (Aug 2026): 203 dependency-upgrade tasks with hidden signature and API changes; the best agent configuration solved 104 of 203, 51.2%.
- repro_fence.py, the two rules and their tests, on GitHub Gist: the standard-library script this post quotes, with the README and the unittest file.