---
title: "Your AI Says the Bug Is Fixed, but the Test Never Failed: Two Checks Before It Ships"
date: "September 19, 2026"
description: "Two pre-commit rules for coding agents: R1 rejects a reproducer that exits 0 before the fix, R2 rejects a public signature that changed since HEAD. Stdlib only."
category: "Systems Architecture"
canonical: "https://ulukaya.dev/posts/the-repro-fence"
---

# Your AI Says the Bug Is Fixed, but the Test Never Failed: Two Checks Before It Ships

An agent fixing a bug has two cheap ways to look done: write the regression test after the fix so it is green on its first run, and reshape a public function to satisfy the one caller in the diff. [Part 5 of this series, the behavior gate,](/posts/the-behavior-gate) is a pre-commit hook that records a content hash of every existing test body, re-runs those tests in a fresh subprocess at the commit boundary, and rejects the commit when a hashed body changed or a hashed test fails. It sees neither move. This part adds two read-only rules in front of the same hook.

		
	

	
	

## Two moves the behavior gate does not see

	
		

### 01. The Friday commit that did not land

		

Friday afternoon I asked the agent to stamp a theme tag into every PNG the social-card renderer writes. Its first patch added a `theme` keyword to `render_html_to_png` and to `build_carousel`, two functions with five callers outside the diff. Every test in the diff passed. The pre-commit hook printed one line per function and exited 1. In a two-file scratch repo the line reads:

		

```
[R2-fence] render.py: 'render_html_to_png' signature changed (html,output_path)d0 -> (html,output_path,theme)d1
exit=1
```

		

The name, the shape at HEAD, the shape now. The agent read it, kept the signature, and moved the theme into module state ([the patch that passes the fence](#setter-fix), below).

	

	
		

### 02. What the behavior gate does not see

		

**The test that was green from the start.** The agent fixes the bug, then writes the regression test, green on its first run. Nothing shows it failed before the fix, and a new test is not in the pinned baseline the behavior gate hashes.

		

**The fix by reshaping.** To satisfy one call site the agent adds a keyword, renames a method, or inlines a public helper. The tests in the diff pass; the callers outside it break at call time.

	

	
	

## The rules

	
		

### 03. R1: the test must be red first

		

`repro_fence.py red --cmd "python3 -m pytest tests/test_x.py::test_bug -q"` runs the reproducer on the current tree with `shell=False` and a hard timeout, and passes only when the command fails. Exit codes 0, 2, 5, 124 and 127 are rejected: a pass, a usage error, a collection error, a timeout, and a missing binary:

		

```
REJECTED_EXIT_REASONS: dict[int, str] = {
    0: "exited 0 on the current tree; a passing reproducer proves nothing",
    2: "exited 2 (usage error); the reproducer command itself is wrong",
    5: "exited 5 (import or collection error); the test never ran",
    124: "timed out; a hang is not a reproduction",
    127: "binary not found or not executable",
}

def check_reproducer(repo: Path, argv: list[str], timeout: int) -> list[Violation]:
    """R1: the reproducer must fail in the failing range on the current tree."""
    if not argv:
        return [Violation("R1-repro", "no reproducer command supplied")]
    code = run_argv(argv, repo, timeout)
    reason = REJECTED_EXIT_REASONS.get(code)
    if reason is None:
        return []
    if code == 124:
        reason = f"timed out after {timeout}s; a hang is not a reproduction"
    return [Violation("R1-repro", f"reproducer {shlex.join(argv)} {reason}")]
```

		

I gamed the first version within a day. `pytest test_x.py || true` exits 0 and is rejected, but `sh -c "pytest test_x.py; exit 1"` exits 1 and sails through. So R1 has a second half, R1-shape: the reproducer may not carry `||`, `&&`, `;`, `exit`, `true`, or `false`, including inside a `sh -c` string. A command that picks its own exit code cannot prove the bug.

	

	
		

### 04. R2: the public surface keeps its shape

		

`repro_fence.py fence --rev HEAD --file a.py --file b.py` reads each file at `rev` with `git show`, parses both versions with `ast`, and compares public symbols on a normalized signature rather than source text, so a docstring edit passes and a changed parameter list does not:

		

```
def signature_of(node: ast.FunctionDef | ast.AsyncFunctionDef) -> str:
    """Normalized parameter shape: `(name,/,name,*va,name,**kw)dN`."""
    spec = node.args
    names = [arg.arg for arg in spec.posonlyargs]
    if spec.posonlyargs:
        names.append("/")
    names.extend(arg.arg for arg in spec.args)
    if spec.vararg is not None:
        names.append("*" + spec.vararg.arg)
    elif spec.kwonlyargs:
        names.append("*")
    names.extend(arg.arg for arg in spec.kwonlyargs)
    if spec.kwarg is not None:
        names.append("**" + spec.kwarg.arg)
    defaults = len(spec.defaults) + sum(1 for d in spec.kw_defaults if d is not None)
    return f"({','.join(names)})d{defaults}"
```

		

The `dN` suffix counts positional and keyword-only defaults together, so appending `extra=None` changes both the name list and the count, `d0` to `d1`. A name at HEAD that is absent now is a removed symbol; a name in both with a different shape is a changed signature. Leading underscores are skipped, new public symbols are free, and a file the parser does not know prints `R2-skip`.

		

```
def diff_symbols(rel: str, before: dict[str, str], after: dict[str, str]) -> list[Violation]:
    """Public symbols that vanished or changed shape between two sources."""
    out: list[Violation] = []
    for name, sig in sorted(before.items()):
        if name not in after:
            out.append(Violation("R2-fence", f"{rel}: public symbol '{name}' was removed"))
        elif after[name] != sig:
            out.append(
                Violation("R2-fence", f"{rel}: '{name}' signature changed {sig} -> {after[name]}")
            )
    return out
```

		

In the pre-commit hook the fence reads the staged copy of each file, so an unstaged edit cannot make a staged one look safe. Above 30 staged files it prints a notice and fails open. On the three-file scratch repo the run is 0.08 s.

	

	
	

## The fix

	
		

### 05. The patch that passes the fence

		

The renderer needs the theme and the signature cannot change. A module-level setter satisfies both: one public function added, zero changed.

		

```
_RENDER_THEME = ["paper"]

def set_render_theme(theme: str) -> None:
    """Records the theme that later render_html_to_png calls stamp into the PNG."""
    _RENDER_THEME[0] = theme

def render_html_to_png(html, output_path):
    """Writes html to output_path as a PNG."""
    ...
    stamp_png(output_path, _RENDER_THEME[0])   # new line, same signature
    return output_path
```

		

Each command handler calls `set_render_theme(theme)` before it renders. The public signature is byte-identical to HEAD, the behavior changed, and the five callers outside the diff were not touched. The commit lands.

		

Four gotchas from two weeks behind the rule. A keyword default is still a signature change; add a public `name_with(..., *, extra=None)` and have the old name delegate. A top-level `def test_*` is public, so renaming a test is rejected. Inlining a public helper is a removed symbol. And one honest false positive: a deliberate CLI change, `audit_social_bundle()d0 -> (slug)d0`. The shape that passed read `sys.argv[1]` inside the unchanged body. That is the cost of the rule.

		
	

	
		

### 06. What each rule can see

		

Six moves, one scratch repo, three rules. Every cell is an exit code read off the terminal.

		
			
				
					
						Move
						Behavior gate (part 5)
						R1 red test
						R2 fence
					
				
				
					
						**Regression test written green after the fix**
						Missed, not in the pinned baseline
						Caught, exit 1 on code 0
						n/a
					
					
						**Reproducer that hangs**
						n/a
						Caught, timed out after 2 s
						n/a
					
					
						**`|| true` appended to the reproducer**
						n/a
						Caught, R1-shape, before anything runs
						n/a
					
					
						**Public keyword added to satisfy one caller**
						Missed while pinned tests pass
						n/a
						Caught, `d0 -> d1`
					
					
						**Public helper inlined away**
						Missed until a pinned test imports it
						n/a
						Caught, removed symbol
					
					
						**Private `_helper` reshaped**
						Missed
						n/a
						Allowed by design, exit 0
					
				
			
		

		

The last row is deliberate. A fence on private names would turn every refactor into an override, and an override typed on every commit stops being a gate. The underscore is the contract.

	

	
	

## The boundary

	
		

### 07. What these gates cannot see

		

**R2 sees Python only.** A `.ts` or `.sh` file prints `R2-skip` and passes.

		

**R2 reads names and arity, not types or semantics.** A function that keeps its parameter list and changes its return contract passes. That is the behavior gate's job.

		

**R1 proves the test fails now, not that it fails for the right reason.** A test that fails on its own typo is red. Exit code 5 catches the import errors; the rest is on the author.

		

Three papers this year measured the problem from outside. [TDAD (Mar 2026)](https://arxiv.org/abs/2603.17973) cut regressions on SWE-bench Verified from 6.08% to 1.82% with a code-to-test map at commit time; test-first prompt instructions alone pushed them to 9.94%. [How Coding Agents Fail Their Users (May 2026)](https://arxiv.org/abs/2605.29442) read 20,574 real sessions; 91.49% of resolutions needed a user correction. [DEPBENCH (Aug 2026)](https://arxiv.org/abs/2608.30300) set 203 upgrade tasks with hidden signature changes; the best configuration solved 104. The fence is that last problem run backwards.

	

	
		

### 08. Same rule, different reader

		

Part 5, the behavior gate, put the pinned tests at the commit boundary. This part adds the contract on either side of the fix: red before it, same shape after it. The two rules are 361 lines of standard library Python with 26 tests.

		

The rule behind the Friday rejection had been a sentence in the system prompt for months. The agent read it at turn one. The hook read the staged files at commit time and printed the diff. Same rule, different reader, and only one returns an exit code.

	

	
		

## Primary research and documentation

		
			- [TDAD: Test-Driven Agentic Development for Regression-Free Code Repair (Mar 2026)](https://arxiv.org/abs/2603.17973): regressions on SWE-bench Verified fell from 6.08% to 1.82% with a code-to-test map at commit time; TDD instructions alone pushed them to 9.94%.

			- [How Coding Agents Fail Their Users (May 2026)](https://arxiv.org/abs/2605.29442): 20,574 sessions across 1,639 repositories; 91.49% of visible resolutions needed explicit user correction.

			- [DEPBENCH: Update from Hell (Aug 2026)](https://arxiv.org/abs/2608.30300): 203 dependency-upgrade tasks with hidden signature and API changes; the best agent configuration solved 104 of 203, 51.2%.

			- [repro_fence.py, the two rules and their tests, on GitHub Gist](https://gist.github.com/ulukaya/edb49aa8755b1991fc8f6b8bab143c9e): the standard-library script this post quotes, with the README and the unittest file.