16 essays, newest first
Writing
Each essay takes one production AI failure apart in plain words and shows the code that fixed it. Every essay is also a podcast episode: add the feed to your podcast app to listen.
Start here
Ghost in the Runtime
How AI agents go wrong in real work, from agreeing with everything to passing tests on broken code, and the gates that catch it.
AI Tokenomics
What AI costs once an agent runs on its own, why spend caps fire late, and which turns should never go to the most expensive model.
The Leaky Abstraction
Why a streaming reply breaks between the model and the screen: torn JSON, dropped tokens, and a browser that hangs up.
The Epistemic Engine
What an agent believes about your system, and how static files and stale documents quietly corrupt it.
Your AI Fixed the Bug and Every Test Passed, but the Tests Skipped the Fix
My agent fixed a bug with a new raise statement. Its 14 tests passed in one second and skipped it. I added a pre-commit hook that lists every skipped line.
Your AI Says the Bug Is Fixed, but the Test Never Failed: Two Checks Before It Ships
My agent reshaped two public functions to suit one caller; five others sat outside the diff. Two git checks block that and a test that passes before the fix.
When the AST Hook Goes Green and the Test Still Fails: The Behavior Gate
An AST hook can be gamed. This second pre-commit hook locks each baseline test by hash, runs it against staged code, and rejects the commit on a failure.
Stop Sending Every Agent Turn to the Frontier Model
Most of an agent's 30 to 60 turns read a file or run a test. I default them to a Workhorse tier model and escalate to the Frontier tier on four signals.
The Crutch vs. the Operating System: Why I Deleted 4,000 Lines of Agent Prompts
Coding agents at 72.8% on SWE-Bench drop to 25% on multi-file repos. I replaced 4,000 lines of markdown prompts with a 45-line AST gate in the commit hook.
Your #1 Arena Model Fails in Real Repositories: The Leaderboard Mirage
The #1 leaderboard model failed 34% of my edge cases. Leaderboards rank single turns and my agents run 30 to 60, so I put compiler gates in the commit loop.
Code Over Context: Why Written Agent Skills Break in Production
10 markdown skill files cost my agent 22,000 tokens per turn and broke on smaller models at turn 4. I distill written skills into deterministic code tools.
The Hybrid AI Standard: Routing Between On-Device AI and the Cloud
Cloud added 300 ms a prompt; on-device froze my app for 2.5 seconds. I run light tasks on the device NPU and send hard ones to a cloud model behind App Check.
One Outdated Doc Fooled My Agent: Check Each Claim Against Live Data Before It Acts
My agent read one Google Doc and reported a migration my team abandoned two weeks earlier. I make it confirm each claim across 4 planes before it acts.
Dropped Tokens: Fixing Multi-Turn Agent Streams That Die Mid-Flight
A 25-second tool call went silent, a corporate proxy cut it at 15 seconds, and a reconnect ran the tool twice. Keep-alives and a Last-Event-ID resume fix both.
Two Writers, One Index: How Static Files Corrupt Agent Memory
Two subagents wrote one record and the second write erased the first 80 ms later. I replaced my markdown summary index with Firestore transactions on Cloud Run.
Network Chunks Corrupted My Text and Crashed My JSON: Buffer Before You Parse
A TCP chunk split a 4-byte emoji in my LLM stream and the UI showed a U+FFFD diamond. I built a stateful Node.js reassembler that buffers partial UTF-8 bytes.
200 Tokens a Second Locked Up My Browser: Client-Side Defense for Agent Streams
Pushing 200 tokens a second into React state locked my browser at 8 FPS. I paint at most once a frame and send keep-alives so idle proxies keep the stream open.
Why Your AI Agent Agrees With Everything: 10 Production Failure Modes
My review agent approved a regex, then reversed itself when I asked the opposite question. I map the 10 biases behind that and the runtime check for each one.
Why a $50 Cloud Spend Cap Won't Save You From an Agent Loop
A runaway loop burned $412.00 past my $50.00 budget before billing stopped. A Firebase spend cap pauses one service for all users, so I built a 3-layer defense.
78% of My AI Bill Was Waste: Where Prompt Discipline Ends and Runtime Guards Begin
78% of my inference bill was repeated prompts and unpruned history. The 11 Principles of AI Tokenomics cover development; I add the guards live runtimes need.
Get the next essay by email
One email per new essay: a production failure, taken apart, and the code that fixed it. Reply to any of them to unsubscribe.