Firebase for Builders Lead & DevRel Manager at Google. Technical strategy, developer onboarding.
I write about the places where production AI systems break, and I build the instruments that show you where yours will. Each essay takes one failure apart in plain words, from the bill that arrives after an agent loop runs all night to the coding agent that passes every test and still ships the wrong change, and shows what fixed it. If you are new here, start with the ten production failure modes of AI agents.
No. 6 in Ghost in the Runtime
Two pre-commit rules for coding agents: R1 rejects a reproducer that exits 0 before the fix, R2 rejects a public signature that changed since HEAD. Stdlib only.
How AI agents go wrong in real work, from agreeing with everything to passing tests on broken code, and the gates that catch it.
What AI costs once an agent runs on its own, why spend caps fire late, and which turns should never go to the most expensive model.
Why a streaming reply breaks between the model and the screen: torn JSON, dropped tokens, and a browser that hangs up.
What an agent believes about your system, and how static files and stale documents quietly corrupt it.
The behavior gate runs the pinned tests, so a regression test written green after the fix and a public function reshaped to satisfy one caller both pass it. R1 runs the reproducer and rejects exit 0, a hang, and a command that picks its own exit code. R2 parses each staged Python file at HEAD and in the index and rejects a public symbol that vanished or changed arity.
An AST pre-commit gate answers questions about the text of a program, so an agent can hollow an assert or edit a fixture and still exit 0. The behavior gate pins each baseline test body by sha256, runs the pinned set in a fresh subprocess under a wall-clock budget, and echoes the failing assertion back verbatim.
Most turns in an agent trajectory are bookkeeping. Route them to a Workhorse tier model by default, escalate on plan steps, repeated gate failures, public export diffs, and schema failures, and cap the escalations.
Why 72.8% SWE-Bench coding agents collapse to 25.0% on multi-file repositories, and how I replaced 4,000 lines of markdown system prompts with a deterministic 45-line Python AST pre-commit verification harness.
Why upgrading to the #1 Chatbot Arena model makes your production agent swarm less reliable. Scalar ELO rankings measure single-turn chat, not multi-turn monorepo execution. Here is how to build mechanical compiler defenses that eliminate state drift.
Stuffing markdown instructions into system prompts burns 20,000 tokens per turn just to establish baseline behavior. Here is how to distill prompt-heavy agent skills into deterministic, zero-token code tools across serverless and on-device runtimes.
Routing every prompt to a cloud LLM destroys latency budgets, while running entire reasoning pipelines locally exhausts client memory. Here is how to dynamically route requests across on-device NPU tiers and Cloud Run serverless backends.
Treating point-in-time documentation as ground truth blindfolds autonomous agents to live system drift. Here is how to architect 4-plane epistemic triangulation across living state, runtime telemetry, and skeptical verification.
Long-running agent tool executions silently sever TCP connections when they exceed NAT idle-timeout thresholds. Part 2 of The Leaky Abstraction details idempotent backend stream reassembly with Cloud Run and Firestore, directly paired with Part 3 on client-side runtime resilience.
Static markdown summary indexes inevitably decouple from live database state, causing multi-turn agents to hallucinate over stale context. Here is how to enforce ACID transaction boundaries and atomic session locks with Firestore.
Naive JSON.parse calls on LLM streams crash production servers whenever TCP packet boundaries split multi-byte UTF-8 characters mid-flight. Here is how to build stateful, byte-level stream reassemblers in Node.js.
Most production agent failures in web and mobile apps are silent transport drops rather than model hallucinations. Part 3 of The Leaky Abstraction covers WebSocket keep-alives, UI backpressure, and App Check attestation, completing the backend stream reassembly architecture from Part 2.
Left unchecked, autonomous agents enter sycophantic echo chambers and infinite action loops that mimic human cognitive biases. Here are the 10 architectural primitives that enforce objective state verification.
Cloud billing spend caps operate as asynchronous account fuses that take hours to trip, long after a runaway prompt loop has drained your budget. Here is how to build a real-time, 3-layer tokenomics circuit breaker.
Prompt engineering discipline optimizes development costs, but only deterministic code-level idempotency guards and atomic token buckets prevent financial ruin during live production traffic spikes.
15 essays. Full archive at /posts.
Your agent loop ran all night; the cap fired late
billing-lag
The reply cut off mid-sentence and came back doubled
chunk-drop
Silent dead air, then the connection died
idle-timeout
Parallel writes quietly overwrote each other
split-brain
Multibyte characters tore at a chunk boundary
stream-tear
Eight engine notes behind the articles, modeling mechanisms rather than products
the-bench
Token spend against a fixed monthly budget, per model tier and cache hit rate
calculators
Prompt text compiled into a deterministic agent spec tree and schema contract
generators
Full log at /til.
Blueprints: 6 reference designs for agent runtimes, memory stores, tokenomics, and brownfield migration at /blueprints.