---
title: "The Hybrid AI Standard: Routing Between On-Device AI and the Cloud"
date: "August 22, 2026"
description: "Cloud added 300 ms a prompt; on-device froze my app for 2.5 seconds. I run light tasks on the device NPU and send hard ones to a cloud model behind App Check."
category: "Architecture"
canonical: "https://ulukaya.dev/posts/the-hybrid-ai-standard"
---

# The Hybrid AI Standard: Routing Between On-Device AI and the Cloud

I ship AI features in production mobile and web apps. I hit the same physical wall from two opposite sides. First, I routed every prompt to a cloud LLM. On a cellular network, each round trip added 300 ms, and that ruined my interactive UI. My token bill also grew linearly with active users. Then I moved inference fully onto the device. When a user switched apps, the mobile OS evicted my model weights from DRAM. When the user came back, my app froze for 2.5 seconds.

		

So pure cloud and pure on-device both fail under real client limits. I built **The Hybrid AI Standard** to solve this in my own apps. I run frequent, privacy-sensitive tasks locally, on the device. I escalate complex reasoning and enterprise RAG to stateless serverless cloud containers, one step at a time.

		
		

> The Hybrid AI Standard: Run frequent, privacy-sensitive tasks locally on the device. Escalate complex reasoning and enterprise data queries to stateless serverless cloud containers.

	

	
	

## What broke in my terminal and the four pillars

	
		

### 01. Why pure cloud and pure on-device both broke in production

		

I inspected my network traces on mobile cellular connections. Before the first token rendered, each user action that crossed the wire added 200 to 500 ms. That was TLS and HTTP handshake overhead. Real-time autocomplete, UI state classification and input validation need a fast answer. For them, that round trip broke my UI responsiveness budget. It got worse when I tried [synchronous split streaming](/posts/the-leaky-abstraction-vol2) between client and server. Mobile packet jitter stalled my pipeline again and again.

		

Next, I moved everything to a small local model with less than one billion parameters. That solved my network latency, but it exposed a severe mobile memory trap. Mobile operating systems take memory back aggressively while the user multitasks. Whenever I left my app for 60 seconds, the OS memory manager evicted my local LLM weights and KV cache from DRAM. My app then reloaded the weights from flash storage. That caused a Time-to-First-Token (TTFT) freeze of several seconds. Also, my local model had no access to my central production databases, and it could not enforce server-side billing quotas.

	

	
		

### 02. The four physical pillars I use to route workloads

		

I stopped guessing which prompts belong on the device and which belong in the cloud. Now I check every feature against four physical mechanics:

		
			- **Low Latency (<40 ms TTFT):** I run UI state classification, intent detection and inline autocomplete locally, on the device's NPU. The network round trip is 0 ms.

			- **Zero Marginal Token Cost:** The user's device absorbs frequent, low-entropy interactions, the simple and predictable ones. I do not pay per token for cloud inference on every keystroke.

			- **Local Data Privacy:** A prompt that the on-device model answers never crosses the network boundary. My router redacts nothing. So a prompt it escalates reaches the cloud as written.

			- **Multitasking and Offline Resilience:** My core UX keeps working offline. During multitasking, the mobile OS can evict my local weights from DRAM. Then the reload misses my router's on-device deadline. So the router falls back to a serverless cloud endpoint, and local memory restores in the background.

		
	

	
	

## My hybrid orchestration backbone

	
		

### 03. Client-side SDKs and stateless serverless containers

		

My on-device model runs in the browser or on the phone's NPU, for example through the [Chrome Built-in AI Prompt API](https://developer.chrome.com/docs/ai/built-in). Sometimes it hits a reasoning wall, misses its latency deadline, or needs enterprise data. Then my client escalates the request to the cloud. I split that escalation into two paths. A request that only needs a larger model goes through a lightweight client-side AI SDK. A request that needs my data goes to a stateless, autoscaling serverless service. In my stack these are **Firebase AI Logic** and **Google Cloud Run**.

		

In my deployment, the client SDK handles payload serialization, token streaming and automatic retries, and it calls the model directly. Requests that need my production data skip the client SDK. They go to a separate [Google Cloud Run](https://cloud.google.com/run/docs) service. That service scales from zero, runs retrieval over my databases, and then calls the model.

	

	
		

### 04. Cryptographic client attestation and rate limiting

		

Say I expose my cloud AI endpoints to client apps with no check on the device or the user. Then I invite automated scraping and token drainage. So I enforce two guardrails at the edge, with no exceptions:

		

First, I use cryptographic client attestation: [Firebase App Check](https://firebase.google.com/docs/app-check), through Apple App Attest, Android Play Integrity or reCAPTCHA Enterprise. It checks that each request comes from my authentic app on an untampered device. Once I enforce it in the Firebase console, requests without a valid attestation are rejected. [Replay protection](https://firebase.google.com/docs/ai-logic/app-check) makes each App Check token single-use. Google is candid that App Check "prevents some, but not all, abuse vectors," so it is one guardrail, not the whole defense.

		

Second, I lower the Firebase AI Logic [per-user rate limit](https://firebase.google.com/docs/ai-logic/quotas). Its default is 100 requests per minute. That is far more than one person needs, and Google recommends tuning it to the app. A tighter limit slows how fast one client can spend my project quota. It does not stop many clients at once.

	

	
	

## Reference architecture and trade-offs

	

Figure 2 follows one request through my router, step by step. The deadline comes from my code in section 05: 1,500 ms. The other times come from the introduction and section 02. A warm answer takes under 40 ms, and an eviction costs a 2.5 second freeze.

	
		
		
		
		
	

	

Figure 2 uses my code's deadline and the times I saw in my own apps. The lab below uses numbers your browser measures. Start your camera, and the lab times four legs on a single frame. First it grabs the frame into a canvas. Then it encodes the frame as a JPEG. Next, the local model writes a one-line caption, if your browser has one. Last, it uploads the same number of bytes to this server. The upload carries random bytes, not the frame, so camera pixels never leave your device. The lab does not call a hosted model; that leg would add inference time on top of the upload.

	

*[Interactive Architecture Simulator: webcam-latency - Explore live at https://ulukaya.dev/posts/the-hybrid-ai-standard#lab-webcam-latency]*

	

	
		

### 05. My TypeScript hybrid routing engine

		

Here is the production TypeScript router I built, `HybridAIRouter`. It keeps one on-device NPU session loaded, which prevents VRAM thrashing, and gives each request a fresh clone of it. It checks that the local runtime is available. It escalates to Gemini through Firebase AI Logic in three cases. The local model is unavailable. Or it misses its deadline while evicted weights reload. Or the caller skips the device tier with `requiresEnterpriseContext`. Requests that need the enterprise data itself go to my separate retrieval service:

		

```
import { initializeApp } from 'firebase/app';
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check';
import { getAI, getGenerativeModel, GoogleAIBackend } from 'firebase/ai';
// Prompt API types: npm i -D @types/dom-chromium-ai

// Firebase AI Logic attaches an App Check token to each call; I enforce it in the
// Firebase console (required for AI Logic from November 2, 2026)
const app = initializeApp({ apiKey: 'FIREBASE_WEB_API_KEY', projectId: 'my-production-app', appId: '1:12345:web:abcdef' });
initializeAppCheck(app, {
  provider: new ReCaptchaEnterpriseProvider('RECAPTCHA_SITE_KEY'),
  isTokenAutoRefreshEnabled: true,
});

const ai = getAI(app, {
  backend: new GoogleAIBackend(),
  // Replay protection: one token per request (firebase v12.14+), which also needs
  // replay protection set to Enforced in the Firebase console
  useLimitedUseAppCheckTokens: true,
});
const LOCAL_DEADLINE_MS = 1500; // a warm on-device answer lands well inside this

export interface RoutingResult {
  text: string;
  tier: 'on-device' | 'cloud';
}

export class HybridAIRouter {
  // Cache the create() promise itself, so concurrent first calls share one session
  private baseSession: Promise | null = null;
  private cloudModel = getGenerativeModel(ai, { model: 'gemini-3.6-flash' });

  public async execute(prompt: string, requiresEnterpriseContext = false): Promise {
    // 1. Attempt On-Device Tier if the task is local and the model is on the device
    if (!requiresEnterpriseContext && 'LanguageModel' in self) {
      // The deadline starts before availability(), so a stalled check counts against it too
      const signal = AbortSignal.timeout(LOCAL_DEADLINE_MS);
      const deadline = new Promise((_, reject) =>
        signal.addEventListener('abort', () => reject(signal.reason)));
      try {
        const text = await Promise.race([this.promptLocal(prompt, signal), deadline]);
        if (text !== null) return { text, tier: 'on-device' };
      } catch (err) {
        // Deadline missed (evicted weights reloading) or session lost: escalate to Firebase AI Logic
        console.warn('On-device tier missed its deadline or failed; escalating to cloud', err);
        if (!signal.aborted) this.baseSession = null; // rebuild a broken session, let a slow one finish loading
      }
    }

    // 2. Cloud Escalation via Firebase AI Logic SDK (App Check token attached automatically)
    const result = await this.cloudModel.generateContent(prompt);
    return { text: result.response.text(), tier: 'cloud' };
  }

  private async promptLocal(prompt: string, signal: AbortSignal): Promise {
    if ((await LanguageModel.availability()) !== 'available') return null; // not on the device: go to cloud
    this.baseSession ??= LanguageModel.create();
    // The base session is never prompted, so each clone starts with an empty history
    const session = await (await this.baseSession).clone({ signal });
    try {
      return await session.prompt(prompt, { signal });
    } finally {
      session.destroy(); // frees the clone only; the base session keeps the model loaded
    }
  }
}
```

		
	

	
		

### 06. Architectural decision matrix

		

I use this empirical decision matrix to audit latency, offline resilience and token spend in each setup:

		
			
				
					
						Dimension
						Pure On-Device
						Pure Cloud
						The Hybrid AI Standard
					
				
				
					
						How long the user waits **Average Latency**
						Fast, until an eviction. 15 to 40 ms (warm) / 2.5 s freeze (DRAM eviction)
						Slow on every request. 250 to 600 ms (WAN RTT bound)
						Fast for the UI. 15 to 40 ms (UI) / 250 ms (Reasoning & RAG)
					
					
						What each request costs **Marginal Token Cost**
						Free. $0.00 (Client NPU silicon)
						Grows with users. Linear with active users
						Far less cloud spend. 70 to 80% reduction in cloud token spend
					
					
						Where the prompt goes **Data Privacy**
						Nothing leaves. 100% Local
						Every prompt leaves. Raw prompt egress over wire
						Only escalated prompts leave. Local answers stay on device; escalated prompts egress as written
					
					
						With no network **Offline Availability**
						Works. Fully functional (within memory bounds)
						Stops. Fails entirely
						Core features work. Core UX remains functional; graceful cloud fallback
					
				
			
		
	

	
		

## Industry validation and benchmarks

		

Recent 2026 systems research confirms that others hit the same physical bottlenecks, on cellular WANs and in mobile operating systems:

		
			- [Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration (Gui et al., Aug 2026)](https://arxiv.org/abs/2608.28726v1): Confirms that one-shot routers, which decide before generation starts, fail when a prompt hits a reasoning wall mid-stream. Watching the token sampling probabilities during generation routes over 10x faster and gives 75% higher throughput than static request routers.

			- [AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inference (Zhang et al., Sep 2026)](https://arxiv.org/abs/2609.02514v1): Confirms that synchronous edge-cloud token verification collapses under mobile packet jitter. An asymmetric local state cache in place of lock-step verification gives a 3.52x throughput speedup, down to 50 Kbps WAN conditions.

			- [mzCache: On-Device LLM Memory Management under Multitasking (Yu et al., Sep 2026)](https://arxiv.org/abs/2609.01338v1): Confirms that mobile OS app switching evicts LLM weights and KV caches from DRAM. Splitting model memory into shared GPU/CPU restoration buffers cuts post-eviction TTFT freezes by 2.1x to 5.5x.

			- [Efficient and Privacy-Aware Edge-Cloud Collaborative Inference for Large Language Models (Li et al., Jul 2026)](https://arxiv.org/abs/2607.13093v4): Confirms that one approach cuts downlink payload bytes by 67.4% and per-token latency by 46.1%: sanitize PII tokens on the device first, then sync an authenticated KV cache with cloud containers.

			- [Chrome Built-in AI and Prompt API Documentation](https://developer.chrome.com/docs/ai/built-in): Standard client-side on-device model execution via browser NPU APIs referenced in Section 03 and Section 05.

			- [Google Cloud Run Documentation](https://cloud.google.com/run/docs): Stateless serverless container execution for cloud reasoning backends referenced in Section 03.

			- [Firebase App Check Documentation](https://firebase.google.com/docs/app-check): Cryptographic client attestation securing cloud endpoints against unauthorized traffic referenced in Section 04.