The Hybrid AI Standard: Routing Between On-Device AI and the Cloud

I inspected my network traces on mobile cellular connections. Before the first token rendered, each user action that crossed the wire added 200 to 500 ms. That was TLSTLS The encryption that protects web traffic. Each new connection starts with a handshake of several messages. and HTTP handshake overhead. Real-time autocomplete, UI state classification and input validation need a fast answer. For them, that round trip broke my UI responsiveness budget. It got worse when I tried synchronous split streaming between client and server. Mobile packet jitterJitter Random changes in how long each network packet takes to arrive. stalled my pipeline again and again.

Next, I moved everything to a small local model with less than one billion parametersParameter One learned number inside a model. A model's size is counted in parameters.. That solved my network latency, but it exposed a severe mobile memory trap. Mobile operating systems take memory back aggressively while the user multitasks. Whenever I left my app for 60 seconds, the OS memory manager evicted my local LLM weights and KV cacheKV cache The attention keys and values a model saved for tokens it already read. Reusing them skips recomputing that prefix. from DRAM. My app then reloaded the weights from flash storage. That caused a Time-to-First-TokenTTFT Time to first token. How long the user waits before the first token of the answer appears. (TTFT) freeze of several seconds. Also, my local model had no access to my central production databases, and it could not enforce server-side billing quotas.

I stopped guessing which prompts belong on the device and which belong in the cloud. Now I check every feature against four physical mechanics:

• Low Latency (<40 ms TTFT): I run UI state classification, intent detection and inline autocomplete locally, on the device's NPUNPU Neural processing unit. A chip in a phone or laptop built to run AI models fast and with little power.. The network round trip is 0 ms.
• Zero Marginal Token Cost: The user's device absorbs frequent, low-entropy interactions, the simple and predictable ones. I do not pay per token for cloud inference on every keystroke.
• Local Data Privacy: A prompt that the on-device model answers never crosses the network boundary. My router redacts nothing. So a prompt it escalates reaches the cloud as written.
• Multitasking and Offline Resilience: My core UX keeps working offline. During multitasking, the mobile OS can evict my local weights from DRAM. Then the reload misses my router's on-device deadline. So the router falls back to a serverless cloud endpoint, and local memory restores in the background.

My on-device model runs in the browser or on the phone's NPU, for example through the Chrome Built-in AI Prompt API. Sometimes it hits a reasoning wall, misses its latency deadline, or needs enterprise data. Then my client escalates the request to the cloud. I split that escalation into two paths. A request that only needs a larger model goes through a lightweight client-side AI SDKSDK Software development kit. A library a vendor ships so my app can call its service.. A request that needs my data goes to a stateless, autoscaling serverless service. In my stack these are Firebase AI Logic and Google Cloud RunCloud Run Google's service that runs my containers on demand and stops them, down to zero, when no requests come..

In my deployment, the client SDK handles payload serialization, token streaming and automatic retries, and it calls the model directly. Requests that need my production data skip the client SDK. They go to a separate Google Cloud Run service. That service scales from zero, runs retrieval over my databases, and then calls the model.

Say I expose my cloud AI endpoints to client apps with no check on the device or the user. Then I invite automated scraping and token drainage. So I enforce two guardrails at the edge, with no exceptions:

First, I use cryptographic client attestationAttestation Proof, signed by the device or platform, that a request comes from my real app and not from a script.: Firebase App Check, through Apple App Attest, Android Play Integrity or reCAPTCHA Enterprise. It checks that each request comes from my authentic app on an untampered device. Once I enforce it in the Firebase console, requests without a valid attestation are rejected. Replay protection makes each App Check token single-use. Google is candid that App Check "prevents some, but not all, abuse vectors," so it is one guardrail, not the whole defense.

Second, I lower the Firebase AI Logic per-user rate limit. Its default is 100 requests per minute. That is far more than one person needs, and Google recommends tuning it to the app. A tighter limit slows how fast one client can spend my project quota. It does not stop many clients at once.

Here is the production TypeScript router I built, HybridAIRouter. It keeps one on-device NPU session loaded, which prevents VRAMVRAM Memory on the graphics or AI chip. Loading a model into it again and again wastes time. thrashing, and gives each request a fresh clone of it. It checks that the local runtime is available. It escalates to Gemini through Firebase AI Logic in three cases. The local model is unavailable. Or it misses its deadline while evicted weights reload. Or the caller skips the device tier with requiresEnterpriseContext. Requests that need the enterprise data itself go to my separate retrieval service:

import { initializeApp } from 'firebase/app';
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check';
import { getAI, getGenerativeModel, GoogleAIBackend } from 'firebase/ai';
// Prompt API types: npm i -D @types/dom-chromium-ai

// Firebase AI Logic attaches an App Check token to each call; I enforce it in the
// Firebase console (required for AI Logic from November 2, 2026)
const app = initializeApp({ apiKey: 'FIREBASE_WEB_API_KEY', projectId: 'my-production-app', appId: '1:12345:web:abcdef' });
initializeAppCheck(app, {
  provider: new ReCaptchaEnterpriseProvider('RECAPTCHA_SITE_KEY'),
  isTokenAutoRefreshEnabled: true,
});

const ai = getAI(app, {
  backend: new GoogleAIBackend(),
  // Replay protection: one token per request (firebase v12.14+), which also needs
  // replay protection set to Enforced in the Firebase console
  useLimitedUseAppCheckTokens: true,
});
const LOCAL_DEADLINE_MS = 1500; // a warm on-device answer lands well inside this

export interface RoutingResult {
  text: string;
  tier: 'on-device' | 'cloud';
}

export class HybridAIRouter {
  // Cache the create() promise itself, so concurrent first calls share one session
  private baseSession: Promise<LanguageModel> | null = null;
  private cloudModel = getGenerativeModel(ai, { model: 'gemini-3.6-flash' });

  public async execute(prompt: string, requiresEnterpriseContext = false): Promise<RoutingResult> {
    // 1. Attempt On-Device Tier if the task is local and the model is on the device
    if (!requiresEnterpriseContext && 'LanguageModel' in self) {
      // The deadline starts before availability(), so a stalled check counts against it too
      const signal = AbortSignal.timeout(LOCAL_DEADLINE_MS);
      const deadline = new Promise<never>((_, reject) =>
        signal.addEventListener('abort', () => reject(signal.reason)));
      try {
        const text = await Promise.race([this.promptLocal(prompt, signal), deadline]);
        if (text !== null) return { text, tier: 'on-device' };
      } catch (err) {
        // Deadline missed (evicted weights reloading) or session lost: escalate to Firebase AI Logic
        console.warn('On-device tier missed its deadline or failed; escalating to cloud', err);
        if (!signal.aborted) this.baseSession = null; // rebuild a broken session, let a slow one finish loading
      }
    }

    // 2. Cloud Escalation via Firebase AI Logic SDK (App Check token attached automatically)
    const result = await this.cloudModel.generateContent(prompt);
    return { text: result.response.text(), tier: 'cloud' };
  }

  private async promptLocal(prompt: string, signal: AbortSignal): Promise<string | null> {
    if ((await LanguageModel.availability()) !== 'available') return null; // not on the device: go to cloud
    this.baseSession ??= LanguageModel.create();
    // The base session is never prompted, so each clone starts with an empty history
    const session = await (await this.baseSession).clone({ signal });
    try {
      return await session.prompt(prompt, { signal });
    } finally {
      session.destroy(); // frees the clone only; the base session keeps the model loaded
    }
  }
}

I use this empirical decision matrix to audit latency, offline resilience and token spend in each setup:

Industry validation and benchmarks

Recent 2026 systems research confirms that others hit the same physical bottlenecks, on cellular WANs and in mobile operating systems:

• Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration (Gui et al., Aug 2026): Confirms that one-shot routers, which decide before generation starts, fail when a prompt hits a reasoning wall mid-stream. Watching the token sampling probabilities during generation routes over 10x faster and gives 75% higher throughput than static request routers.
• AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inference (Zhang et al., Sep 2026): Confirms that synchronous edge-cloud token verification collapses under mobile packet jitter. An asymmetric local state cache in place of lock-step verification gives a 3.52x throughput speedup, down to 50 Kbps WAN conditions.
• mzCache: On-Device LLM Memory Management under Multitasking (Yu et al., Sep 2026): Confirms that mobile OS app switching evicts LLM weights and KV caches from DRAM. Splitting model memory into shared GPU/CPU restoration buffers cuts post-eviction TTFT freezes by 2.1x to 5.5x.
• Efficient and Privacy-Aware Edge-Cloud Collaborative Inference for Large Language Models (Li et al., Jul 2026): Confirms that one approach cuts downlink payload bytes by 67.4% and per-token latency by 46.1%: sanitize PII tokens on the device first, then sync an authenticated KV cache with cloud containers.
• Chrome Built-in AI and Prompt API Documentation: Standard client-side on-device model execution via browser NPU APIs referenced in Section 03 and Section 05.
• Google Cloud Run Documentation: Stateless serverless container execution for cloud reasoning backends referenced in Section 03.
• Firebase App Check Documentation: Cryptographic client attestation securing cloud endpoints against unauthorized traffic referenced in Section 04.