Scaling

Overview

The first version works fine. Then the same question gets asked a thousand times, all of it goes straight to the provider, and you get a rate limit and a bill on the same afternoon.

Scaling an AI product is mostly about not making the call. Cache what has been answered, batch what can wait, queue what arrives in bursts, and fail over when a provider has a bad hour. This lesson is those four layers, in the order a request should meet them.

Theory

The cheapest model call is the one you never make. Exact-match caching keys on the prompt and returns the stored completion — trivial to build and it catches more traffic than you expect, because real users ask the same things. Semantic caching matches near-duplicates by embedding, which catches more and risks returning an answer to a subtly different question. Start exact, measure, then decide.

Batching trades latency for throughput. Embedding a thousand documents one call at a time is slow and rate-limit-prone. One call with a hundred inputs is dramatically faster per item. It is the right move for background work and the wrong move for anything a user is waiting on.

Queueing smooths bursts. Traffic arrives in spikes; provider limits are steady. A queue absorbs the difference and turns a wall of 429s into a slightly longer wait. It also gives you one place to add priority when you need it.

Backoff needs jitter. Exponential backoff alone means every client that failed together retries together, recreating the spike that caused the failure. Randomising the delay spreads them out — it is one line, and it is the difference between recovery and a thundering herd.

Visual Diagram

Trace a request left to right. It first checks the cache — exact-match, then semantic if the first misses — and returns immediately on a hit. On a miss, if the work is long-running, it's pushed onto a queue rather than blocking the caller. When a worker picks it up and calls the model, the call is wrapped in retry logic: on a rate-limit error, it waits (exponentially, with jitter) and retries against the same provider up to a limit; past that limit, it fails over to a secondary provider rather than surfacing an error to the user. Only after all of that does a result get written back and, on the way out, written into the cache for the next request that looks like it.

Examples

Concrete scaling work an AI engineer does, roughly in the order it becomes necessary as traffic grows:

• Exact-match caching a classification endpoint that receives the same handful of common support questions repeatedly, cutting call volume by half or more with a one-line hash-and-lookup.
• Adding jittered exponential backoff to every model call, so a burst of traffic that trips a rate limit degrades gracefully (slightly slower) instead of catastrophically (a wave of user-facing errors).
• Moving a "summarize this 40-page document" feature off the request path and onto a queue with a worker pool, so the browser gets an immediate "we're working on it" instead of a 90-second hanging request that times out.
• Configuring a secondary provider for a chat feature, so a primary-provider outage degrades quality temporarily instead of taking the feature down entirely.
• Batching a nightly enrichment job (tagging every new record from the day) through a provider's batch API instead of one-at-a-time calls, cutting cost on work nobody is waiting on synchronously.

Code

A resilient model-call wrapper: exact-match cache first, jittered exponential backoff on rate-limit errors, and failover to a secondary provider once retries are exhausted. This is the shape that sits underneath every model call in a system that has to survive real traffic, not just a demo.

import { generateText, type LanguageModel } from "ai";
import { createHash } from "node:crypto";

interface CallOptions {
  primary: LanguageModel;
  fallback: LanguageModel;
  system?: string;
  prompt: string;
  maxRetries?: number;
}

const cache = new Map<string, { text: string; expiresAt: number }>();
const CACHE_TTL_MS = 5 * 60 * 1000;

function cacheKey(system: string | undefined, prompt: string): string {
  return createHash("sha256").update(`${system ?? ""}::${prompt}`).digest("hex");
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

/** Full jitter: wait a random duration between 0 and the exponential cap.
 * Prevents every retrying client from synchronizing on the same schedule,
 * which is what causes a retry storm after a shared rate limit trips. */
function backoffDelayMs(attempt: number): number {
  const cap = 30_000;
  const base = 500;
  const exponential = Math.min(cap, base * 2 ** attempt);
  return Math.random() * exponential;
}

function isRateLimited(error: unknown): boolean {
  const status = (error as { status?: number; statusCode?: number })?.status
    ?? (error as { statusCode?: number })?.statusCode;
  return status === 429 || status === 529;
}

async function callWithRetry(model: LanguageModel, system: string | undefined, prompt: string, maxRetries: number) {
  let lastError: unknown;
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await generateText({ model, system, prompt });
    } catch (error) {
      lastError = error;
      if (!isRateLimited(error) || attempt === maxRetries) throw error;
      await sleep(backoffDelayMs(attempt));
    }
  }
  throw lastError;
}

/**
 * Cache-first, retry-with-jitter, failover model call. This is the boring,
 * necessary layer that sits between "a prompt" and "a reliable feature" —
 * every technique from the Theory section shows up here except batching
 * and queueing, which operate one level up, around calls like this one.
 */
export async function resilientGenerate(options: CallOptions): Promise<{ text: string; cached: boolean }> {
  const key = cacheKey(options.system, options.prompt);
  const hit = cache.get(key);
  if (hit && hit.expiresAt > Date.now()) {
    return { text: hit.text, cached: true };
  }

  const maxRetries = options.maxRetries ?? 3;
  let text: string;
  try {
    const result = await callWithRetry(options.primary, options.system, options.prompt, maxRetries);
    text = result.text;
  } catch {
    // Primary exhausted its retries — fail over rather than surface an
    // error, accepting a possibly lower-quality answer over an outage.
    const result = await callWithRetry(options.fallback, options.system, options.prompt, 1);
    text = result.text;
  }

  cache.set(key, { text, expiresAt: Date.now() + CACHE_TTL_MS });
  return { text, cached: false };
}

/** A minimal concurrency-limited queue for long-running jobs, so 500
 * incoming documents don't become 500 simultaneous provider requests. */
export function createQueue(concurrency: number) {
  const tasks: (() => Promise<void>)[] = [];
  let active = 0;

  function runNext() {
    if (active >= concurrency || tasks.length === 0) return;
    const task = tasks.shift()!;
    active++;
    task().finally(() => {
      active--;
      runNext();
    });
  }

  return {
    push<T>(job: () => Promise<T>): Promise<T> {
      return new Promise((resolve, reject) => {
        tasks.push(() => job().then(resolve, reject));
        runNext();
      });
    },
  };
}

The failover path is deliberately narrow: it only triggers after the primary's own retries are exhausted, and it uses a lower retry budget on the fallback rather than repeating the same exponential schedule twice — the goal is "stay up, degraded" in seconds, not "eventually succeed" in minutes.

Real Product Example

Vercel's AI Gateway (the same category of provider-abstraction layer this platform runs its own generation calls through) is a concrete, shipped example of this entire lesson as a product. It sits between an application and multiple model providers, and its core mechanism is exactly the diagram above: requests can be routed with a configured fallback list, so if a primary provider returns a rate-limit or server error, the gateway retries against the next provider in the list automatically, without the application code changing. It also exposes per-provider, per-model usage and cost — the observability half of scaling — so a team can see which provider is absorbing traffic during a failover event instead of discovering it after the invoice arrives. The lesson generalizes past any one vendor: the moment your product depends on an external API's availability, "one provider, no fallback, no backoff" is a single point of failure you chose, not one that was forced on you.

Common Mistakes

Senior Tips

Exercises

Mini Project

Load-harden the shipped app. Take the app you instrumented with monitoring on Day 25 and make it survive load it wasn't built for.

1. Add exact-match (or semantic, if your task fits) caching to your highest-volume model call.
2. Wrap every model call in retry-with-jitter, capped at a sane number of attempts.
3. Configure a fallback model and trigger failover manually (point the primary at an invalid model name or API key temporarily) to confirm the fallback actually engages and the app stays up.
4. If you have any AI task that takes more than a few seconds, move it behind a queue with a concurrency limit, and confirm the request-handling code no longer blocks on it.
5. Re-run your monitoring dashboard from Day 25 during a simulated burst of concurrent requests and check that cost, latency, and error rate all stay within limits you'd be comfortable shipping.

Resources

• Anthropic — Rate Limits: how usage tiers, requests-per-minute, and token-per-minute limits work, plus the response headers that tell you how close you are to one.
• Anthropic — Prompt Caching: provider-side caching of repeated prompt prefixes (system prompts, long context) — a complementary technique to the application-level caching built in this lesson's Code section.
• AWS Architecture Blog — Exponential Backoff and Jitter: the original, still-canonical explanation of why jitter matters and a comparison of full, equal, and decorrelated jitter strategies.