APIs

Overview

Calling a model API is one fetch. Calling it in a way that survives a bad network, a rate limit, and an impatient user who hits the button twice is the actual job.

This lesson is the request path: why these APIs stream, how an idempotency key stops a retry becoming a duplicate charge, which failures are worth retrying and which are not, and why a timeout you set yourself beats a generic 504 from the platform.

Theory

Streaming exists because generation is slow and incremental. A model produces tokens one at a time, so waiting for the whole reply before showing anything wastes seconds you could have spent showing progress. Streaming does not make it faster — it makes the wait visible, which is most of what "fast" means to a user.

Idempotency stops a retry becoming a duplicate. The client sends a key with the request; the server records it alongside the result. A retry with the same key returns the stored result instead of calling the model again. One key per user intent, not per attempt — regenerating it on retry defeats the whole thing.

Rate limits are two-sided. The provider limits you, and you should limit your users. Without the second half, one script hammering your endpoint spends your entire quota and everyone else gets a 429 they did nothing to earn.

Retry only what is worth retrying. A 429 or a 503 is transient — back off exponentially, add jitter so your clients do not all return in lockstep, and cap the attempts. A 400 or a 401 is permanent; retrying it burns time and buries the real error.

Timeouts protect everything downstream. A model call with no timeout can hold a connection until the platform kills it, and you get a generic 504 with nothing to debug. Set your own budget, shorter than the platform's, so the failure is yours and it says something useful.

Visual Diagram

Read it as a pipeline: the client sends a request carrying an Idempotency-Key. Your API first checks whether that key has already been processed — if so, it replays the stored result and skips the model entirely. If not, it calls the model provider wrapped in a retry-with-backoff loop, which only retries the failure types worth retrying. Once a response starts coming back, it's streamed to the client as SSE chunks rather than buffered and sent as one block, and the final result is persisted against the idempotency key before the response completes — so any retry of the exact same request short-circuits to the saved answer instead of calling the model again.

Examples

• A chat completion endpoint that streams tokens via SSE and retries a 503 from the provider up to three times before falling back to an error message.
• A "regenerate this image" button that sends an Idempotency-Key so a flaky connection retry doesn't produce (and bill for) two images from one click.
• A batch summarization job processing a thousand documents overnight, where each document's request is rate-limited to stay under the provider's requests-per-minute ceiling instead of firing all thousand at once.
• A webhook receiver for a provider's async batch API, which itself needs to be idempotent, since webhook delivery systems commonly retry on their own.

Code

interface RetryOptions {
  maxAttempts?: number;
  baseDelayMs?: number;
}

/**
 * Retries a model call with exponential backoff and jitter. A model
 * provider is just another third-party API — it rate-limits, it has bad
 * minutes, and a blind retry-forever loop makes an outage worse for
 * everyone sharing that provider capacity. This retries only transient
 * failures and gives up immediately on the rest.
 */
async function withRetry<T>(
  fn: () => Promise<T>,
  { maxAttempts = 3, baseDelayMs = 500 }: RetryOptions = {},
): Promise<T> {
  let lastError: unknown;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      const status = (err as { status?: number }).status;

      // Only retry transient failure classes — a bad request or bad API
      // key will fail identically every time, so retrying just delays the
      // inevitable error.
      const retryable = status === 429 || status === 500 || status === 503;
      if (!retryable || attempt === maxAttempts) break;

      const jitter = Math.random() * 250;
      const delay = baseDelayMs * 2 ** (attempt - 1) + jitter;
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }

  throw lastError;
}

import { generateText } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { createClient } from "@supabase/supabase-js";

const provider = createOpenAICompatible({
  name: "lovable-ai-gateway",
  baseURL: "https://ai.gateway.lovable.dev/v1",
  headers: { "Lovable-API-Key": process.env.LOVABLE_API_KEY ?? "" },
});

const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_ROLE_KEY!);

/**
 * POST /api/complete. The client sends an Idempotency-Key header — a UUID
 * it generates once per user action — so that a retried request (e.g.
 * after a flaky mobile connection times out and the client resends) never
 * triggers a second, separately billed model call.
 */
export async function POST(req: Request) {
  const idempotencyKey = req.headers.get("Idempotency-Key");
  if (!idempotencyKey) {
    return new Response("Idempotency-Key header is required", { status: 400 });
  }

  const { data: existing } = await supabase
    .from("completions")
    .select("result")
    .eq("idempotency_key", idempotencyKey)
    .maybeSingle();

  if (existing) {
    // Same key seen before — replay the saved result instead of calling
    // the model again.
    return Response.json({ result: existing.result, replayed: true });
  }

  const { prompt } = await req.json();

  const { text } = await withRetry(() =>
    generateText({ model: provider("gpt-4o-mini"), prompt }),
  );

  await supabase.from("completions").insert({ idempotency_key: idempotencyKey, result: text });

  return Response.json({ result: text, replayed: false });
}

Real Product Example

Stripe's API is the industry-reference implementation of idempotency, and the exact mechanism this lesson's Idempotency-Key header borrows from: every POST request can carry an Idempotency-Key, Stripe stores the resulting status code and body against that key, and any retry with the same key — whether from a network timeout, a client bug, or a user double-clicking "Pay" — returns the original result instead of charging a card twice. Stripe prunes keys after 24 hours, so retries are safe within the window that actually matters (a client recovering from a dropped connection) without the server needing to remember every request forever. AI products adopt the identical pattern for the identical reason: a model call, like a charge, is expensive and has a side effect (tokens billed, a row written) that must not happen twice because a client retried.

Common Mistakes

Senior Tips

Exercises

Mini Project

Resilient completion endpoint. Combine every piece from this lesson into one endpoint.

1. Build POST /api/complete requiring an Idempotency-Key header, returning 400 if it's missing.
2. Check for an existing result under that key before calling the model; return the saved result if found.
3. Wrap the model call in withRetry, retrying only 429/500/503.
4. Persist the result keyed by the idempotency key once the call succeeds.
5. Manually test: send the same request twice with the same key and confirm the second response is marked replayed: true and no second model call was made (check your provider's usage dashboard or a log line).

Resources

• MDN — Server-sent events: the protocol underlying most model-provider streaming and the EventSource API for consuming it directly.
• Stripe — Idempotent requests: the reference implementation this lesson's Idempotency-Key pattern is modeled on, including key retention and generation guidance.
• Anthropic — Streaming Messages: a concrete, production SSE event stream (message_start, content_block_delta, message_stop) from a real model provider.