Programming

Overview

You already know how to write software. This lesson is about the specific habits that stop working once a model is one of your dependencies.

A normal function call is deterministic: same inputs, same output, and failure arrives as an exception you catch. A model call is none of that. The same prompt gives different text, failure often looks like a perfectly formed wrong answer, and retrying is a legitimate strategy rather than an admission of defeat. Everything that feels strange about building on models traces back to that shift, so it is worth getting straight before you write any of it.

Theory

Errors are values, not exceptional events. A model call fails constantly — rate limits, timeouts, malformed output — and that is the normal path, not the exception. Return a result you can branch on rather than throwing from three layers down and hoping someone catches it.

Idempotency makes retries safe. If calling twice does the same thing as calling once, you can retry freely. If it does not, every retry is a potential duplicate charge or duplicate email. Design for it before you add the retry, not after.

Non-determinism changes what "correct" means. You cannot assert equality on a model's output. Correctness becomes a property — does it validate against the schema, does it score well against labelled examples — rather than an exact match. This is the single biggest mental shift in the course.

Partial success is a real outcome. Three of five documents embedded; the model answered but the tool call failed. Neither "worked" nor "failed" describes it, so your types need a third shape or you will silently drop the remainder.

Timeouts and budgets belong in the signature. How long may this take, how many tokens may it spend, how many retries may it make — these are inputs, not constants buried inside. Callers have different budgets, and the ones that do not say so get yours.

Visual Diagram

Picture two boxes side by side. The left box, "deterministic path," has one line in and one line out — call it with x, get f(x), every time, forever. The right box, "path with a model call," has one line in but a fan of several lines out: a successful, well-formed response; a successful but subtly-wrong response; a rate-limit error; a timeout; a malformed-JSON response that technically "succeeded" at the HTTP layer. Good AI-engineering code is the funnel on the right-hand side of that second box — the layer that takes all five of those possible outcomes and narrows them back down to one predictable, typed result the rest of your program can rely on: either a validated value, or an explicit, typed failure. That funnel is not an add-on; in AI product code it is most of the actual engineering.

Examples

• A retry wrapper with exponential backoff and jitter: the third call in a burst backs off longer than the first, and a small random jitter is added so that many clients retrying at once don't all slam the server in the same instant.
• An idempotency key threaded through a "generate and save" flow: a user clicks "regenerate" twice quickly on a slow connection; the second click reuses the same key as the first in-flight request, so the backend can detect and collapse the duplicate instead of writing two rows.
• A discriminated union return type for a function that can succeed, fail validation, or fail transport — three distinct outcomes a caller can exhaustively switch over instead of guessing what a caught exception might mean.
• A batch processor that reports partial results: uploading 100 documents for embedding, 6 fail due to size limits, and the UI shows "94 of 100 processed, 6 need attention" instead of either silently dropping the 6 or failing the whole batch.

Code

// A discriminated union makes failure a value your caller must handle,
// not an exception they might forget to catch.
type Result<T, E> = { ok: true; value: T } | { ok: false; error: E };

type CallError =
  | { kind: "timeout" }
  | { kind: "rate_limited"; retryAfterMs: number }
  | { kind: "invalid_response"; raw: string };

/**
 * Wraps an unreliable async call with a timeout, retries with exponential
 * backoff + jitter, and an idempotency key so a retried "duplicate" request
 * is provably safe to send twice.
 */
async function callWithRetry<T>(
  fn: (idempotencyKey: string) => Promise<T>,
  opts: { maxAttempts?: number; timeoutMs?: number } = {},
): Promise<Result<T, CallError>> {
  const maxAttempts = opts.maxAttempts ?? 3;
  const timeoutMs = opts.timeoutMs ?? 10_000;
  // One key for the whole logical operation — every attempt reuses it,
  // so the receiving system can de-duplicate retries safely.
  const idempotencyKey = crypto.randomUUID();

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), timeoutMs);

    try {
      const value = await fn(idempotencyKey);
      clearTimeout(timer);
      return { ok: true, value };
    } catch (err) {
      clearTimeout(timer);
      const isLastAttempt = attempt === maxAttempts;

      if (controller.signal.aborted) {
        if (isLastAttempt) return { ok: false, error: { kind: "timeout" } };
      } else if (isLastAttempt) {
        return {
          ok: false,
          error: { kind: "invalid_response", raw: String(err) },
        };
      }

      // Exponential backoff with jitter: base delay doubles each attempt,
      // plus up to 30% random jitter so concurrent retries don't sync up.
      const baseDelay = 250 * 2 ** (attempt - 1);
      const jitter = baseDelay * 0.3 * Math.random();
      await new Promise((r) => setTimeout(r, baseDelay + jitter));
    }
  }

  // Unreachable, but TypeScript needs an exhaustive return path.
  return { ok: false, error: { kind: "timeout" } };
}

The caller never sees a thrown exception for an expected failure mode — it gets a Result it must check, and the kind field on CallError lets it decide per-failure-type what to do (show "try again in a moment" for a rate limit, log and alert for a malformed response). That's the fundamental shift: failure became data.

Real Product Example

Stripe's payments API is the canonical example of idempotency done right, and it's worth studying even though it predates the LLM era, because the exact same failure mode — "did my request actually go through, or did the network just drop the response?" — shows up constantly in AI products (did that "generate and charge credits" call actually deduct credits once, or twice, because the client retried after a slow response?). Stripe requires an Idempotency-Key header on POST requests; if you send the same key twice, Stripe returns the stored result of the first request instead of performing the operation again, even if the second request arrives seconds or hours later. This turns "should I retry this uncertain request?" from a risky judgment call into a safe default: you can always retry, because the key guarantees the side effect happens at most once. Any AI product that debits credits, sends a notification, or writes a billable usage record per generation needs the same pattern — without it, a flaky network combined with an eager retry loop will double-charge real users in production, not in theory.

Common Mistakes

Senior Tips

Exercises

Mini Project

Retry-safe classifier wrapper. Using the callWithRetry function from the Code section as a starting point:

1. Write a fake unreliableClassify(text: string, idempotencyKey: string) function that randomly throws about 30% of the time (simulate rate limits and timeouts with different thrown values).
2. Wire it through callWithRetry and confirm, by running it many times, that it eventually succeeds far more often than a single unwrapped call would.
3. Add a discriminated CallError case your fake function can throw, and handle it distinctly in the caller (don't just log it — show a different message per error kind).
4. Write one property-based test-style assertion (not exact-string) confirming the final successful result always matches your expected shape.

This mini project is deliberately small — the point is the pattern, not the amount of code. You'll reuse this exact wrapper shape, nearly unmodified, the first time you wrap a real model call in Week 3.

Resources

• MDN — Control flow and error handling: the baseline mechanics of try/catch/throw this lesson assumes you already have and builds on top of.
• TypeScript Handbook: the canonical reference for the type system used to model Result and discriminated unions throughout this path's code samples.
• Stripe — Idempotent requests: the real-world reference implementation of the idempotency-key pattern discussed in Real Product Example, worth reading in full.