Monitoring

Overview

An AI feature fails differently from the rest of your app. Nothing throws, latency looks fine, the dashboard is green — and the answers have been quietly getting worse for two weeks.

Ordinary monitoring cannot see that, because nothing is technically broken. This lesson covers what you actually have to record: a trace id that ties one user action to every call underneath it, tokens and cost per request, latency as percentiles rather than averages, and the outcome signal that tells you whether answers were any good.

Theory

A trace ties one user action to everything it caused. Generate an id at the edge and attach it to the retrieval call, the generation, and every tool call underneath. Without it you have a pile of unrelated log lines and no way to answer "what happened on that request."

Track cost and tokens per request, not per month. A monthly bill tells you that something got expensive. Prompt and completion tokens tagged per trace tell you which feature, which is the only version you can act on.

Latency is percentiles, never averages. An average hides the tail, and the tail is what users complain about. Watch p95 and p99. For streaming, measure time-to-first-token separately from total duration — they are different experiences and they fail for different reasons.

Quality drifts without any deploy. Providers update models underneath you. Log outcomes — did the user accept the answer, retry, or leave — and watch that number over time. Latency dashboards look perfectly healthy while answers quietly get worse.

Log prompts carefully. They contain whatever your users typed, which means PII. Redact before storage, set a retention window, and keep the logs behind the same access controls as the rest of your user data.

Visual Diagram

Read it as a single request fanning out into a tree. The root is the trace, created the moment a user action starts. Each child span is one unit of work inside it — a retrieval call, a model generation, a tool invocation — and each span independently records its own start time, end time, token counts (if applicable), and status. The dashboard you actually look at every day is built by aggregating thousands of these trees: p95 latency is computed across all root spans, cost is summed across all token-bearing spans, and an eval score is computed by replaying sampled traces against your scoring function. No individual span tells the story; the aggregate does.

Examples

Concrete monitoring signals an AI engineer wires up, roughly in the order they get built:

• A per-request log row with trace id, model, redacted prompt/completion, input/output tokens, cost, latency, and status — the raw material everything else is computed from.
• A cost dashboard grouped by feature and by user, so a spike is attributable ("the new summarization feature costs 4x what triage costs per call") instead of a mystery line going up.
• A latency percentile chart (p50/p95/p99) segmented by model, so a model swap's latency impact is visible before users complain, not after.
• A nightly eval run against a frozen test set, graphed over time, so a score that drops after a prompt change or a provider's silent model update shows up as a visible step in a chart, not a vague feeling that "answers seem worse lately."
• Alerting on error rate and cost-per-hour thresholds, the same way you'd alert on any other production service — an LLM integration that starts erroring or burning 10x normal spend is an incident, not a curiosity.

Code

A tracing wrapper around a single model call: it assigns a trace id, redacts before logging, computes cost from token usage, and captures latency and status whether the call succeeds or throws. It also exposes a percentile function, because "average latency" is close to useless for deciding whether users are having a bad time.

import { generateText, type LanguageModel } from "ai";
import { randomUUID } from "node:crypto";

interface GenerationLog {
  id: string;
  traceId: string;
  model: string;
  promptRedacted: string;
  completionRedacted: string;
  inputTokens: number;
  outputTokens: number;
  costUsd: number;
  latencyMs: number;
  status: "ok" | "error";
  errorMessage?: string;
  createdAt: string;
}

// Per-million-token pricing kept as data, not hardcoded per call site, so a
// price change or provider swap is a one-line diff instead of a code review
// across every call.
const PRICING_PER_MILLION: Record<string, { input: number; output: number }> = {
  "gpt-4o-mini": { input: 0.15, output: 0.6 },
  "claude-sonnet-5": { input: 3, output: 15 },
};

// Deliberately not a general-purpose PII scrubber — that needs a dedicated
// model or library to be trustworthy. This is the minimum bar: strip the
// shapes that are cheap to catch and expensive to leak before anything
// touches a log line or a third-party observability sink.
function redact(text: string): string {
  return text
    .replace(/[\w.+-]+@[\w-]+\.[\w.-]+/g, "[email]")
    .replace(/\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b/g, "[phone]")
    .replace(/\b(?:\d[ -]*?){13,16}\b/g, "[card]");
}

function costUsd(model: string, inputTokens: number, outputTokens: number): number {
  const price = PRICING_PER_MILLION[model];
  if (!price) return 0; // unknown model: log zero rather than crash the request
  return (inputTokens / 1_000_000) * price.input + (outputTokens / 1_000_000) * price.output;
}

const sink: GenerationLog[] = []; // stand-in for a real database insert

/**
 * Wraps one model call with end-to-end tracing: a stable trace id the
 * caller can correlate back to a user action, redacted prompt/completion
 * logging, token-based cost, and latency — captured whether the call
 * succeeds or throws.
 */
export async function tracedGenerate(params: {
  model: LanguageModel;
  modelId: string;
  system?: string;
  prompt: string;
  traceId?: string;
}) {
  const traceId = params.traceId ?? randomUUID();
  const startedAt = performance.now();

  try {
    const result = await generateText({ model: params.model, system: params.system, prompt: params.prompt });
    const latencyMs = performance.now() - startedAt;
    const inputTokens = result.usage.inputTokens ?? 0;
    const outputTokens = result.usage.outputTokens ?? 0;

    sink.push({
      id: randomUUID(),
      traceId,
      model: params.modelId,
      promptRedacted: redact(params.prompt),
      completionRedacted: redact(result.text),
      inputTokens,
      outputTokens,
      costUsd: costUsd(params.modelId, inputTokens, outputTokens),
      latencyMs,
      status: "ok",
      createdAt: new Date().toISOString(),
    });

    return { text: result.text, traceId, latencyMs };
  } catch (error) {
    sink.push({
      id: randomUUID(),
      traceId,
      model: params.modelId,
      promptRedacted: redact(params.prompt),
      completionRedacted: "",
      inputTokens: 0,
      outputTokens: 0,
      costUsd: 0,
      latencyMs: performance.now() - startedAt,
      status: "error",
      errorMessage: error instanceof Error ? error.message : String(error),
      createdAt: new Date().toISOString(),
    });
    throw error;
  }
}

/** p50/p95/p99 latency over the logged window — the numbers that actually
 * describe user experience, since a mean hides the slow tail. */
export function latencyPercentiles(logs: GenerationLog[] = sink) {
  const sorted = logs.map((l) => l.latencyMs).sort((a, b) => a - b);
  const pick = (p: number) => sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * p))] ?? 0;
  return { p50: pick(0.5), p95: pick(0.95), p99: pick(0.99), count: sorted.length };
}

Every field here earns its place: traceId lets you reconstruct one user's bad experience across multiple spans, promptRedacted/completionRedacted keep debugging possible without keeping raw PII around, and status plus errorMessage mean a failed call is still a data point instead of a silent gap in your dashboard.

Real Product Example

Langfuse is a clean example because its whole product is this lesson, shipped. Every model call an integrated application makes is recorded as a "generation" span nested inside a "trace" — the same shape as the diagram above — capturing the input, output, token usage, latency, and cost automatically from the provider's response headers. Teams attach evaluation scores to traces after the fact (a human rating, an LLM-as-judge score, a user's thumbs up/down), which turns the trace store into the same nightly-eval mechanism described in the Theory section: you can graph average eval score over time, per model version, and see the exact day a provider's model update or a prompt edit moved the number. The mechanism that makes it useful isn't the UI — it's the discipline of tagging every span with enough metadata (which prompt version, which model, which feature) that a regression can be sliced down to a cause instead of just observed as "things got worse sometime last week."

Common Mistakes

Senior Tips

Exercises

Mini Project

Instrument the shipped app. Take the ai-saas project you deployed on Day 23 and add real observability to it.

1. Wrap every model call in the app with a tracing function like tracedGenerate — trace id, redacted logging, token counts, cost, latency, status.
2. Build a small dashboard (even a single page rendering a table) showing: total cost today, p50/p95 latency over the last 100 requests, and error rate.
3. Wire your Day 13-18 eval set to run against the live prompt on a schedule (a cron job, a GitHub Action, or even a manually-run script counts for this exercise) and store the score with a timestamp.
4. Deliberately break something — swap in a worse prompt for five minutes — and confirm your dashboard and eval score both show it. If neither does, the instrumentation isn't done yet.

Resources

• AI SDK Core — Telemetry: how to enable OpenTelemetry-based tracing for AI SDK calls, including privacy controls for filtering sensitive data before it's exported.
• Langfuse — Documentation: an open-source LLM observability platform whose trace and generation model is the concrete shape used throughout this lesson.
• Google SRE Book — Monitoring Distributed Systems: the "four golden signals" (latency, traffic, errors, saturation) that underlie monitoring any production system, AI or not.
• Hamel Husain — Your AI Product Needs Evals: a field report on why evaluation infrastructure, run continuously rather than once before ship, is what separates products that survive contact with real users from ones that don't.