Node.js

Overview

A model call can run for thirty seconds. Almost everything about serving that well is a Node question, not an AI one.

This lesson covers the runtime decisions underneath a streaming endpoint: what a stream really gives you that an array of chunks does not, how Edge and Node trade against each other when a response takes half a minute, why a timeout is the only thing that distinguishes "still generating" from "hung", and which of your own work will block the event loop.

Theory

A stream is chunks with backpressure built in. Not just "data that arrives async" — the consumer can signal that it is not ready, and the producer waits. That signal is the whole point, and it is what an array-of-chunks approach throws away.

Edge and Node trade differently, and streaming sits on the line. Edge gives you fast cold starts and global placement, with capped duration and no Node built-ins. Node gives you time, memory, and the full ecosystem, with slower starts. A long generation is exactly the case where that limit bites.

"Still generating" and "hung" look identical from outside. Both are an open connection producing nothing right now. Only a timeout tells them apart. Pick a budget from real p99 latency, not a round number, and make the message say which one happened.

Model calls do not block the thread, but your own work might. Awaiting a provider is I/O — Node handles thousands concurrently. Parsing a huge response or embedding a document in-process is CPU, and that does block the single thread, stalling every other request on the box.

Visual Diagram

Draw it as a decision at the top — "does this route need Node-only APIs or heavy CPU work?" — branching to either an Edge runtime or a Node.js runtime box. Both boxes feed into the same shape below them: a pipeline of provider stream → transform → response stream, wrapped in an outer box labeled "timeout budget," with a small clock icon showing that the whole pipeline is racing against a maximum duration that gets enforced regardless of which runtime is doing the work. The point of the diagram is that runtime choice and timeout enforcement are two independent decisions layered around the same core streaming pipeline — you pick a runtime once per route, but you enforce a timeout on every single request.

Examples

• An edge-runtime API route that proxies streamText output straight to the client, doing nothing else CPU- intensive, benefiting from edge's low latency and global distribution for a pure pass-through workload.
• A Node.js-runtime route that parses an uploaded PDF (a real Node dependency, CPU-bound) before sending its text to a model, correctly placed on Node rather than edge because of the native dependency and CPU cost.
• A Transform stream that rewrites profanity or PII out of a model's output chunk-by-chunk as it flows from provider to client, without ever buffering the full response.
• A route wrapped in AbortSignal.timeout(30_000) that cancels the upstream provider fetch and returns a clean "generation timed out, please try a shorter prompt" response instead of hanging until the platform's own hard limit kills the function ungracefully.

Code

// A Node.js route handler (works the same shape in an edge runtime, minus
// any Node-only APIs) that streams a model response with an enforced
// maximum duration and clean teardown on timeout.
import { streamText } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";

const provider = createOpenAICompatible({
  name: "example-gateway",
  baseURL: "https://ai.gateway.lovable.dev/v1",
  headers: { "Lovable-API-Key": process.env.LOVABLE_API_KEY ?? "" },
});

const MAX_GENERATION_MS = 30_000;

export async function POST(request: Request): Promise<Response> {
  const { prompt } = (await request.json()) as { prompt: string };

  // AbortSignal.timeout gives us a signal that fires automatically —
  // no manual setTimeout/clearTimeout bookkeeping to get wrong.
  const timeoutSignal = AbortSignal.timeout(MAX_GENERATION_MS);
  // Also respect the client disconnecting (e.g. closing the tab mid-stream).
  const combinedSignal = AbortSignal.any([timeoutSignal, request.signal]);

  const result = streamText({
    model: provider("gpt-4o-mini"),
    prompt,
    abortSignal: combinedSignal,
  });

  // Adapt the model's text stream into an HTTP-level ReadableStream,
  // so bytes are forwarded to the client as they're generated — never
  // buffered in full server-side memory before the response starts.
  const encoder = new TextEncoder();
  const body = new ReadableStream<Uint8Array>({
    async start(controller) {
      try {
        for await (const chunk of result.textStream) {
          controller.enqueue(encoder.encode(chunk));
        }
        controller.close();
      } catch (err) {
        if (combinedSignal.aborted) {
          // Timed out or client disconnected — end the stream quietly,
          // there's no client left to receive an error payload.
          controller.close();
        } else {
          controller.error(err);
        }
      }
    },
  });

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

Notice the two signals combined with AbortSignal.any: one enforces the business-level time budget (don't let a single generation run longer than 30 seconds regardless of cause), the other propagates the client's own disconnect (request.signal, provided by the platform) so an abandoned tab stops billing tokens immediately instead of running to completion for no one.

Real Product Example

Vercel's own AI-oriented function tooling is a direct, shipped example of these tradeoffs being made explicit as product configuration rather than left implicit. Vercel Functions let a route declare its runtime (Node.js or Edge) and its maximum duration per route, and specifically call out streaming as a supported capability of both runtimes with the caveat that a function still can't stream indefinitely — there's always a configured maximum duration backing it, exactly the "timeout budget" concept from this lesson. Teams building AI features on Vercel routinely put the thin "forward the model's stream to the client" routes on Edge for the latency and global- distribution benefit, and keep anything doing real CPU work (parsing uploads, running local models, heavier orchestration with Node-only packages) on the Node.js runtime — the same split described in Theory above, just made into an explicit per-route configuration choice rather than a hidden default.

Common Mistakes

Senior Tips

Exercises

Mini Project

Streaming generation endpoint with a hard timeout. Build a small local Node server (plain http module or a minimal framework, your choice):

1. Write a fake fakeGenerate(prompt: string): AsyncGenerator<string> that yields word-sized chunks every ~200ms, same shape as the JavaScript lesson's mini project, but make it occasionally (randomly) "hang" by never yielding again after a few chunks — simulating a stuck generation.
2. Wire it into an HTTP route that streams the chunks to the response as they're produced, using a real ReadableStream/response pipeline (not buffering into a string first).
3. Add an AbortSignal.timeout() of 3 seconds. Confirm that when fakeGenerate hangs, the route still closes the response cleanly at the 3-second mark instead of hanging forever.
4. Log time-to-first-chunk and total duration for each request, and confirm the numbers look right for both a normal run and a timed-out one.

Resources

• Node.js — Stream: the authoritative reference for Readable, Writable, and backpressure mechanics referenced throughout Theory.
• Node.js — Worker threads: how to move genuinely CPU-bound work (large-document parsing, local tokenization) off the main thread so it doesn't stall concurrent requests.
• Vercel — Function runtimes: the concrete edge-vs-Node.js runtime tradeoffs and per-route configuration this lesson's Real Product Example is based on.
• Vercel AI SDK — Generating text (streamText): the streamText API used in this lesson's Code sample, including its abort-signal support.