Memory

Overview

Ask a chatbot to remember your name, come back tomorrow, and it has no idea who you are. People call this a memory problem. It isn't — the model has no memory to fail, and never did.

Every request is stateless. The model sees exactly what you send it and nothing else. The "memory" in any AI product is engineering you build: deciding what to resend, what to store, what to look up, and what to throw away. This lesson is about that machinery — the context window, the store behind it, and the summarization step that moves information between them.

Theory

Two different things share the name. Short-term memory is the context window: rebuilt on every request, bounded by tokens, billed every time, and gone when the request ends. Long-term memory is a store you own — a database that outlives the conversation. Confusing the two is why people are surprised the model "forgot."

The model remembers nothing. Every request is stateless. The illusion of memory is you, resending history. Once you internalise that, the design question stops being "how do I make it remember" and becomes "what do I resend, and what do I look up."

Persist what a future turn will need. Decisions, stated preferences, facts about the user, resolved outcomes. Discard the rest: pleasantries, superseded drafts, intermediate reasoning that led somewhere you already recorded. Storing everything is the same as storing nothing, because retrieval gets worse as the pile grows.

Summarization is the bridge. When history outgrows the window, compress the oldest turns into a summary, write it to the store, and carry the summary forward instead of the raw messages. Do it on a schedule you control — waiting until you hit the limit means it happens mid-conversation, at the worst moment.

Visual Diagram

Picture two boxes. The left box is small and fixed-size: the context window, holding the system prompt, a handful of recent turns verbatim, and a compact summary of everything older. The right box is large and grows over time without a size limit: a database or vector index holding the full history, every fact ever extracted, everything the rolling summary compressed away. An arrow labeled "summarize / extract" runs from old content leaving the left box into the right box. A second arrow labeled "retrieve" runs back from the right box into the left box — only the small slice relevant to the current turn, not the whole thing. The model only ever sees the left box's current contents.

Examples

• A customer support bot keeps the last 6 messages of the live chat verbatim, plus a rolling summary of the conversation so far ("customer is disputing a duplicate charge on order #4821, already confirmed their email"), plus a retrieved snippet of that customer's last three support tickets pulled from a database by customer ID — not their entire ticket history.
• A coding assistant keeps the current file and the last few edits verbatim, and instead of remembering every file it has ever touched in the project, re-reads a file from disk when it needs it — the filesystem is the long-term memory, and "memory" here mostly means knowing which files matter right now.
• A personal assistant app extracts structured facts ("prefers morning meetings," "allergic to peanuts") into a user-profile table rather than trying to summarize a growing conversation transcript, because those facts need to survive verbatim and correctly for months, not approximately survive a few rounds of re-summarization.

Code

A rolling-summary compaction strategy: keep recent turns verbatim, fold older turns into a summary once the raw transcript crosses a size threshold, and always feed the existing summary back in so repeated compaction doesn't silently drop what a previous summary already captured.

import { generateText } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";

const provider = createOpenAICompatible({
  name: "example-gateway",
  baseURL: "https://ai.gateway.lovable.dev/v1",
  headers: { "Lovable-API-Key": process.env.LOVABLE_API_KEY ?? "" },
});

interface Turn {
  role: "user" | "assistant";
  content: string;
}

interface ConversationMemory {
  summary: string; // long-term: compacted history
  recent: Turn[];  // short-term: verbatim
}

const KEEP_RAW_TURNS = 8;           // how many recent turns stay verbatim
const SUMMARY_TRIGGER_CHARS = 6000; // rough proxy for "getting close to the token budget"

/**
 * Appends a turn and compacts older history into `summary` once the raw
 * transcript grows past a threshold. This is what stands between a
 * long-running conversation and either silent truncation (the provider
 * drops old messages) or a blown context window (and the bill that comes
 * with resending it every turn).
 */
export async function appendTurn(
  memory: ConversationMemory,
  turn: Turn,
): Promise<ConversationMemory> {
  const recent = [...memory.recent, turn];
  const rawChars = recent.reduce((n, t) => n + t.content.length, 0);

  if (recent.length <= KEEP_RAW_TURNS && rawChars < SUMMARY_TRIGGER_CHARS) {
    return { ...memory, recent };
  }

  const toCompact = recent.slice(0, -KEEP_RAW_TURNS);
  const stillRaw = recent.slice(-KEEP_RAW_TURNS);

  // Fold the EXISTING summary back in as input, not just the new raw turns —
  // otherwise each compaction pass only "remembers" the last batch and
  // silently drops whatever the previous summary had already captured.
  const { text: newSummary } = await generateText({
    model: provider("gpt-4o-mini"),
    system:
      "Summarize this conversation so far in under 150 words. Preserve concrete facts " +
      "(IDs, amounts, decisions made) and drop small talk and retries that didn't change the outcome.",
    prompt:
      `Existing summary: ${memory.summary || "(none)"}\n\n` +
      toCompact.map((t) => `${t.role}: ${t.content}`).join("\n"),
  });

  return { summary: newSummary, recent: stillRaw };
}

/** Converts memory back into the messages array the model actually sees. */
export function toPromptMessages(memory: ConversationMemory) {
  const summaryMessage = memory.summary
    ? [{ role: "system" as const, content: `Earlier context: ${memory.summary}` }]
    : [];
  return [...summaryMessage, ...memory.recent];
}

Notice what makes this compaction rather than truncation: nothing is silently dropped without first passing through the summarizer, the existing summary is always an input to the next one so information doesn't leak away over repeated rounds, and the trigger is based on actual content size rather than an arbitrary turn count, so one very long message can still force compaction early. In a real app, ConversationMemory gets persisted (a conversations row in Postgres, for example — this platform already runs on Supabase) rather than living only in a request-scoped variable, which is what makes it survive between separate API calls in the first place.

Real Product Example

Anthropic's memory tool (generally available on the Claude API) is a clean, documented example of long-term memory implemented as files rather than a growing prompt. Claude is given a memory tool backed by a directory your application controls; before starting a task, Claude checks /memories for earlier progress, and as it works it writes what it learns to files there rather than trying to hold everything in the active context window. On a later call — even a fresh session with no shared conversation history — Claude reads those files back and continues, achieving persistence across sessions without the context window ever growing without bound. Anthropic's own guidance pairs this with compaction, a related server-side feature that automatically summarizes older parts of a long conversation once it approaches the context limit: memory handles what must survive verbatim across sessions, compaction handles keeping a single long session's active context small. The two together are exactly this lesson's split — a bounded short-term window, and a long-term store you deliberately write to and read from — implemented as a real, shipped feature rather than an app-specific pattern.

Common Mistakes

Senior Tips

Exercises

Mini Project

Compaction retrofit. If any earlier project in this path (the Day 12 chatbot is a good candidate) keeps conversation history by simply appending every message, spend 30 minutes retrofitting it with the pattern from this lesson.

1. Instrument the existing code to log the character (or token, if you have a counting utility) size of the messages array actually sent on each call.
2. Have a conversation long enough to push that size past a threshold you pick — 20+ turns, or several long messages.
3. Add the rolling-summary compaction from the Code section: recent turns verbatim, older turns folded into a regenerated summary.
4. Compare the logged size before and after adding compaction, at the same point in a similarly long conversation. Write down the percentage reduction.
5. Manually verify one fact from early in the conversation is still correctly reflected in the model's answers after compaction has run — compaction that silently drops the one thing the user actually cares about is worse than no compaction at all.

Resources

• Anthropic Engineering — Effective Context Engineering for AI Agents: the deepest treatment of "what belongs in context and what doesn't" this path points to — read this one closely.
• Anthropic — Memory Tool: the shipped, file-backed memory mechanism used as this lesson's Real Product Example, including its pairing with server-side compaction.
• Vercel AI SDK — Chatbot Message Persistence: how to actually store and reload a conversation's messages between requests in an AI SDK-based app, including resuming a stream a client disconnected from mid-generation.