Sign in

Embeddings

intermediate~45 minembeddingsvectorssimilarityrag

What a vector actually is, cosine similarity, and why chunking strategy — not the embedding model — is where most RAG systems fail.

Embeddings

Overview

Keyword search fails on the most ordinary thing a user does: asking the same question in different words. "How do I reset my password" finds nothing in a document titled "Account recovery," because they share no words.

Embeddings fix this by turning text into coordinates. Similar meanings land near each other, so search becomes a distance calculation instead of a string match. This lesson covers how that works, why chunking — not the vector math — is where most retrieval systems actually break, and what it costs you to change embedding model later.

Theory

A vector is an array of floats, and similarity is geometry. An embedding model maps text to a fixed-length array — 768 or 1536 numbers. Text with similar meaning lands close together. Cosine similarity measures the angle between two vectors, which is why "how do I reset my password" retrieves a doc titled "Account recovery" with no words in common.

More dimensions is not free. Higher dimensions capture more nuance and cost more to store, more to index, and more per query. Most products do fine on the smaller model, and you should measure before assuming otherwise.

Chunking is where RAG actually fails. Long before any vector math, you decided where to cut the document. Split on a fixed character count and you will slice a code sample or a table in half — retrieval still returns something, similarity scores still look reasonable, and the answers are quietly wrong. Split on headings instead.

Changing embedding model means re-embedding everything. Vectors from two different models are not comparable, so a model swap is a full re-index of your corpus. Budget for it, or pin the model and treat it like a schema.

Search is not the only use. The same vectors give you clustering, deduplication, recommendations, and "find me more like this" — all from one column you already have.

Visual Diagram

DocumentProse, docs, transcripts, anything unstructured.
ChunkSplit on headings, not on a fixed character count.
A chunk that splits a code block quietly ruins retrieval.
EmbedEach chunk becomes a fixed-length vector.
Store with its sourceVector plus a stable id you can cite back to.
Query by distanceThe question is embedded too; nearest neighbours win.
Text becomes geometry: chunk, embed, then search by distance instead of keyword.

Read the diagram left to right. A source document is split into chunks by the chunking strategy — this is the step where most quality is won or lost. Each chunk is sent to an embedding model, which returns a fixed-length vector. All those vectors live in the same high-dimensional space, where geometric closeness (measured by cosine similarity) stands in for semantic closeness. A query at inference time is embedded with the same model into that same space, and the nearest chunk-vectors to the query-vector are the retrieval result — which is exactly the mechanism Day 15's vector database is built to do quickly at scale.

Examples

  • Semantic search on a help center. A user searches "how do I get my money back" and the system retrieves an article titled "Refund Policy" even though the query shares zero words with the title, because both embed to nearby vectors.
  • Deduplicating support tickets. Two tickets phrased completely differently ("app won't open," "crashes on launch") embed close enough together to be flagged as likely duplicates before a human ever reads both.
  • A bad chunk, concretely. A 2,000-token chunk that contains a product's pricing table, its refund policy, and an unrelated FAQ about mobile apps produces one vector that's a mediocre match for all three topics and a great match for none — a query about pricing will retrieve it, but so will a query about the mobile app, diluting relevance for both.
  • Dimension truncation in practice. Requesting a 256-dimension embedding instead of the model's default 1536 for a use case like coarse topic clustering, where losing fine-grained nuance is an acceptable tradeoff for a 6x reduction in storage.

Code

Embedding text and comparing it, using the Vercel AI SDK's embed, embedMany, and cosineSimilarity:

import { embed, embedMany, cosineSimilarity } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";

const provider = createOpenAICompatible({
  name: "example-gateway",
  baseURL: "https://ai.gateway.lovable.dev/v1",
  headers: { "Lovable-API-Key": process.env.LOVABLE_API_KEY ?? "" },
});

const embeddingModel = provider.textEmbeddingModel("text-embedding-3-small");

/**
 * A naive fixed-size chunker with overlap. Splits on paragraph boundaries
 * first so a chunk never starts or ends mid-sentence when it can be avoided.
 */
function chunkText(text: string, maxChars = 1200, overlapChars = 150): string[] {
  const paragraphs = text.split(/\n\s*\n/);
  const chunks: string[] = [];
  let current = "";

  for (const paragraph of paragraphs) {
    if (current.length + paragraph.length > maxChars && current.length > 0) {
      chunks.push(current.trim());
      // Carry the tail of the previous chunk forward so a concept that spans
      // a boundary still appears in full inside at least one chunk.
      current = current.slice(-overlapChars);
    }
    current += `\n\n${paragraph}`;
  }
  if (current.trim()) chunks.push(current.trim());
  return chunks;
}

/** Embeds every chunk of a document in one batched call. */
export async function embedDocument(text: string) {
  const chunks = chunkText(text);
  const { embeddings } = await embedMany({ model: embeddingModel, values: chunks });
  return chunks.map((chunk, i) => ({ chunk, embedding: embeddings[i] }));
}

/** Finds the single most relevant chunk for a query — the core of retrieval. */
export async function findMostRelevant(
  query: string,
  indexed: { chunk: string; embedding: number[] }[],
) {
  const { embedding: queryEmbedding } = await embed({ model: embeddingModel, value: query });

  return indexed
    .map((item) => ({ ...item, score: cosineSimilarity(queryEmbedding, item.embedding) }))
    .sort((a, b) => b.score - a.score)[0];
}

embedMany matters here as much as the math: batching every chunk into one call instead of looping embed one chunk at a time is the difference between one network round trip and hundreds, and most embedding providers price and rate-limit batched calls more favorably than the equivalent number of single calls.

Real Product Example

Gmail's Smart Reply, one of the earliest large-scale production uses of text embeddings, generates its short suggested-response chips by working in embedding space rather than generating free text per email. Google's engineering team (Kannan et al., "Smart Reply: Automated Response Suggestion for Email") built the system to encode an incoming email into a vector, then compare it against a large pre-built set of candidate response vectors to find semantically appropriate replies — and, critically, to cluster those candidates in embedding space first, so the two or three suggestions shown to the user are diverse (not three near-duplicate phrasings of "sounds good") rather than just the three single highest-scoring matches. The chunking-equivalent decision in their pipeline was choosing what unit of text to encode (the email, not the whole thread) — get that unit wrong and the embedding represents the wrong thing to compare against.

Common Mistakes

Chunking by a fixed character count with no regard for structure

Slicing a document every 1000 characters regardless of sentence or paragraph boundaries routinely cuts a chunk in half mid-idea, producing an embedding that represents neither the sentence before the cut nor the one after it well. Prefer splitting on natural boundaries — paragraphs, then sentences — and only fall back to a hard character limit when a single paragraph itself exceeds it.

Comparing embeddings from two different models

Every embedding model defines its own vector space; a similarity score computed between a vector from model A and a vector from model B is meaningless, even if the numbers "look like" they're in a plausible range. This bug is especially easy to introduce silently when a query path and an indexing path independently import different SDK defaults. Pin the model name explicitly on both sides and assert it matches at index-read time.

Treating chunk size as a one-time decision instead of an empirical one

Copying "512 tokens with 50 token overlap" from a blog post and never revisiting it ignores that the right chunk size depends on your content (dense legal text needs different chunking than chat transcripts) and your query patterns (short factoid queries want small precise chunks; broad summarization queries want larger context-rich ones). Treat chunk size as a hyperparameter you evaluate against real queries, the same way Day 13 treated a prompt as something you evaluate rather than assume.

Re-embedding on every query instead of caching document embeddings

Document embeddings are expensive to compute and cheap to store — embed each document chunk once, persist the vector, and only ever re-embed the (much shorter) incoming query at request time. Teams that accidentally re-embed static content on every request pay for the same API call over and over for no benefit.

Senior Tips

Senior tip

Store the chunk's source metadata (document ID, section heading, position) right next to the vector, not in a separate lookup table you join later. When retrieval returns a chunk, you almost always need to show the user where it came from or fetch neighboring chunks for more context — denormalizing this at write time avoids an extra round trip on every single query, which matters far more than it looks like it should once you're serving real traffic.

Senior tip

Overlap chunks by content, not by a fixed character count, when a concept naturally spans a boundary. A flat character-count overlap is a reasonable default, but for structured content (a table, a numbered procedure), it's often worth detecting the structure and either keeping it in one chunk regardless of size or duplicating the whole structural unit into both neighboring chunks — a table split across two chunks is useless in either one.

Senior tip

Benchmark chunk size and overlap with a small labeled retrieval eval, the same discipline as Day 13's prompt evals. Take 15-20 real queries with a known correct source chunk, try two or three chunking configurations, and measure hit rate — "was the correct chunk in the top-k results" — directly. This turns a decision usually made by intuition into one made by measurement, and it's cheap to run before you've built the rest of the RAG pipeline.

Senior tip

Don't default to the largest embedding model available "to be safe." Larger embedding models are slower and more expensive per call with diminishing retrieval-quality returns for many practical tasks — a smaller model combined with good chunking regularly beats a larger model with bad chunking. Validate the cheaper model against your eval set before assuming you need the expensive one.

Exercises

Exercise

Take a paragraph of text and manually split it into three chunks: one that cuts mid-sentence, one that splits cleanly on a paragraph boundary, and one that's the whole thing unsplit. Predict, before running any code, which chunk you'd expect to retrieve well for a query about the paragraph's main topic — then explain why in one sentence.

Exercise

Using the embedDocument and findMostRelevant functions from the Code section, embed a short document (a README, an FAQ) with maxChars = 300 and again with maxChars = 1500. Run the same three queries against both and note which chunk size retrieved the more precisely relevant chunk for each query.

Exercise

Explain, in your own words, why cosine similarity between an embedding from text-embedding-3-small and one from a completely different provider's model would be meaningless even if both vectors happened to have the same number of dimensions.

Mini Project

Chunk-and-embed a document. Build the smallest possible retrieval evaluation to make chunking strategy concrete instead of theoretical.

  1. Pick a real document with some internal structure — a README, a policy page, a long FAQ (500-2000 words).
  2. Write two chunkers: one naive (fixed character count, no regard for structure) and one structure-aware (splits on paragraph/heading boundaries, keeps related content together).
  3. Embed the document both ways using embedDocument from the Code section.
  4. Write five realistic queries a real user might ask about the document's content.
  5. For each query, run findMostRelevant against both chunk sets and record which one returned the more useful chunk. Tally the score across all five queries.

You should end up with a concrete number — "the structure-aware chunker won 4 of 5 queries" — instead of a hunch. That number is the whole argument for treating chunking as an engineering decision.

Resources