Sign in

Prompt Engineering

intermediate~40 minprompt-engineeringllmsevalsstructured-output

Prompting as engineering, not folklore — structured output, schema-constrained generation, and evals that survive a model upgrade.

Prompt Engineering

Overview

Prompt engineering has a folklore problem. Search for it and you get listicles of magic phrases — offer the model a tip, tell it to take a deep breath — presented as spells rather than what they are: brittle correlations someone found against one model checkpoint.

The real thing looks like ordinary software engineering. You define the contract the output must satisfy, usually a schema. You write test cases before you write the prompt. You change one variable at a time and measure against a fixed set of labelled examples. This lesson is that discipline: schemas, evals, and why the tricks stop working after every model upgrade.

Theory

Prompting is programming with a very unusual compiler. No type checker, no linter, no guaranteed determinism — even at temperature: 0. Your instruction becomes a probability distribution over tokens, sampled repeatedly, and what comes back is text. Everything below is about closing the gap between "text" and "data you can trust."

Structured output is the highest-leverage technique there is. Give the model a schema instead of asking it to describe something in prose. generateObject hands you a value typed exactly like your Zod schema, not a string you hope is JSON. "Does the model understand the task" becomes "does the parsed value validate" — a boolean your code can branch on.

Evals are what make "better prompt" mean anything. Without labelled examples and a scoring function, "I improved it" is unfalsifiable — you changed some wording, looked at three outputs, and they seemed fine. Fifteen to twenty real hand-labelled cases turn every prompt change into something you can A/B.

Prompt tricks do not survive model upgrades. A phrase that nudged one checkpoint is an artifact of that checkpoint's training, not a property of language. Depend on the smallest set of load-bearing instructions — the schema, the explicit constraints, the eval set — and treat the rest as phrasing you are free to rewrite.

Show, do not adjectivise. "Respond concisely" is a weak signal. Two examples of the concise output you want is a strong one. When an eval shows a format or tone failure, reach for examples before more adjectives.

Visual Diagram

Write the eval set15-20 real, hand-labelled examples.
Draft prompt + schemaThe schema is the contract; the wording is disposable.
GenerateSchema-constrained, so output is data, not a string you hope parses.
Score against labelsOne number. Not a vibe.
Keep or revertScore went up, keep it. Didn't, throw it away.
Regression? Add that case to the eval set before fixing it.
Runs again on every prompt, schema or model change. It has no end state.
The prompt engineering loop: eval set first, then change one thing and score it.

Read the diagram as a cycle, not a pipeline with a fixed end. You start on the left with a labeled eval set — inputs paired with the correct output. A draft prompt plus a schema go into the model, producing a candidate output. That candidate is scored against the eval set's expected answers, producing a number. If the number improved, you keep the change; if it didn't, you revert it. The loop has no terminal state — it runs every time you touch the prompt, the schema, or the model version, for the entire lifetime of the feature.

Examples

  • Support ticket classification. Instead of "classify this ticket," a schema constrains the output to { category: "billing" | "bug" | "feature-request" | "other", urgent: boolean } — there is no way for the model to return a category that doesn't exist in your system.
  • Extraction from unstructured text. Pulling a shipping address, an order number, and a complaint reason out of a free-form customer email into a typed object, instead of regex-matching prose.
  • An eval set for a summarization feature. Twenty real support transcripts, each with a human-written "ideal" summary, scored automatically with an LLM-as-judge prompt that compares candidate summaries to the ideal ones on a 1–5 scale.
  • A regression test that runs in CI. Every pull request that touches a prompt template runs the eval set and fails the build if the aggregate score drops below a threshold — the same instinct as a unit test suite, applied to a non-deterministic component.

Code

Two things, side by side: schema-constrained generation with generateObject, and a minimal eval runner that turns "did the prompt get better" into a number instead of a feeling.

import { generateObject } from "ai";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { z } from "zod";

const provider = createOpenAICompatible({
  name: "example-gateway",
  baseURL: "https://ai.gateway.lovable.dev/v1",
  headers: { "Lovable-API-Key": process.env.LOVABLE_API_KEY ?? "" },
});

const ticketSchema = z.object({
  category: z.enum(["billing", "bug", "feature-request", "other"]),
  urgent: z.boolean(),
  reasoning: z.string().describe("One sentence explaining the category choice."),
});

type Ticket = z.infer<typeof ticketSchema>;

/**
 * generateObject validates the model's output against ticketSchema before
 * this function ever sees it — malformed output throws here, not three
 * files downstream where a raw JSON.parse would have silently corrupted data.
 */
export async function classifyTicket(body: string): Promise<Ticket> {
  const { object } = await generateObject({
    model: provider("gpt-4o-mini"),
    schema: ticketSchema,
    prompt: `Classify this support ticket:\n\n${body}`,
  });
  return object;
}

// --- A minimal eval runner ---

interface EvalCase {
  input: string;
  expected: Ticket["category"];
}

const evalSet: EvalCase[] = [
  { input: "I was charged twice this month, please refund one charge.", expected: "billing" },
  { input: "The app crashes every time I upload a PNG larger than 5MB.", expected: "bug" },
  { input: "Could you add dark mode to the dashboard?", expected: "feature-request" },
  // ...15-20 real examples, not synthetic ones, catch the failures that matter.
];

/** Scores the current prompt/schema against the fixed eval set. Run this in CI. */
export async function runEval(): Promise<{ accuracy: number; failures: EvalCase[] }> {
  const failures: EvalCase[] = [];
  for (const testCase of evalSet) {
    const result = await classifyTicket(testCase.input);
    if (result.category !== testCase.expected) failures.push(testCase);
  }
  const accuracy = (evalSet.length - failures.length) / evalSet.length;
  return { accuracy, failures };
}

The eval runner is deliberately unglamorous — a loop and a comparison. That's the point: you don't need a sophisticated evaluation framework to get the core benefit. You need a fixed set of real examples and a function that returns a number, run before and after every prompt change.

Real Product Example

GitHub Copilot Chat's slash commands (/fix, /explain, /tests, /doc) are a clean, shipped example of structured prompt engineering rather than freeform chat. Instead of relying on a single general-purpose system prompt to infer what the user wants from an ambiguous message like "this is broken," each slash command routes to its own purpose-built prompt template with its own instructions, its own expected output shape, and its own context-assembly logic — /tests pulls in the function under test and the project's existing test file conventions; /fix pulls in the diagnostic or error message from the editor. This is prompt engineering as system design: rather than one prompt trying to do everything well, the product decomposes the task space into known intents and gives each one a dedicated, testable prompt. It's the same instinct as replacing one giant API endpoint with several small, well-typed ones.

Common Mistakes

Parsing prose instead of requesting structured output

Asking a model to "respond with the category on the first line and urgency on the second" and then splitting the response by newline is fragile in ways that only show up in production: the model adds a preamble, wraps the answer in markdown, or reorders the lines under a slightly different phrasing of the same question. Use generateObject with a schema so the format is enforced by the SDK, not by string parsing you wrote yourself.

Iterating on a prompt by re-reading outputs instead of scoring them

Tweaking a prompt, generating five outputs, and deciding "that looks better" is not evaluation — it's confirmation bias with extra steps. Without a fixed eval set, you cannot tell a genuine improvement from a prompt that got better at the three examples you happened to look at and worse at everything else. Score every change against the same eval set, every time.

Copying prompt tricks from a blog post without an eval to verify they help your task

A technique that improved someone else's benchmark on someone else's model is a hypothesis for your use case, not a fact. Prepending "think step by step," adding a fake tip offer, or any other technique you found online should go through the same loop as any other prompt change: does it move your eval score, on your task, on your model. If you can't measure it, don't ship it on faith.

Writing one enormous prompt that tries to handle every case

A single sprawling system prompt with a dozen conditional instructions ("if the user asks X, do Y, unless Z...") degrades in ways that are hard to debug, because you can't tell which instruction the model is actually attending to. Prefer Copilot Chat's approach from this lesson's Real Product Example: decompose into several smaller, purpose-specific prompts routed by intent, each simple enough to reason about and eval independently.

Senior Tips

Senior tip

Keep a "known regressions" file next to your eval set. Every time a real user hits a failure that your eval set didn't catch, add that exact case to the eval set before you fix the prompt. This is the prompt-engineering equivalent of "write a failing test before you fix the bug" — it guarantees the same failure mode can never silently come back after a future model upgrade or prompt rewrite.

Senior tip

Separate "does the format validate" from "is the content correct" as two different scores. A response can pass Zod validation and still be wrong — a well-formed JSON object with the wrong category. Track schema-pass rate and accuracy-against-label as two separate numbers; a schema-pass rate that's near 100% but a flat or falling accuracy tells you the model is confidently producing well-shaped garbage, a very different problem than a formatting bug.

Senior tip

Pin a model version for anything with an eval gate, and upgrade deliberately. Auto-upgrading to "latest" is convenient until an unannounced weight update shifts your eval score without a code change on your side to explain it. Treat a model version bump exactly like a dependency bump: run the full eval suite against the new version before switching, on purpose, in a commit you can revert.

Senior tip

Use a cheap, fast model to grade a more expensive model's output, but validate the grader itself first. LLM-as-judge scoring scales far better than manual review, but a judge prompt can have its own biases (favoring longer answers, being lenient on category tasks). Before trusting it, run the judge against a handful of outputs you've scored by hand and confirm it agrees with you — an unvalidated judge just moves the folklore problem one level up instead of solving it.

Exercises

Exercise

Take the classifyTicket function from the Code section. Write five new eval cases, deliberately including one ambiguous ticket that could plausibly fall into two categories. Run runEval and note whether the ambiguous case fails — then decide, in one sentence, whether the fix is a better prompt or a better label.

Exercise

Find a prompt engineering tip you've seen online (a specific phrase, a formatting trick, a "roleplay" framing). Design a two-line eval — one input where you'd expect the trick to help, one where you'd expect it not to matter — and explain what result would convince you the trick is real versus placebo.

Exercise

Rewrite the ticketSchema from the Code section to add a confidence: z.number().min(0).max(1) field, and change classifyTicket's caller to route anything below 0.7 confidence to a human instead of auto-processing it. Explain in a sentence why this is a stronger production pattern than trusting every classification equally.

Mini Project

Eval-driven ticket classifier. Build this in the order the lesson argues for — eval set first, prompt second — and notice how different it feels from writing the prompt first.

  1. Write 20 labeled examples of real or realistic support tickets, each with a hand-assigned category. Store them as a typed array, like the evalSet in the Code section.
  2. Write the schema and the generateObject call. Run the eval. Write down the accuracy.
  3. Make exactly one change to the prompt (not the schema) intended to fix the biggest failure category. Run the eval again. Did the number move in the direction you predicted?
  4. Repeat step 3 twice more, changing one thing at a time, keeping a short log of "change → score" pairs.
  5. Add one deliberately adversarial ticket (sarcastic, multi-issue, or in a different tone than your other examples) to the eval set and see whether your final prompt survives it.

The deliverable isn't a perfect classifier — it's the log. That log is the difference between "I think this prompt is better" and "I can show you it's better."

Resources