Build the eval set before you build the feature.
That sentence is easy to agree with and hard to do. Most teams build the feature, demo it, get excited, and ship it. The eval set shows up six weeks later as a post-mortem action item, drafted by someone who was not there for the original build, and forgotten before the next release.
This is not a discipline failure. It is an incentive failure.
A working demo generates applause. An eval set generates work. The incentive gradient points the wrong direction until you have lived through a production regression you could not root-cause.
The picture
Two development paths, starting from the same prompt change.
Path A: prompt change → deploy to staging → someone eyeballs it → "looks good" → ship → user reports something strange three weeks later → the team debates whether it is a regression or a never-worked edge case → nobody knows because there is no record of what the system used to do.
Path B: prompt change → eval suite runs in CI → pass rate moves from 87% to 79% on the CoT-reasoning task type → the diff surfaces in the PR review → team investigates → root cause is that the new prompt broke structured output on multi-step questions → fix, rerun, 89% → ship.
Path A is what most teams do.
Path B is what shipping teams do.
The difference is not sophistication. It is the presence or absence of a fixed artifact that defines "better."
Why it matters now
The 2024–2026 era is when "ship without evals" stops being a startup speed advantage and starts being a compounding tax.
Every prompt change without a regression signal pushes risk forward. The risk is invisible until it surfaces as a customer-visible failure that no one can root-cause because no one knows what the system used to do two releases ago. The support ticket reads "it used to work." The team reads it and shrugs.
The compounding part is underappreciated.
Each untracked regression narrows your future option space. You cannot safely roll back if you do not know what the previous behavior was. You cannot A/B test prompt variants if you have no baseline. You cannot hire a second engineer to work on the model layer because there is nothing to hand over — the "product" lives in one person's intuition.
The eval set is the artifact that converts intuition into institutional memory. Without it, the product is not really maintainable by anyone except the original author working carefully, slowly, and from memory.
A source you should trust
- AI Manual Chapter 4 — eval before launch. The single-prompt frame this course extends. Read the section on minimum-viable eval design before shipping any AI-powered surface.
- Anthropic's capability evaluation methodology. Operator-grade primary source on how a frontier lab thinks about measuring its own systems — and by extension, how a product team should think about measuring theirs.
- Hamel Husain's "Your AI Product Needs Evals." The essay that moved the conversation from research-lab practice to shipping-team practice. The term "compounding evals" comes from here.
- OpenAI Evals framework documentation. More useful as a thinking scaffold than as a direct dependency; the framing of task, output, and grader as three separate design decisions is worth internalizing.
A recipe
The minimum-viable eval set — designed to be built in one afternoon before you ship the first version of any AI feature:
- Pull twenty examples from real user input. If the feature is pre-launch, construct credible synthetic proxies from adjacent production traffic. Synthetic is worse than real; it is not worse than nothing.
- For each example, write the expected behavior in the simplest form that works — a target string, a set of required elements, a rubric with three levels. Do not write a paragraph of nuance. Write the decision rule.
- Choose the simplest grader that makes that decision correctly. Exact match for structured outputs. A small Python predicate for numeric checks. A rubric scored by a human for open-ended quality. LLM-as-judge only when the simpler rungs cannot decide — and even then, validate against a human sample before trusting the numbers.
- Run the suite on every prompt change, retrieval-index update, and model upgrade. Failures block merge.
- Grow the set when production surfaces a new failure mode you did not anticipate. Every novel customer complaint is a candidate eval example.
The twenty-example threshold feels low. It is. But twenty well-chosen, well-graded examples beats two hundred vague ones. The discipline is in the grading, not the volume.
The smell of it going wrong
- "We'll add evals after we ship" — said in week three, executed by week twenty, abandoned by week twenty-two.
- The eval set contains only happy paths. Every example works correctly on the current prompt. That is not a test suite; it is a victory lap.
- Pass rate is a single number with no breakdown by task type. You cannot tell if the number moved because citations improved and structured output regressed.
- The eval set has not been updated in three months, even though three new features shipped. The set is now measuring a product that no longer exists.
- Two engineers have different mental models of what "passing" means on a specific example. The grader was never written down.
A judgment call from real work
The Ostronaut retrieval harness was built twice. The first version shipped without a structured eval. Retrieval behavior was validated informally — someone ran a few representative queries, results looked right, and the feature went live. For several months, the system worked well enough that the absence of measurement was invisible.
Then a chunker change modified how markdown headings were handled. The change was well-intentioned, tested against a handful of manual queries, and landed in production on a Tuesday. By Thursday, retrieval quality on content-heavy queries had degraded noticeably. The support signal was slow and ambiguous. Nobody could articulate exactly what had changed, because nothing was measured before.
The investigation that followed produced the first version of what became the orphan_gap_pct metric: the fraction of content chunks that contained a load-bearing concept with no retrieval path to the vector index. The metric existed because the team needed to explain the regression, not because they had designed for measurement from the start.
The second version of the harness was built around that metric and a small golden set of retrieval queries graded against expected result sets. It ran on every indexing change. The orphan_gap_pct had a threshold. Violations blocked merge.
The before-and-after was stark. In the first version, a chunker bug was invisible for 72 hours. In the second version, a different chunker bug was caught in CI within one PR cycle and fixed before it ever reached staging.
The eval set did not make the team smarter. It made their knowledge durable.
There is a broader point here about what kind of product you are actually building. An AI feature without an eval set is a feature that only works while the original author is paying attention. That is not a product — it is a performance. A product can be handed to a new team member, upgraded, regressed, fixed, and upgraded again, with each step documented and comparable. The eval set is what makes that lifecycle possible.
If you are a PM or a team lead, the eval set is also your audit trail. When a model provider updates their model silently and your system regresses, the eval set tells you what broke, when, and by how much. Without it, you are dependent on user complaints to tell you something changed. User complaints are a lagging indicator. The eval set is a leading one.
In the next lesson, we will distinguish the eval you publish from the eval that decides whether you ship — because confusing those two objects is the most common mistake teams make after they have built a working golden set.
Rules from this lesson
- The eval set is the smallest non-optional artifact in an AI product. Without it, the product is not maintainable, not comparable across versions, and not transferable to a new team member.
- Build it before the feature, not after. The cost of building it after a production incident is ten times the cost of building it before.
- Twenty examples is enough to start; a hundred is enough to ship confidently; a thousand is when you have a genuine knowledge moat about your task distribution.
- The eval set converts intuition into institutional memory. When the original author leaves, the set is what stays.