Once you add an AI feature, you keep finding reasons to tweak the prompt. Someone says the answers are too long, so you shorten them; a new model comes out, so you want to try it. But when you ask "Is it better than before?" after the change, the answer often comes from typing in a few questions by hand and concluding, "Seems fine."
The problem is that fixing one thing can quietly break another. Summaries get better but the table format breaks, or the tone gets friendlier but a required notice goes missing. This post explains how to build an LLM eval set that catches these problems, based on Anthropic's and OpenAI's official documentation. It is part 6 of our AI feature development series, following demo to production, API costs, structured output, streaming, and prompt caching.
What an eval set is
An eval set is a test sheet that collects "for this question, the answer should look like this." You write down each question (input) alongside the expected answer or grading criteria, and every time you change the prompt or model, you have it take the same test again and compare the scores.
OpenAI's documentation describes the same flow: define the task as an eval, prepare data with inputs and ground-truth answers, set up graders, run it, then review the results, revise the prompt, and run it again. Anthropic's documentation stresses defining specific success criteria before all of that — not "a good answer," but measurable statements like "the classification is correct" or "it summarizes in three sentences or fewer."
Collecting questions: start with the ones you will actually get
The quality of an eval set depends on how closely its questions resemble reality. Anthropic's documentation recommends mirroring the distribution of questions you will see in real work.
- Real questions: if the feature is live, pull them from logs; before launch, pick them from questions the people in charge actually receive (with personal information removed)
- Edge cases: irrelevant questions, questions missing information, very long inputs, questions that can be read two ways
- Rules that must hold: questions that must not be answered, notices that must always be included
- Volume: it is fine to start small. Anthropic's documentation explains that many automatically graded items beat a handful of hand-graded ones
After writing your first few dozen questions, the documentation also describes having AI generate similar variations to expand the set. Just have a person skim AI-generated questions before adding them.
Want a written plan for where to start?AI Readiness Assessment, KRW 490K · priorities and expected scope in a reportRequestChoosing a grading method: code, AI, or human
Even within one eval set, the grading method differs by item. Anthropic's documentation presents three, in order of speed and reliability.
| Grading method | Best for | Watch out for |
|---|---|---|
| Code-based grading | Classifications, numbers, and JSON formats with a fixed correct answer | May miss correct answers phrased differently |
| AI grading | Summary quality, tone, rule compliance | Write a detailed rubric and first check that it agrees with human judgment |
| Human grading | Setting initial standards, ambiguous items | Slow and costly, so use it only where truly needed |
When you have AI do the grading, make it output only fixed answers such as "correct/incorrect" or a 1–5 score instead of "good/bad," and have it write its reasoning before the score — this makes its judgments more stable. Anthropic's documentation also recommends, where possible, grading with a different model from the one that produced the answer.
Recording results and running it again
An eval set is not something you run once and forget — it is a habit of re-running it every time you make a change.
- Run it first with your current prompt and model, and record a baseline score
- Write down what you changed in one line, such as a prompt edit, a model swap, or a change in reference material
- Run the same eval set again and compare item by item. Look at "newly failing items" before the overall score
- Add failure cases you discover in production to the eval set as new items
Build it up this way and you can answer "Is it OK to switch to the new model?" with item-by-item results instead of a hunch. If you also log response time and cost in the same table, you can compare quality, speed, and cost at a glance.
Summary
LLM evaluation starts not with elaborate tooling, but with the habit of retaking the same test every time. Collect real questions, decide how to grade each item, and re-run and record after every change, and editing prompts becomes far less nerve-racking. If you are just starting to add AI features, build one small eval set before launch.
References
- Anthropic, Define success criteria and build evaluations: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
- OpenAI, Working with evals: https://developers.openai.com/api/docs/guides/evals
We review your current system and data, then outline the migration order and scope.
We build web apps, mobile apps and business systems, then add and run the AI they need. We build and operate our own meeting notes, contract and company analysis AI.
See our projects