20 cases before launch: a minimal eval set
You do not need a platform to start testing an AI feature. You need about twenty real cases, a rule for passing, and a rule for when not to ship.
Plenty of teams launch an AI feature after trying it on five examples that worked. A small test set, built before launch, gives you a baseline and can flag when a change makes results worse.
Anthropic's engineering team says that 20 to 50 simple tasks drawn from real failures make a good start. It adds a caveat: a starter set like that can catch large early changes, while a mature system may need bigger and harder suites to notice smaller effects. Its guide also says that early on, manual testing and intuition can take you a long way, and that evals matter more once a system is in production and growing. Even so, writing them early makes you say what success means.
Where the cases come from
Use real examples of requests, with permission. Strip out names, emails and anything private before you save them. Include difficult or unusual cases, and aim for variety rather than twenty copies of the easy one.
Here is one possible starting mix. It is my suggestion, not a standard: ten typical cases, five hard ones, three that should be refused or escalated, and two near-duplicates where the changed detail should change the decision, such as the same refund request inside and outside the return window.
Write the criteria first
For each case, write down what a good answer must contain and must not contain, in a form someone can check. "Says how long the product can be returned and does not promise an exception" works. "Helpful wording" is too vague until you define what it means. OpenAI recommends clear evaluation criteria, so use a rubric that a second reviewer could apply the same way.
If your feature takes actions, judge what really happened, such as the record created or the email drafted, and not only what the agent said.
A sample row and results table (hypothetical)
Imagine a shop whose policy is: returns within 30 days, store credit after that.
Input: a customer asks for a refund 45 days after purchase. Expected: says the 30-day return period has passed and offers store credit. Forbidden: promising a refund or inventing a different policy. Why it exists: it tests the edge of the policy. Severity: high. Must-pass: yes.
Illustrative results, not real measurements. Case 1, a typical question: passed 3 of 3 trials. Case 2, the 45-day refund, must-pass: passed 2 of 3 trials, so the release is blocked. Case 3, a near-duplicate at day 25: passed 3 of 3 trials.
Keep the result for every case, because an average hides the one that matters.
Run it, then change one thing
Run every case with a fixed model, prompt and data setup, and save the result as your baseline. Then change one thing and run again. Outputs vary, so repeat each case a few times. A single run can flag a regression, but it does not prove one.
When a case fails, look for the cause. It might be a missing instruction, missing context, a tool error, or a mistake in the grading itself. The fix is not always a new prompt.
Decide the release rule in advance
Mark some cases as must-pass, for example safety or policy cases. Decide before you run anything that one must-pass failure blocks the release, whatever the average says. Add a case after every real failure, and keep a few held-out cases that you never tune against, so the score keeps its meaning as the suite grows.
Twenty cases will not prove a feature is safe. They can flag when a change makes it worse.
Sources
Originally published at https://withavi.ai/en/posts/small-eval-set-before-launch/
Avi.Bi