Plenty of teams launch an AI feature after trying it on five examples that worked. A small test set, built before launch, gives you a baseline and can flag when a change makes results worse.

Anthropic's engineering team says that 20 to 50 simple tasks drawn from real failures make a good start. It adds a caveat: a starter set like that can catch large early changes, while a mature system may need bigger and harder suites to notice smaller effects. Its guide also says that early on, manual testing and intuition can take you a long way, and that evals matter more once a system is in production and growing. Even so, writing them early makes you say what success means.

Where the cases come from

Use real examples of requests, with permission. Strip out names, emails and anything private before you save them. Include difficult or unusual cases, and aim for variety rather than twenty copies of the easy one.

Here is one possible starting mix. It is my suggestion, not a standard: ten typical cases, five hard ones, three that should be refused or escalated, and two near-duplicates where the changed detail should change the decision, such as the same refund request inside and outside the return window.

Write the criteria first

For each case, write down what a good answer must contain and must not contain, in a form someone can check. "Says how long the product can be returned and does not promise an exception" works. "Helpful wording" is too vague until you define what it means. OpenAI recommends clear evaluation criteria, so use a rubric that a second reviewer could apply the same way.

If your feature takes actions, judge what really happened, such as the record created or the email drafted, and not only what the agent said.

A sample row and results table (hypothetical)

Imagine a shop whose policy is: returns within 30 days, store credit after that.

Input: a customer asks for a refund 45 days after purchase. Expected: says the 30-day return period has passed and offers store credit. Forbidden: promising a refund or inventing a different policy. Why it exists: it tests the edge of the policy. Severity: high. Must-pass: yes.

Illustrative results, not real measurements. Case 1, a typical question: passed 3 of 3 trials. Case 2, the 45-day refund, must-pass: passed 2 of 3 trials, so the release is blocked. Case 3, a near-duplicate at day 25: passed 3 of 3 trials.

Keep the result for every case, because an average hides the one that matters.

Run it, then change one thing

Run every case with a fixed model, prompt and data setup, and save the result as your baseline. Then change one thing and run again. Outputs vary, so repeat each case a few times. A single run can flag a regression, but it does not prove one.

When a case fails, look for the cause. It might be a missing instruction, missing context, a tool error, or a mistake in the grading itself. The fix is not always a new prompt.

Decide the release rule in advance

Mark some cases as must-pass, for example safety or policy cases. Decide before you run anything that one must-pass failure blocks the release, whatever the average says. Add a case after every real failure, and keep a few held-out cases that you never tune against, so the score keeps its meaning as the suite grows.

Twenty cases will not prove a feature is safe. They can flag when a change makes it worse.

Sources

• Anthropic: Demystifying evals for AI agents
• OpenAI: Evaluation best practices

Originally published at https://withavi.ai/en/posts/small-eval-set-before-launch/

Avi.Bi

Give a coding agent a fuzzy task and it may quietly fill the gaps with assumptions. A clearer task means less of that guessing. It will not guarantee correct code, so a good task also says how you will check the result.

A quick word on speed, because it is tempting to trust the feeling. In a 2025 controlled study with random assignment, METR followed 16 experienced open-source developers through 246 tasks in repositories they knew well, using early-2025 tools. With AI, the tasks took 19% longer, even though the developers expected to go faster. That result belongs to those tools and that setting. In February 2026 METR wrote that newer tools probably help more, but that selection effects, in who took part and in which tasks they chose to submit, made the new estimate unreliable. What I take from it: the feeling of speed is not a measurement, so check the work.

Start with what done looks like

Before anything else, decide how you will know the task is finished. When requirements are missing, the agent has to guess. Write the outcome in one sentence, then the check that shows it.

The two vendor guides lean on different halves of this. Claude Code's guide suggests giving the agent something it can run to verify its own work: tests, a command, a screenshot to compare. GitHub's Copilot guide puts the weight on clear scope and acceptance criteria. You want both.

Practice on something low-risk. Anything that touches login, permissions or payments deserves much more care: explicit tests for the failure cases, and a person doing a security review.

Give the context that matters, not all of it

Name the files involved, point to one existing example worth following, and state the constraint that matters. Say what must stay untouched. Resist pasting everything you know, because a long task can bury the details that count.

If the change spans several files, ask for a plan first and read it. A reviewed plan can catch a wrong turn before any code exists.

A full example (hypothetical)

Here is an imagined Python project.

Goal: add a --dry-run flag to the export command, so it prints what it would write and changes nothing.

Context: the command lives in myapp/export.py. Handle the flag the way --verbose is handled in myapp/sync.py, and reuse the existing writer class.

Constraints: no new dependencies, no changes to deploy or CI files, never read or print secrets or environment files, nothing changed outside myapp/ and tests/. Any test that exercises the writer runs only in an isolated temporary directory, never in a real output location.

Done when: python -m pytest tests/test_export.py runs without errors and every test passes, including two new ones. One checks that --dry-run writes no files. The other checks that a normal run still writes them. Report back with the test output and the list of changed files.

Out of scope: renaming flags and refactoring the writer.

What you check as the human

Read the whole diff. Confirm that the new --dry-run test fails without the change and passes with it, and that the normal-run test passes both before and after. Check that the writer tests use a temporary directory, that nothing outside myapp/ and tests/ changed, and that no dependency or config file moved. Then run the command yourself, once with the flag and once without, in a scratch folder.

Before you approve, make sure you can explain the change in your own words. If you cannot, ask the agent to clarify it, and split the task if it was too big.

The template

Five fields cover most tasks: Goal, Context, Constraints, Done when, Out of scope.

DORA's 2025 research describes AI as an amplifier of a team and its practices, which means it amplifies strengths and weaknesses alike. So my advice is to start with clear tasks, tests, and small changes that someone reviews.

Sources

• METR: Measuring the impact of early-2025 AI on experienced open-source developer productivity
• METR: We are changing our developer productivity experiment design (Feb 2026)
• Claude Code: best practices
• GitHub Copilot: best practices for using Copilot to work on tasks
• DORA: State of AI-assisted Software Development 2025

Originally published at https://withavi.ai/en/posts/spec-for-ai-coding-agent/

Avi.Bi