Golden dataset for LLM evaluation: how many examples you need
It depends on the error you can accept. 20 to 50 reviewed examples catch gross regressions. To report a pass rate within about 10 points you need about 100 graded examples per category, and within 5 points about 385. Telling two models apart on a 3 point gap takes about 1,000.
What a golden dataset is
A golden dataset (also called a golden set, eval set or test set) is a list of inputs your AI will really see, each with the answer you expect or the rule a good answer must meet. Every time you change a prompt, a model or a retrieval step, you run the whole set and compare the score with last time.
Each row usually holds four things: the input, the expected answer or a short grading rule, a category, and why the row is there. Here is what that looks like for a support assistant.
| Input | Expected answer or rule | Category | Why it's in the set |
|---|---|---|---|
| Can I return shoes I wore once? | Says returns need unworn items; offers exchange for faults | Returns policy | Most common question last month |
| My order 4471 never came | Asks for the email on the order; never invents a tracking number | Order status | The bot once made up a tracking link |
| Cancel my plan and delete my data | Points to the account deletion page; doesn't promise a date | Account and privacy | Legal asked for this check |
What makes it golden is the checking, not the size. Langfuse puts it plainly: coverage of distinct failure modes matters more than raw count, and 100 diverse items beat 1,000 near-duplicates.
How many examples: it depends on what the set is for
| What the set is for | Size | Source |
|---|---|---|
| Trying evaluation on a prototype | About 10 | Google Cloud codelab |
| Catching gross regressions | 20 to 50 reviewed items | Langfuse |
| A full regression set that mirrors production | 100 to 1,000 | Langfuse |
| Results you can call statistically meaningful | 100 to 500 | Google Cloud codelab |
| Checking an LLM judge, per failure mode | 100 to 200 labelled examples | Hamel Husain and Shreya Shankar |
| Telling two models apart on a 3 point gap | About 1,000 | Evan Miller, Anthropic |
Read the table from the top. A small set tells you something broke. A bigger one tells you how often it breaks. Only a large one tells you whether version B is really better than version A, or got lucky on this run.
Before you pick a number, write down the decision the score will drive: ship or don't ship, model A or model B, fix this category first. The decision sets the error you can live with, and the error sets the size.
Size it by the error you can accept
What is this set for?
How far off can your pass rate be?
What pass rate do you expect?
To report a pass rate you can quote
246 graded examples per category, so 984 in all for 4 categories.
95% confidence, standard sample-size formula for a proportion. Assumes the examples are independent; related examples need more.
The maths is the standard sample-size formula for a proportion: examples needed = 1.96² × p × (1 - p) ÷ margin², where p is the pass rate you expect and 1.96 gives 95% confidence. The error is widest when the pass rate is near 50%, so if you don't know what to expect, plan for 50%.
| Graded examples | Pass rate near 50% | Near 80% | Near 90% |
|---|---|---|---|
| 50 | ±13.9 points | ±11.1 points | ±8.3 points |
| 100 | ±9.8 points | ±7.8 points | ±5.9 points |
| 200 | ±6.9 points | ±5.5 points | ±4.2 points |
| 400 | ±4.9 points | ±3.9 points | ±2.9 points |
| 1,000 | ±3.1 points | ±2.5 points | ±1.9 points |
So with 100 examples and a measured 80%, the true rate could sit anywhere from about 72% to 88%. A change that moves your score from 80% to 84% on that set may be nothing at all. Near 95% and above, with under about 100 examples, this simple formula is too optimistic; use a Wilson interval instead.
Every category you report needs its own examples
A set of 400 gives about ±5 points on the overall score. Split it across four categories of 100 and each category is back to about ±10 points. If you only look at the overall number, size for that. If you'll decide to fix refunds before shipping because the refunds score dropped, refunds needs enough examples of its own.
- List the categories you would act on separately: task types, languages, customer tiers, risky topics.
- Give the ones that drive decisions the full count from the calculator. Others can share a smaller pool.
- Add rare but serious cases on purpose, even if they make up 1% of traffic. A random sample will miss them.
- Write down which categories you report, so nobody reads a 12 example slice as if it were measured.
Comparing two models or prompts
Evan Miller's paper "Adding Error Bars to Evals" (Anthropic, 2024) makes three points that change how many examples a comparison needs:
- Pair the comparison. When both models answer the same questions, compare question by question. Models tend to find the same questions easy or hard, so the error on the difference shrinks. Miller calls this a free reduction in variance.
- Small gaps need big sets. In his worked example, telling apart two models 3 points apart needs about 969 questions, so he suggests new evals have at least 1,000. Large gaps show up with far fewer.
- Related questions count for less. Several questions about one document, or one question in many languages, aren't independent. On one real eval the honest error bar was up to 3 times wider than the simple one.
Two habits help. Ask each question more than once and average, which cuts the noise from the model's own randomness. And look at the examples where the two versions disagree: they tell you more than the headline score.
How to build a golden dataset that holds up
- Start from real inputs. Pull them from production logs or support tickets, with personal data removed. Invented questions miss how people really write.
- Remove near-duplicates. Ten wordings of the same question count as about one.
- Write the grading rule before the answers. A short rubric (what a good answer must do, what it must never do) keeps labellers consistent.
- Label each expected answer, then have a second person check it. Where they disagree, fix the rubric, not just the row.
- Keep a part you never tune on. If you edit prompts until the set passes, the set stops measuring anything.
- Add every real failure. When the AI gets something wrong in production, it goes into the set, with its category.
- Version it. Date the set and note what changed, so last month's score and this month's mean the same thing.
Anthropic's guidance on test cases favours volume: more questions graded automatically beat fewer graded by hand. The two fit together. People check the expected answers in the golden set once, carefully; after that, code or an LLM judge can grade each run against them. If you use an LLM judge, check it against human labels first; our guide to AI answer review covers how that work is done.
If your team has no time to label, a managed team can build the set from your inputs and rubric, with each item checked by a second person. Send 50 to 100 of your items and the rubric as a paid test, and judge the labels before committing to the rest. Start a brief.
A golden dataset is not fine-tuning data
Fine-tuning has its own, smaller numbers. OpenAI's fine-tuning guide sets a minimum of 10 training examples and suggests starting with 50 well-crafted demonstrations, then evaluating the results. Those are examples the model learns from. Your golden set must stay separate, or the score only shows how well the model remembers it.
Questions
How many examples should a golden dataset have?
Size it by purpose. 20 to 50 reviewed examples catch gross regressions. About 100 graded examples per category give a pass rate within about 10 points, and 385 within 5 points. Comparing two models on a small gap takes about 1,000.
Is 50 examples enough for an LLM eval?
Enough to catch obvious breaks, not to measure. At 50 examples the error on a pass rate near 50% is ±13.9 points, so a 5 point change is noise.
What is a golden dataset in AI?
A fixed set of real inputs, each with an expected answer or grading rule that a person has checked, used to score every new version of a model, prompt or pipeline the same way.
How many test cases do I need to compare two models?
Run both on the same items and compare item by item. For a 3 point gap, Evan Miller's worked example needs about 969 questions; bigger gaps need far fewer.
Can an LLM write my golden dataset?
It can draft inputs and suggest answers, which saves time. A person should check every expected answer, because a wrong one quietly skews every score you run against it.
How often should I update a golden dataset?
Add real failures as they happen and review the whole set when your product or users change. Version each change so scores over time stay comparable.
Sources
- Golden dataset evaluation: starter and regression set sizes Langfuse
- AI Evals: everything you need to know (100 to 200 labels per failure mode) Hamel Husain and Shreya Shankar
- Adding Error Bars to Evals: a statistical approach to language model evaluations Evan Miller, Anthropic (arXiv, 2024)
- Evaluating single LLM outputs with Vertex AI Evaluation Google Cloud
- Create strong empirical evaluations Anthropic
- Supervised fine-tuning: how many examples OpenAI
- Sample size determination: estimation of a proportion Wikipedia
- Binomial proportion confidence interval: Wilson score interval Wikipedia