How to choose a data annotation vendor: 10 answers to get in writing
Choose a data annotation vendor on written answers, not a sales call. Ask each shortlisted vendor the same 10 questions: who does the work, how agreement is measured, what the price leaves out, what triggers free rework, who owns the vendor and how your data is deleted. Then run a paid pilot on your own items before signing anything long.
Judge the system, not the pitch
Vendor pages describe quality in adjectives. Accuracy figures often come with no definition and no sample size, and the "top vendors" lists you find in search are usually written by a vendor that ranks itself first. Treat all of it as marketing until you have the same answers from each vendor, in writing.
Sama's own buying guide puts the principle well: buy outcomes rather than tasks, evaluate the whole system rather than the people on the call, and require proof. Shortlist three to five vendors, send each the same questions, and drop any that won't answer in writing.
The 10 answers to get in writing
| Ask | A good answer shows | Worry if |
|---|---|---|
| 1. Who will do our work, and how are they trained on our rubric? | Named roles, a calibration round on your gold set, and the same people over time | "Our global crowd" with no calibration step |
| 2. How do you measure agreement, on what sample? | The statistic, the sample size and what happens below the bar | A percentage with no definition |
| 3. How are disagreements settled? | A second reviewer or lead decides, and the reason is recorded | Majority vote with no record |
| 4. What does the unit price leave out? | Tooling, review, rework, project management and minimums, each named | One number and "it depends" |
| 5. What triggers free rework, and how fast? | A clear definition of wrong and a fix time in days | Rework billed as new work |
| 6. When would a 500-item pilot start and finish? | Two dates, in writing | Different start times on different pages |
| 7. Who owns you, and who else can see our data? | Owners named; access limited to the people on your work | No answer about investors or subcontractors |
| 8. Which certifications, and can we see the report? | The report itself, with its date and scope | Badges only, or "in progress" |
| 9. How is our data deleted at the end? | A deadline and a written confirmation | "We keep it for quality purposes" |
| 10. What share of the rate reaches the people doing the work? | A plain answer about pay and conditions | A refusal to discuss it |
Question 2 matters most for LLM work. What a good agreement score looks like, and how to set one, is in our RLHF guide. Question 10 is about quality as much as fairness; the evidence is in our data annotation guide.
Red flags in a vendor's pitch
- Promises perfect accuracy. Human judgement always disagrees a little. A vendor that claims none isn't measuring.
- Won't run a pilot. A vendor confident in its work will do a small paid batch on your own items.
- Vague about process. You can't get a plain description of who reviews, who checks and who decides.
- Accuracy with no method. A figure with no definition, sample or date tells you nothing.
- Dates that keep moving. One page says a week, another says ten days, and the call says "a few days".
- No one who knows your field. Medical, legal or code data needs reviewers who understand it, not only general raters.
Two or more of these: keep looking, or make the pilot the only commitment.
Run the pilot before the contract
- Build a gold set. 50 to 100 items you have answered yourself, mixed in without the vendor knowing which ones.
- Pilot two or three vendors at once, on the same items, for two to four weeks.
- Audit a sample of every batch: 5 to 10% is a common starting point. Compare against your gold set and your own review time.
- Write the rework rule into the pilot: what counts as wrong, and how fast it comes back fixed.
- Agree deletion now: for example within 30 days of the end, with written confirmation.
- Decide on numbers: agreement on the gold set, rework rate, delivery against the promised dates and the hours your team spent checking.
A ready-made pilot plan for rating and review work, with task sizes and pass marks, is in our RLHF guide.
If the data includes personal information
India's Digital Personal Data Protection Act, 2023 calls a vendor that processes personal data for you a data processor, and you the data fiduciary. The DPDP Rules were notified on 14 November 2025 with an 18-month phased start. The fiduciary can be fined up to ₹250 crore for failing to take reasonable security safeguards, so your contract should require them of the vendor.
SOC 2 reports cover up to five areas: security, availability, processing integrity, confidentiality and privacy. Ask which ones a vendor's report covers. For data from EU users, our RLHF guide covers the GDPR and EU AI Act points.
Share only what the task needs. Strip names, emails and account numbers before data leaves your systems, wherever the work allows.
What we put in writing
HerWorkCircle is a managed team of skilled women across India doing AI rating, labelling and data checking, in English. Before you pay anything we agree the work, the price per piece or per hour and the rework terms in writing, then run a pilot on your own items. Every piece is checked by a second person, and wrong work is redone free.
Questions
How do I choose a data annotation vendor?
Shortlist three to five, send each the same written questions on people, agreement, price, rework, ownership and deletion, then run a paid pilot on your own items before a long contract.
What questions should I ask a data labeling vendor?
Who does the work and how they are calibrated, how agreement is measured, what the price leaves out, what triggers free rework, who owns the vendor, which certifications they hold and how data is deleted.
How long should a data annotation pilot run?
Two to four weeks on the same items for each vendor is common, long enough to see agreement, rework and delivery against promised dates.
What is a gold set?
A set of items you have already answered yourself, mixed into the vendor's work so you can measure their accuracy against your own judgement.
What are red flags in an annotation vendor?
Claims of perfect accuracy, refusing a pilot, a vague process, accuracy figures with no method, start dates that keep changing and no reviewers who know your field.
Does the DPDP Act apply to data labeling vendors?
If the data includes personal data of people in India, yes: the vendor is a data processor and you are the data fiduciary, responsible for reasonable security safeguards.
Sources
- Data labeling vendor evaluation guide Sama
- Data annotation services: a buyer's guide iMerit
- How to choose a data annotation service provider CVAT, April 2026
- How to run a pilot before committing to an annotation vendor Label Studio
- SOC 2 and the trust services criteria AICPA and CIMA
- Digital Personal Data Protection Rules, 2025 (explainer) Press Information Bureau, November 2025
- Hire expert LLM evaluators Shaip