How to staff an AI data labeling project
A practical guide for AI teams: how to scope the work, choose the right people, run a pilot, keep quality high and scale up and down.
1. Scope the work before you hire anyone
- Write down the task types (labeling, ranking, writing examples, safety review), the volume, the deadline and how you will measure quality.
- Estimate hours per item from a small sample you do yourself. That number drives headcount more than anything else.
- Decide what data reviewers will see and what devices and access controls they need.
2. Decide between generalists and subject-matter experts
- Generalists suit tasks anyone can judge: is this image a car, is this sentence polite.
- Experts are needed when judging the answer takes training: a weld defect, a medication question, a motor fault, a legal clause.
- Many projects use both: experts write and audit, generalists do volume work under expert review.
3. Write guidelines and a calibration set
- Guidelines should show worked examples of good and bad answers, not just rules.
- Build a gold set of items with agreed answers; every new reviewer completes it before live work.
- Expect to revise the guidelines after the pilot.
4. Run a paid pilot
- Start with a small team for one or two weeks.
- Measure accuracy against the gold set, agreement between reviewers and time per item.
- Keep the reviewers who meet the bar; use what you learn to size the full team.
5. Keep quality high at scale
- Mix gold items into live work and track each reviewer’s score.
- Have senior reviewers audit a share of every reviewer’s output.
- Retrain or remove reviewers who drift; reward the ones who stay accurate.
6. Plan for scaling up and down
- Labeling work comes in waves. Use a staffing partner that can add and release people quickly without long notice periods.
- Keep a bench of calibrated reviewers you can call back for the next project.
This guide is general information, not legal advice. Review your plans with your own counsel.
Questions
How many reviewers do I need?
Divide total items by the items one reviewer completes per week at your quality bar, then add a margin for ramp-up and turnover. A pilot gives you the real numbers.
Can Echelon supply both generalists and experts?
Yes. We recruit general annotators and verified experts, such as tradespeople and clinicians, and manage scheduling and quality checks.
Need workers?
Tell us what you need and we’ll respond within 24 hours. Crews can be on site in as little as 3 days.
