At 5,000 manually labeled frames, a model may work on routine footage yet fail on a stubborn group of edge cases. The obvious answer is more data. The common wrong answer is hiring a full annotation team before knowing which data will fix the problem.
Annotation is not the main issue. Timing is. Before product-market fit, a team that builds a complete annotation setup is optimizing for a problem it has not defined closely enough. It may not know which classes need more examples, which conditions matter, or whether the failure comes from the taxonomy. A larger label count alone is not enough. It is only useful after the missing coverage and the real failure mode are understood.
The quicker route is to reach 10,000 well-labeled frames at low cost, deploy, watch what breaks, and use those observations to choose the next labeling sprint. This is how a two-person engineering team can handle the work.
Set the Frame Budget Before Labeling
10,000 frames can look substantial until you check class coverage. For a warehouse robot with 8 object classes appearing at different rates, balanced coverage requires a class-by-class budget.
Suppose emergency stop zones occur in 3% of frames. In a 10,000-frame dataset, that is about 300 instances. If reliable detection requires at least 500, the gap exists before labeling begins. The total frame target therefore needs to reflect the rarest classes, not just the overall count.
Before labeling, make a rough coverage table. For every class, estimate its frequency in the footage and calculate the total frames needed to meet its minimum target. Also record the confidence range you will accept, such as 0.60 to 0.70. This simple projection shows whether 10,000 frames is sufficient or whether the sample needs to be weighted toward rare conditions.
Sample for Variety, Not Just Time
Most robotics teams record continuously, creating long stretches of repeated footage. A 3-hour session at 30fps yields more than 300,000 frames. A random 3% pull will mostly capture ordinary operation and leave unusual states underrepresented.
To get 10,000 useful frames, sample for diversity rather than elapsed time. First tag footage segments by condition: bright, dim, or glare lighting; approach, contact, transport, or release phases; and the other operating conditions visible in the recordings. Pull from each group instead of treating every moment as equally informative.
Reviewing two to three hours of footage for this pass pays back quickly. You remove duplicate frames, spread coverage across conditions, and expose rare edge cases that a random sample may barely contain.
For 8 object classes under changing lighting, a condition-stratified set of 10,000 frames will usually make a stronger first model than a random 15,000-frame sample drawn from the same footage pool.
Let AI Handle the Predictable Majority
Factory and warehouse video often repeats the same scene, especially with fixed cameras. A gripper above a conveyor, viewed from one angle in steady lighting, can look much the same in frame 100 and frame 8,000. Those predictable examples are a good fit for an AI annotation pass, with people checking the uncertain cases.
Start by defining each class precisely and, when available, supplying a few example keyframes for each one. Run the AI annotation pass over the sampled frames, then send low-confidence results and unusual conditions into a review queue. Keep the automatic output only when it matches the class definition.
With a tight taxonomy and a condition-stratified sample, 70 to 80% of frames may clear the confidence threshold without human annotation time. For a 10,000-frame target, that leaves about 2,000 to 3,000 frames for review or correction.
Make the Review Queue the Work Unit
In an AI-assisted workflow, the human unit is not an empty frame. It is an AI-generated label that needs acceptance or correction. Checking those proposed annotations is three to five times faster per frame than drawing every box from the beginning, provided the taxonomy and predictions are usable.
At 20 to 25 frames per minute, accepting or correcting labels, a queue of 2,500 flagged frames takes about 100 to 125 minutes. Split between two engineers during a two-week sprint, that works out to roughly 50 to 60 minutes per person each week.
Review speed improves when the taxonomy is precise. Spend 2 hours at the start of a sprint reviewing and refining definitions against a pilot batch of 50 frames. Adjust the rules for the errors the AI actually makes, then apply the revised setup to the remaining queue.
Label Failure Conditions Before General Coverage
When a model has a known failure at 5,000 labeled frames, adding another 5,000 random frames may enlarge the dataset without addressing it. Start with the condition in which the failure appears.
Pull 200 to 500 frames from the relevant lighting, object orientation, and camera angle conditions. Label class boundaries carefully, retrain, and evaluate before spending the rest of the budget on broad coverage.
If targeted labels improve performance, you have learned what kind of data the model needs and can direct the remaining budget accordingly. If performance does not move, the bottleneck may be elsewhere, such as the taxonomy, footage, or model rather than the frame count.
We have seen teams spend three weeks labeling general coverage, then find that the failure was a class-definition problem. Two visually different objects had been merged into one class, so more examples of the same definition could not resolve the issue.
Know When Two Engineers Are Not Enough
For two engineers who are also building the software, a sustainable pace is about 1,000 to 1,500 reviewed frames per week with AI-assisted annotation. At that rate, reaching 10,000 frames takes six to eight weeks alongside development work.
The warning sign is iteration speed, not the raw frame total. You have outgrown this cadence when retraining produces new failures faster than the team can clear the labeling queue. At that point, the backlog, rather than the dataset size, is the constraint.
Until that point is clear, scaling the annotation operation is premature. In an early-stage setting, the useful goal is a model that works well enough in the target environment to reveal the next requirement. A carefully assembled 10,000-frame dataset with failure-focused follow-up can be more informative than a 50,000-frame collection built without a clear sampling plan.