The team arrived with a plan for four dedicated annotators to label approximately 120,000 frames from distribution-center pick-and-place footage over three weeks. The budget and project owner were already in place. Their question was whether our pipeline could match the expected quality within that effort, not whether the schedule needed cutting.
We delivered the first training-ready dataset in eight days. One annotator spent about three to four hours each day in the review queue. That reduced calendar time by roughly 60 percent and lowered total labor substantially. This article breaks down the changes behind that result, along with the cases they did not solve.
The Initial Setup
The site was a good candidate for structured annotation. Each pick station used four fixed overhead cameras at 2.4 meters, with one camera covering each of four zones on a belt-driven bin carousel. Lighting came from stable overhead LEDs. The label set was limited to five visible classes: conveyor bin, target object type A, a boxed consumer product, target object type B, a flexible pouch, and the robotic gripper in open or closed position.
Five classes, four fixed cameras, and steady lighting made this a strong candidate for AI-assisted annotation. The original estimate ignored that pattern, treating the work as a generic large labeling job based on past experience.
Why the First Plan Took So Long
The initial workflow had annotators draw every bounding box from the beginning, one clip after another, in a general labeling tool. At roughly 200 to 400 labeled frames per annotator per hour for multi-class boxes, 120,000 frames meant 300 to 600 annotator-hours. Four annotators working six hours daily would need about 12 to 25 days.
That estimate fit the workflow. The workflow failed to fit the footage.
In a fixed-camera pick-and-place cell, most appearance changes follow repeatable motion. The gripper travels along a defined arc, bins follow known paths into and out of the pick zone, and the same background and objects recur. Drawing each frame independently throws away that temporal structure instead of using it to narrow the review task.
Three Workflow Changes
First, we reduced the taxonomy. The original seven classes included three states of one object, "gripper approaching," "gripper at contact," and "gripper retracting." We combined those into five positional classes and removed distinctions that the camera resolution could not consistently support. Fewer visually clear classes reduced ambiguity for the annotator and confusion for the detector.
Second, we used motion trajectories for propagation. We manually labeled 350 frames sampled from all four camera views and three SKU configurations during calibration. Those anchors supported models for the gripper arc and bin paths. The system then carried boxes forward along those paths and attached a confidence score to each result.
Third, we split review into confidence tiers rather than showing every uncertain frame in sequence. Tier one held boxes between 0.65 and 0.80, which were usually right and needed a quick check. Tier two held boxes below 0.65, where adjustment or full relabeling was more likely. Tier one moved at about three times the speed of drawing boxes from scratch, while tier two received slower, closer review.
The Review Queue in Practice
From 120,000 frames, propagation produced boxes above 0.80 confidence for approximately 107,000 frames. They went to export without individual review, subject to a final sample audit. The other 13,000 frames entered review, with about 9,000 in tier one and 4,000 in tier two.
Tier one averaged roughly 450 frames per hour. Tier two was closer to 120 frames per hour. Across eight working days, human review took approximately 25 hours, scheduled with breaks so the annotator could maintain attention and label quality.
The closing audit sampled 500 high-confidence frames at random. The acceptance rate was 94 percent, so 6 percent were adjusted. That result sits within the normal range for production annotation and was comparable to the agreement levels we see in fully manual workflows.
What Stayed Manual
Two frame groups still needed complete manual labels and received no speedup from propagation.
The first was grasp failure footage. A missed pick or sliding bin moved objects away from the modeled path, sending propagation confidence close to zero. These cases made up roughly 3 percent of the footage, but they are valuable examples because a pick-and-place model must account for failed grasps as well as successful ones. We marked them separately and sent them through priority annotation instead of leaving them in the general queue.
The second group was the first 50 frames after each SKU changeover. Once the carousel switched products, the trajectory model had no reference for the new configuration. The annotator labeled these frames manually and created fresh anchors before propagation restarted.
Together, those groups represented about 8,000 frames, or 6.5 percent of the dataset. Every one required ordinary ground-up annotation.
The Limitations
The limit here is the setup: fixed cameras, steady light, few classes, and repeated frames. A 60 percent reduction may fit similar cells, but not mobile cameras, changing light, broad taxonomies, or frequent SKU changes.
The eight-day result also depended on about one day of setup for taxonomy review, calibration labels, and trajectory configuration. Teams that skip that session may move more slowly on the first batch, even if later batches improve once the calibration is in place.
For robotics teams focused on pick-and-place, the useful gain is not simply moving from three weeks to one. The harder question is whether the dataset includes grasp failures, SKU transitions, and unusual object positions. In this pilot, once predictable frames were filtered out, annotation effort shifted toward checking those less common cases.