An annotator reviewing a factory clip may see a part enter from the left, follow the belt, pass beneath a sensor or camera, and leave on the right, then watch the same sequence repeat. That repetition is the useful signal. Instead of asking a person to redraw every box, we learn the recurring motion and use it to carry labels through the footage, while reserving manual review for frames that do not fit the pattern.
Why Assembly Line Geometry Helps
Assembly lines are easier to annotate automatically than natural-scene video because their structure is constrained. The camera stays fixed, object classes are defined ahead of time, and a project commonly has between five and fifteen distinct labels. Objects also tend to follow known paths, so their likely location in the next frame is easier to estimate.
Natural-scene footage is different: the camera moves, objects can appear anywhere and at any orientation, and the scene changes from frame to frame. Labels such as "person" or "car" also cover substantial visual variation. Those conditions remove much of the spatial and temporal predictability that makes assembly line footage workable for this approach.
That observation determines our pipeline. We do not treat every frame as an isolated detection task. We represent the line as a system with known motion, then annotate against that system.
Trajectory Label Propagation
We begin by extracting trajectories rather than detecting objects independently in every frame. Dense optical flow runs over short clips, typically 30 seconds, to reveal the main motion vectors in the scene. We cluster those vectors to separate recurring paths, then use the resulting motion structure to guide box placement.
A standard conveyor setup usually produces two to four dominant trajectory clusters. After building that motion model, detections can move forward across frames with position priors. If a detector locates an object at position (x, y), the system predicts where it should appear next based on the object's path and the local motion. A 1 object detection therefore supplies more than a label for one frame.
For a 10-minute clip at 30 fps, the footage contains roughly 18,000 frames. Calibration labels from a human annotator on 200 to 400 key frames provide the anchor points for propagation. Labels then carry forward through the remaining frames automatically, with low-confidence results separated for review rather than accepted without inspection.
Building the Confidence Queue
Propagated boxes do not all deserve equal trust. We score each one using optical flow consistency inside the box, detector confidence from the nearest valid detection, and agreement with the trajectory model. In one configuration, scores below 0.75 are treated as uncertain, while 0.90 indicates a strong match.
Boxes under the selected threshold enter the review queue. On a well-configured assembly line project, roughly 8 to 15 percent of frames reach that queue, which is the portion a human annotator actually sees. The threshold can be set per project, since a line with more occlusion or changing illumination needs a different tolerance from a stable setup.
The queue displays the frame, the propagated box, its confidence score, and a flag showing which signal contributed most to the low score. A reviewer can accept the box, move its edges, fix the class label, or complete the annotation manually when propagation is not usable.
Where the Method Breaks Down
We want to be clear about the limits. Assembly line footage can contain each of these failure modes, so they belong in the project plan.
The most frequent problem is an SKU changeover. When the line switches part types during a shift, the trajectory model has no prior behavior for the new object. The detector also lacks a calibrated representation of that class, so the resulting boxes and confidence scores can fall until new calibration labels are supplied.
Occlusion at transfer points is another weak spot. When belt sections meet or an object moves between conveyors, its position may be unclear for several frames. Propagation can then place a box incorrectly around partially overlapping objects, which is why those frames are routed to review.
Lighting changes create a third failure mode. Bay doors opening, shift transitions, or overhead lights cycling can alter the image appearance. The detector was calibrated on a different distribution, so its confidence may drop and optical flow may no longer agree with the learned path. These frames need inspection instead of automatic acceptance.
These cases do not create quietly wrong labels in the workflow. They become additional review entries, so automation loses efficiency gradually instead of stopping outright.
The Annotator's Actual Work
Trajectory propagation and confidence filtering change the task from repeated drawing to focused judgment. A reviewer mainly decides whether a proposed box is acceptable, then adjusts boxes or completes annotations when the evidence is weak.
In our early-access pilots, annotators on well-configured assembly line projects spend roughly half their time accepting or rejecting boxes, about 30 percent adjusting them, and 20 percent manually annotating flagged frames. The exact balance shifts with lighting, occlusion, and taxonomy quality, but the work is no longer a redraw on every frame.
Taxonomy Setup Has Outsized Impact
Taxonomy definition is the configuration choice with the largest downstream effect, and it happens before annotation begins. If the mounted camera cannot reliably distinguish two proposed classes, the detector cannot consistently separate them either. Class boundaries must match what the footage actually shows.
Here is a useful check: if two annotators sometimes choose different labels for the same frame, refine the taxonomy before starting automation. A detector cannot settle an ambiguity that people cannot resolve from the available view.
The strongest configuration combines a clean taxonomy, stable lighting, and limited SKU diversity. In that setting, trajectory propagation typically cuts total annotation labor by 55 to 70 percent compared with drawing every frame from scratch. More complex environments reduce the benefit and increase the share sent to human review.