What happens to a video after a new team uploads its first clip? The route from raw footage to a training-ready export includes format processing, project settings, AI annotation, human review, and export formatting. Choices at each stage affect the final dataset. This walkthrough follows the sequence and notes where those choices matter most.
Step 1: Upload and Format Processing
Vision Lab accepts MP4, AVI, MOV, and MKV files up to 4K resolution. Upload processing checks the format and extracts frame count, frame rate, resolution, and color space. Those details appear in the project dashboard right away, giving you a chance to verify the clip before annotation settings are configured.
Check distortion correction during upload. Most fixed overhead cameras in factories and warehouses can leave it off. Robot-mounted cameras with wide-angle or fisheye lenses often need correction, since lens geometry can shift coordinates near the edges and reduce training quality. If correction is skipped for that specific robot-mounted wide-angle case, edge-near bounding boxes will usually be systematically misplaced. This advice does not apply to every camera, especially a calibrated narrow-angle setup.
Upload recordings in 10 to 15 minute segments instead of one uninterrupted file. Long clips can process correctly, but they take longer and make condition metadata harder to assign. Shorter clips let you record lighting state, task phase, and shift time per segment. Later, those fields make condition filtering and coverage checks much easier.
Step 2: Taxonomy Configuration
Taxonomy setup has one of the clearest effects on later annotation quality. Before the AI pass, define the project classes, meaning the objects to detect, plus action event types when temporal annotation is needed.
Each class needs a name and short definition, with 1 or 2 example keyframes optionally attached. Those examples steer annotation toward the visual instances in your footage rather than a generic category. A pallet in one facility may differ substantially from a standard training-set pallet, and representative keyframes help narrow that mismatch.
Naming deserves deliberate attention. A class called "robot-arm" is sufficient when footage contains one arm type. If the facility has 2 visually distinct arm types that the model must separate, create separate classes and name the distinction clearly. Preventing confusion at taxonomy time is easier than repairing it after labeling. Spending 30 minutes on definitions before a run can save hours of correction.
For an action event, specify its name, observable start condition, and observable end condition. Clear boundaries make temporal tags more consistent from frame to frame. "Grasp starts when the robot begins grasping" is too vague and creates boundary noise. "Grasp starts when gripper fingers are at least 20% closed" gives reviewers a condition they can label consistently.
Step 3: The AI Annotation Pass
Once the taxonomy is ready, Vision Lab processes the uploaded clips. A 10-minute clip at 30fps contains 18,000 frames, and the pass generally takes 25 to 40 minutes. Runtime varies with class count and with whether segmentation masks are requested alongside bounding boxes.
The pass returns 3 things: bounding boxes or segmentation masks for each frame, a confidence score for every annotation instance, and a flag for any frame containing an instance below the project threshold. Flagged frames enter human review.
Set the confidence threshold per project. The default is 0.65, so a frame with an instance below 0.65 is flagged. There is no universal right setting. With a fixed overhead camera, 3 classes, and stable lighting, 0.65 may be overly cautious. With robot-mounted footage, changing light, and 8 classes, the same threshold may miss too much uncertainty.
Start with a 500-frame pilot at the default threshold. Review the flagged frames manually, separate real annotation errors from cautious flags, then adjust the threshold before processing the full batch. This calibration usually takes about 30 minutes and can materially shrink the later review queue.
Step 4: Human Review Queue
After annotation, flagged frames are ordered by ascending confidence score. The lowest-confidence items appear first, putting the least certain annotations ahead of borderline cases.
Each queue item displays the frame with annotations overlaid. Accept keeps a correct result. Edit opens the annotation editor so you can fix or add labels. Correct an incorrect frame instead of discarding it, because discarded frames create temporal gaps that the coverage report will expose.
In factory and warehouse footage, about 60 to 70% of frames below the default threshold are often correct and need no change. The other 30 to 40% may contain a missed object, a wrong class, or a box edge misplaced by more than 15%. At 25 accepted frames per minute and 8 corrected frames per minute, clearing 300 flagged frames takes roughly 45 to 60 minutes. This estimate has held reasonably across scene types in our internal testing, but unusual motion or occlusion can push it higher.
If more than 50% of flagged frames require edits rather than acceptance, revisit the taxonomy or threshold. That correction rate usually indicates genuine uncertainty around the class boundaries. Adding representative keyframes and tightening definitions often lowers corrections on later batches. This approach is a poor fit when every frame requires specialist judgment that cannot be expressed through visible rules.
Step 5: Class Coverage Report
Before export, the dashboard summarizes labeled instances per class, frames per class, and any class below its configured minimum. Classes under the minimum appear in amber.
Review that summary before downloading. If the plan called for 1,500 instances of a safety-critical class such as "emergency-stop-zone" but the report contains 420, the shortfall matters even before its cause is known. The footage may contain fewer examples than expected, or annotation may have missed many instances. Both causes can be investigated, and finding the gap at dataset review is cheaper than discovering it during model evaluation.
Minimum instance counts are configurable by class. Set conservative minimums for safety-critical or operationally important classes, even when they appear infrequently. That gives the report a useful warning signal instead of a blanket green result.
Step 6: Export
Export creates a structured file in the selected format: COCO JSON, YOLO TXT, or Pascal VOC XML. For action events, COCO JSON adds a temporal_events key with frame ranges for each event type, following the extension pattern used by several computer vision frameworks.
Send the package to an S3 bucket or GCS path through a signed URL, or download it from the dashboard. The package contains a dataset manifest JSON with per-class statistics, the chosen annotation file, and a SHA-256 hash of the export content for version tracking.
COCO JSON works with PyTorch DataLoader. Minimal loading example:
from torchvision.datasets import CocoDetection
dataset = CocoDetection(
root="./frames/",
annFile="./annotations/instances.json"
)For a 10,000-frame batch, upload through an export-ready dataset typically takes under 48 hours, including annotation and a typical review queue. Review time is the main variable. A calibrated threshold and clearly defined taxonomy can reduce it substantially from the default setup, while difficult footage can extend the window.