When robotics teams first create video event labels, they often list whatever appears while reviewing footage: reach, pre-grasp, grasp, lift, transport, place, retract. That creates 6 or 7 labels for one pick-and-place cycle. Problems emerge during annotation, when two people disagree about where "reach" ends and "pre-grasp" starts, leaving the trained model uncertain about boundaries throughout the task.
The reverse error is just as common: reducing the entire task to active and idle. Annotation becomes consistent, but the model loses the intermediate structure needed for anything beyond counting cycles.
Useful taxonomies sit between these extremes. Reaching that middle requires working backward from the model's objective to labeling choices, rather than labeling forward from "what happens in the video."
Start With the Model's Intended Task
Before naming any event type, state the model's downstream job clearly. The appropriate taxonomy changes with the answer.
For task-completion recognition in an activity log, including cycle counting, throughput measurement, or verification, the key distinction is "action completed" versus "action not completed." A binary flag for each task type, with an explicit completion state, is often enough. Exact timing within the action matters less than detecting completion reliably.
Mid-action anomaly detection, such as gripper slippage, unexpected contact, or a bad approach angle, requires finer temporal resolution. An anomaly often appears in one particular phase, so that phase must be a distinct, labelable event. For a pick-and-place gripper, separating "approach" from "pre-contact," when the fingers are near but not touching the object, is where slip detection needs to trigger.
For learning from demonstration, labels should preserve decision points, the moments when the robot commits to a trajectory or grasp orientation. Those frames provide the supervision signals used by a learning-from-demonstration system.
Treat the event taxonomy as a design document for the model's objective. Define that objective first, then make the labels learnable.
Precise Temporal Boundaries Shape Label Quality
Imprecise temporal boundaries cause most disagreement between annotators. "Grasp" could mean when the fingers start closing, first touch the object, or secure it against loss. At 30 fps, those readings may differ by 8 to 20 frames, adding noise exactly where the model must learn the boundary.
For every event type, specify exact start and end conditions using observable visual states, not intent:
Rather than "grasp starts when the robot initiates the grasp motion," write: "grasp starts when gripper fingers are visible in at least 20% closed configuration." Rather than "lift ends when the object is clear of the surface," write: "lift ends when the object's lowest point is visibly above the surrounding surface plane by at least one object-height."
If independent annotators agree on 90% of boundaries under these definitions, they are specific enough. Below 85% agreement, split the event if it covers two moments, or merge it if the visual states cannot be separated without intent information absent from the video.
The Cost of Tagging Too Much
A single task cycle with 10 or more fine-grained event types can create transition ambiguity, a problem that may not appear in early training. At a boundary between visually similar events, such as "approach" and "pre-contact," separated by only a few centimeters of gripper travel, the model receives conflicting signals. Similar frames receive different labels, so the model learns that those visual states are ambiguous.
For temporal prediction, fuzzy event boundaries usually hurt more than having fewer event types. A 5-event taxonomy with precise, shared boundaries will almost always beat a 10-event taxonomy with vague ones, even when the larger taxonomy appears more informative in theory.
Fewer event types are not automatically preferable. The benefit of each added type must be weighed against the work of defining a boundary that annotators can label consistently. If agreement on the boundary cannot be achieved, it is a preference, not a defined event.
Build the Taxonomy in Iterations
A reliable workflow starts with 4 to 6 event types based on clear, visually distinct boundaries. Label a pilot of 500 to 1,000 frames, measure agreement for each boundary, and revise any event below 85% agreement before labeling the full dataset.
This exposes unclear boundaries before weeks of annotation are lost. It can also show which distinctions should be merged. For cycle counting, the model may not need separate "transport" and "pre-placement descent" labels, so combining them reduces complexity without removing useful signal.
Pilot labeling can reveal a different limitation: the footage may not show the action that matters. One pick-and-place team mounted cameras at conveyor height and found the gripper almost fully hidden behind the conveyor wall at object contact, the key event for its anomaly detection system. A "contact" label was conceptually right but impossible to assign from that footage. That is a camera-placement issue, not a taxonomy issue, and a 500-frame pilot should expose it before 50,000 frames are labeled.
Background Events and Context
Industrial footage often includes continuous processes beside the robot: moving conveyor belts, traveling overhead gantries, and cycling indicator lights. These may be useful context labels or mere noise. Their value depends entirely on the task.
Ask whether this background state changes how you interpret the robot's main action. If a gantry passes over the workspace and the robot pauses or changes trajectory in response, the gantry state is worth tagging. If a conveyor always runs at constant speed and does not condition robot behavior, labeling its state adds no signal.
A simpler check is to remove the background tag and ask whether any prediction would change on a frame where that event is present or absent. If no concrete example exists, do not label it.
This guards against annotating everything "just in case." Each added type costs labeling time and can create another ambiguous boundary. Include background events only when the tag connects directly to the model's objective.
A Practical Place to Start
For a first taxonomy covering pick-and-place or assembly, 4 events handle most initial needs without unnecessary boundary confusion: approach, the arm moving toward the target; grasp or engage, the gripper-defined contact phase; transport, the object held while the arm moves; and release or disengage, the object placed while the gripper clears it.
This structure is usually easy to label consistently, covers phases relevant to anomaly detection and cycle counting, and can expand as requirements become more specific. Testing a model with 4 events before jumping to 9 is generally the safer choice.