It is easy to assume that better detection makes better picks, but our bin-picking comparison showed a different result. Two annotation passes on the same 8,000-frame dataset from one early-access pilot team produced a box-trained model and an instance-mask model. Detection accuracy was nearly tied: 87 percent for boxes and 89 percent for masks. With irregular objects, pick success was 66 percent for the box model and 88 percent for the mask model. The label choice alone created that 22-point difference.
The choice is less about detection scores than about the output the robot's next task actually needs.
What Boxes Capture
A box locates an object within the limits of its enclosing rectangle. It answers, "Is something here, and roughly where?" That is often enough for navigation, path planning, and other coarse spatial decisions.
A box says nothing about an object's contour, its exact pixel coverage, or its orientation inside the rectangle. An L-shaped conveyor part and a cylindrical pallet pin can therefore produce boxes with similar centers and sizes. The downstream planner receives the same spatial description even though their grasping needs differ.
The shortcoming is modest when an object is rectangular or otherwise regular and the box closely follows its boundary. Its center can approximate a pick point, and a sufficiently specific object class may provide orientation cues. With irregular geometry, partial occlusion, or a major protrusion in one direction, the rectangle hides more than it reveals.
What Masks Contribute
An instance mask marks each pixel as belonging to the object or not. For a complex part on a conveyor, it follows the object's contour instead of stopping at the smallest enclosing rectangle. That boundary can support grasp-point estimation, such as locating the centroid of the support polygon or selecting stable contact surfaces.
Masks also represent overlap differently. When objects cover one another, a box detector has to assign the shared area to a box. Instance segmentation can distinguish each object's extent pixel by pixel. That matters in bin picking, where parts commonly pile over one another.
Masks also make shape changes visible over time. Comparing predictions across frames can show whether a surface-area change points to tipping, rotation, or partial occlusion by another object. A box detector supplies only center and size, so separating those cases requires extra reasoning.
When Boxes Make Sense
For a mobile robot moving through a warehouse or factory, the practical question is where obstacles and clearance zones sit and how much room remains for maneuvering. Boxes answer that question with lower annotation cost and faster runtime inference. The robot does not need a pallet's pixel-level outline to route around its approximate position and size.
Boxes also fit presence checks such as "is this bin empty or occupied?", whole-object quality categories that do not depend on shape, and conveyor counting. In these cases, adding masks raises annotation effort without supplying information the task uses.
Our trajectory propagation pipeline also moves boxes more quickly. A propagated box updates four values: x, y, width, and height. A propagated mask must follow rotation, deformation, and changing partial occlusion. For high-volume labeling of straightforward detection tasks, boxes are much more efficient from annotation through export.
When Masks Earn Their Cost
Irregular-object pick and place is the clearest mask use case. When grasp planning uses model output to estimate contact location or gripper orientation, the object boundary matters. For a non-rectangular part, the box center may be a poor pick point, while the box dimensions cannot show the direction of its long axis.
Assembly insertion, surface finishing, and screw fastening likewise need enough shape information to position the end effector precisely. Their tight tolerances make the 5 to 10 percent grasp-success improvement masks typically provide over boxes operationally meaningful.
Tight-clearance collision avoidance is another example. When an arm reaches into clutter and planning depends on accurate object extent, a coarse box can create needless detours or underestimate size and cause a collision. A mask gives the planner a closer description of occupied volume.
The Annotation Cost Gap
Mask labeling takes three to five times longer per object than box labeling. On a 10,000-frame dataset averaging four objects per frame, that becomes tens of thousands of extra annotation hours at manual rates.
At Vision Lab, mask propagation narrows that difference. We carry contours forward with optical-flow warping instead of box propagation and require a higher confidence threshold before accepting a mask, since mask mistakes are harder to spot during a quick review. On assembly-line footage with stable objects, mask labor costs roughly twice as much as box labor, not five times. It remains the more expensive option.
Calling an all-mask policy efficient is misleading. The real constraint is whether a class affects manipulation or only coarse detection. Use masks for manipulation-critical classes, and use boxes when the model output supports navigation or presence checks.
A Practical Framework
For most robotics projects, the workable answer is a class-by-class mix rather than all boxes or all masks. Find the classes that the downstream planner uses for shape-sensitive decisions, including grasp-point estimation, tight-clearance routing, and contact-surface selection. Give those classes instance masks, and label the rest with boxes.
This hybrid setup preserves mask quality where manipulation depends on it while keeping the overall labeling burden manageable. In Vision Lab, each class can receive its own annotation type in the project taxonomy, allowing both modes to run in one project export.
If it is still unclear which classes the planner will use for shape-sensitive decisions, begin with boxes everywhere. A later mask pass can target particular classes. Starting with masks on every class and then learning that half served only coarse detection creates an expensive change that is difficult to reverse.