Perception for Manipulation

Grasp planning computes force closure against a known object pose. Notice what that assumes: somebody upstream has already said where the object is, in the robot's own frame, to a tolerance the gripper can act on. That upstream somebody is the perception pipeline, and this is the module about it.

Six stages, in a fixed order, each consuming what the last one produced. Calibrate the camera so a pixel means a direction. Calibrate it against the robot so that direction means something in the robot's frame. Measure depth so the direction becomes a point. Detect and segment so points become an object. Estimate pose so the object becomes a graspable rigid body. The instrument below combines three adjustable error magnitudes by root-sum-of-squares as an illustrative modelling choice. It does not establish that calibration, depth and pose errors are statistically independent, or that its output bounds the positioning error of a real pipeline.

The budget the pipeline has to fit inside

Start at the end, with the number the whole chain is judged on. Three of the stages contribute a positioning error at the gripper: the hand-eye transform, the depth measurement, and the pose estimate. Two of those are fixed offsets. The hand-eye residual is an angle, so it costs lateral millimetres in proportion to how far away the object is. With the default opaque target and depth and pose settings, set the hand-eye slider to half a degree and drag the working distance out: a small contribution at 15 cm becomes the largest modeled contribution, while the far-end verdict is marginal, not will jam.

Compare the composed number against the tolerance real tasks hold. The clearance band above is a parallel-jaw gripper's lateral slack, and even a well-calibrated pipeline sits close to it. Precision insertion is a different world: assembly results in this literature are reported at 0.5 mm clearance Lum 2026↓Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?Tyler Ga Wei Lum, Kushal Kedia, C. Karen Liu et al., 2026, an order of magnitude below what a vision chain delivers alone. That is not a sensor defect. It is why precise assembly closes on force and contact rather than on vision, and why grasp planning treats the pose it is handed as an input rather than a certainty.

The rest of the module is the six stages, in order, and what each one contributes to that budget.

Calibration: what a pixel means

A camera is a device that discards a dimension. It maps a 3D point onto a 2D sensor, and the mapping it uses is described by the intrinsicscamera intrinsicsThe parameters that describe the camera itself rather than where it sits: focal length in pixels along each axis, the principal point where the optical axis meets the sensor, and the lens distortion coefficients. Zhang's method recovers them from a handful of views of a planar target held at arbitrary unknown orientations, which is why calibration on a robot is a printed checkerboard rather than a metrology rig. Without them a pixel is only a direction in an unknown parameterisation, so no image measurement converts into a metric ray.: focal length in pixels on each axis, the principal point, and the lens distortion terms. Before those are known, a pixel coordinate is a number with no geometric meaning.

Zhang's 2000 method uses a planar calibration target. Show the camera a planar target at several arbitrary orientations, none of them measured; the paper models radial lens distortion and gives a closed-form solution followed by nonlinear refinement under a maximum-likelihood criterion Zhang 2000↓A flexible new technique for camera calibrationZ. Zhang, IEEE Trans. Pattern Analysis and Machine Intelligence, 2000.

Zhang Zhang 2000↓A flexible new technique for camera calibrationZ. Zhang, IEEE Trans. Pattern Analysis and Machine Intelligence, 2000 compares this setup with classical techniques that use expensive equipment such as two or three orthogonal planes.

Extrinsics are the other half: where the camera sits relative to something else. Two cameras on a stereo rig need their relative pose. A camera and a robot need the harder version.

Hand-eye calibration: what a pixel means to the robot

A perfectly calibrated camera reporting perfectly accurate 3D points in the camera's frame is useless to an arm that plans in its own frame. Hand-eye calibrationhand-eye calibrationEstimating the rigid transform between a camera and the robot frame it must report into: for a wrist-mounted camera, the transform from the gripper to the camera, and for a fixed camera, the transform from the robot base to the camera. Tsai and Lenz posed it as solving AX = XB from a set of robot motions paired with the camera's observed motion of a static target, and their formulation is still what most implementations run. The residual matters asymmetrically: a translation error is a constant offset, while a rotation error is an angle, so its cost in millimetres grows with how far away the target is. is the step that connects them: for a wrist-mounted camera, the rigid transform from the gripper flange to the camera; for a camera bolted to a post, the transform from the robot base to the camera.

The 1989 papers on it share one form: move the robot through a set of known motions while the camera watches a static target, and each motion pair gives you an equation of the form $AX = XB$, where $A$ is the robot's motion, $B$ is the camera's observed motion, and $X$ is the unknown transform. Tsai and Lenz gave the efficient closed-form technique still used, solving rotation first and then translation from the accumulated constraints Tsai 1989↓A new technique for fully autonomous and efficient 3D robotics hand/eye calibrationR. Y. Tsai, R. K. Lenz, IEEE Trans. Robotics and Automation, 1989.

The instrument uses an illustrative ray-to-plane model, not a full three-dimensional hand-eye error model. Here $d$ is the separation along the normal to a target plane, and $\theta$ is the inclination of a ray relative to that normal. The lateral intersection offset used by the instrument is

$$e_\theta(d) = d \, \tan \theta$$

With $d$ expressed in millimetres, the local example at one degree gives about 1.7 mm at 10 cm and 17 mm at 1 m. These are evaluations of the authored formula, not measurements reported by the calibration paper or predictions that a grasp will succeed or fail. The formula is not a general norm of a rigid-transform error; nor is an exact tangent construction the same claim as a small-angle approximation.

Depth sensing: what fails, and on what

The examples below report specifications and limitations of named devices. They do not establish a universal accuracy or speed ranking across sensing technologies.

Stereo matches features between two images and triangulates. It needs texture: the matching is a correspondence search, so a featureless surface gives it nothing to correspond. The March 2026 RealSense D400-series datasheet lists ±2% Z-accuracy for D410/D415 and D43x at distances up to 2 m, with an 80% region of interest at HD resolution. Table 4-15 gives different range limits for other model groups: up to 4 m for D450/D455/D455f/D456 and up to 0.5 m for D401/D405. Depth accuracy compares valid pixels with ground truth. Factory KPIs reflect typical conditions and can change with the environment. The D405 protocol uses a textured target in typical room light (about 250 lux); the other models use a texture-less white target, default 150 mW laser power and auto exposure. Section 4.7 defines the measurement region as within 80% of the depth field of view; it is not a whole-image guarantee RealSense 2026↓RealSense Product Family D400 Series DatasheetRealSense, RealSense, as of 2026-08-22. RealSense's tuning checklist for the D415 and D435 says that D400 depth error scales as the square of distance. It advises moving closer where possible, but staying outside the minimum operating distance, MinZ. The guide also describes resolution and range trade-offs when changing MinZ, and notes that focus and different left/right views can limit very-near-range matching Grunnet-Jepsen 2026↓Tuning depth cameras for best performanceAnders Grunnet-Jepsen, John N. Sweetser, John Woodfill, RealSense documentation, as of 2026-08-22. A pattern projector is the standard mitigation, painting texture onto surfaces that have none Keselman 2017↓Intel RealSense Stereoscopic Depth CamerasLeonid Keselman, John Iselin Woodfill, Anders Grunnet-Jepsen et al., CVPR Workshops 2017.

Industrial 3D scanning. The PhoXi 3D Scanner L product page separately lists calibration accuracy of 0.200 mm (1 σ), temporal noise of 0.190 mm (1 σ), a scanning range of 870 to 2150 mm, and a scanning time of 250 to 2750 ms. The separate range field does not establish uniform accuracy throughout that range. These are the named model's published specification fields, not a family-wide ranking, a per-material accuracy guarantee or a robot control frequency Photoneo 2026↓PhoXi 3D Scanner LPhotoneo, Photoneo, as of 2026-08-22.

Time of flight measures the round-trip of modulated light per pixel. It does not need texture at all, which is its advantage, and it has a characteristic failure the other two do not: multi-path interference, where one pixel integrates light that arrived by more than one route. Microsoft's Azure Kinect DK documentation lists five reasons for invalid depth pixels: outside the active IR illumination mask, saturated IR signal, low IR signal, filter outlier, and multi-path interference. Such pixels are indicated by a depth value of zero, not a measured zero-distance surface. Under ambiguous depth, the page describes light reflecting from one wall onto another in a corner, and mixed foreground/background signals around object edges. It also notes that fast motion can increase edge invalidation because of the raw-depth exposure interval Microsoft 2019↓Azure Kinect DK depth cameraMicrosoft, Microsoft Learn, as of 2026-08-22, 2019.

The failure taxonomy that decides how a cell is built

The examples below distinguish material-related problems from weak signal, ambiguous stereo correspondence and view-dependent invalidation. The RealSense and Azure Kinect guidance describes particular sensors and setups, not a rule that every listed surface defeats every depth-sensing family.

• Transparent surfaces refract rather than reflect, so the sensor ranges whatever is behind them. Transparent objects "often appear as noisy or distorted approximations of the surfaces that lie behind them", which is the premise the ClearGrasp work was built on: it infers surface normals, transparent-surface masks and occlusion boundaries and then replaces the depth the sensor returned Sajjan 2020↓ClearGrasp: 3D Shape Estimation of Transparent Objects for ManipulationShreeyak Sajjan, Matthew Moore, Mike Pan et al., ICRA 2020.
• Specular reflections. Table 3-50 of the RealSense datasheet says they “May cause image saturation” on D400, while the D400f column says “Saturation mitigated”. Mitigated does not mean eliminated RealSense 2026↓RealSense Product Family D400 Series DatasheetRealSense, RealSense, as of 2026-08-22. The time-of-flight documentation lists saturated IR signal as an invalidation cause in its own right Microsoft 2019↓Azure Kinect DK depth cameraMicrosoft, Microsoft Learn, as of 2026-08-22, 2019.
• Dark surfaces and low signal. Azure Kinect can invalidate a pixel when its IR signal is too weak to generate depth Microsoft 2019↓Azure Kinect DK depth cameraMicrosoft, Microsoft Learn, as of 2026-08-22, 2019. RealSense's D415/D435 guidance instead relates D4xx depth quality to the input images and describes grainy images in low light. It warns about both underexposure and overexposure. Active illumination changes the setup: for its projected-pattern example, reducing background light while leaving the projector on can improve the pattern's signal-to-noise ratio Grunnet-Jepsen 2026↓Tuning depth cameras for best performanceAnders Grunnet-Jepsen, John N. Sweetser, John Woodfill, RealSense documentation, as of 2026-08-22. These are signal, exposure and illumination conditions, not a claim that every visibly dark object loses depth.
• Thin objects versus repetitive structures. RealSense names fences and wire grids as repetitive scenes in which left/right correspondence can become ambiguous. Its D4 processor compares the best and second-best match scores; similar scores can indicate aliasing, not necessarily two exactly equal matches. Increasing DSSecondPeakThreshold rejects more results as black or zero depth. This guidance concerns repetitive-pattern matching, not a blanket failure claim for every thin object Grunnet-Jepsen 2026↓Tuning depth cameras for best performanceAnders Grunnet-Jepsen, John N. Sweetser, John Woodfill, RealSense documentation, as of 2026-08-22.
• Self-occlusion versus view-dependent invalidation. In its multi-path discussion, the Azure Kinect documentation says that a surface area invalidated from one camera view may reappear from another. That is a view-dependent multi-path example, not a demonstration about generic self-occlusion or a guarantee that multi-view capture recovers every missing surface Microsoft 2019↓Azure Kinect DK depth cameraMicrosoft, Microsoft Learn, as of 2026-08-22, 2019.

Two things about the numbers above should be said out loud. The 2 percent is measured against a texture-less white target with the laser projector at default power RealSense 2026↓RealSense Product Family D400 Series DatasheetRealSense, RealSense, as of 2026-08-22, and the 0.200 mm publishes no target condition at all Photoneo 2026↓PhoXi 3D Scanner LPhotoneo, Photoneo, as of 2026-08-22. And no datasheet in this list publishes an accuracy figure for a transparent or a specular surface: that figure is not disclosed, which says something different from saying the sensor has no error there. It is why a research literature on transparent-object depth exists at all.

Once you have depth you have a point cloudpoint cloudA set of 3D points, usually with no ordering and no connectivity, which is what a depth camera or a lidar produces once its measurements are back-projected through the camera intrinsics. The awkwardness for learning is that the set is unordered, so a network reading it must be invariant to permutation of its own input. PointNet answered that with a shared per-point encoder followed by a symmetric pooling function. PointNet++ groups metric-space neighbourhoods and applies local PointNets recursively to learn features at increasing scales. These are summaries, not a guarantee that all geometry survives pooling; density-adaptive grouping addresses sparsely sampled neighbourhoods., and a network that consumes one has to be invariant to the ordering of its own input, since the set carries no canonical order. PointNet solved that with a shared per-point encoder and a symmetric pooling function Qi 2017↓PointNet: Deep Learning on Point Sets for 3D Classification and SegmentationCharles R. Qi, Hao Su, Kaichun Mo et al., CVPR 2017. PointNet++ groups metric-space neighbourhoods and applies PointNet within them, building local features at progressively larger scales. It uses sampled centroids, radius-based neighbourhoods and coordinates relative to each centroid; pooling summarizes these local features rather than preserving all geometry. Small neighbourhoods can contain too few samples, so its density-adaptive variants combine information across scales Qi 2017↓PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric SpaceCharles R. Qi, Li Yi, Hao Su et al., NeurIPS 2017.

Detection: from pixels to candidate objects

Classical detection returns boxes over a class list fixed at training time, which is a poor fit for a robot that will be asked about objects nobody enumerated. Open-vocabulary detection removes the fixed list. Grounding DINO extends the DINO detector with language and vision fusion in the feature enhancer, language-guided query selection and cross-modality decoder. It accepts category names or referring expressions, but the paper reports weak referring-expression performance without REC training data Liu 2024↓Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionShilong Liu, Zhaoyang Zeng, Tianhe Ren et al., arXiv 2024. Table 2 reports 52.5 AP on COCO 2017 validation for Grounding DINO L with a Swin-L backbone, pretrained on O365, OpenImage and GoldG without COCO training images. Here, zero-shot means transfer without using COCO’s training split, not that its object categories were absent from pretraining Liu 2024↓Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionShilong Liu, Zhaoyang Zeng, Tianhe Ren et al., arXiv 2024.

DINOv2 learns image and patch features through self-supervised pretraining on the curated LVD-142M image dataset. Its transfer evaluations keep the image encoder frozen while training task-specific predictors: linear classifiers for image classification and semantic segmentation, and linear or DPT depth heads. “Without fine-tuning” the encoder does not mean that no downstream predictor is trained Oquab 2023↓DINOv2: Learning Robust Visual Features without SupervisionMaxime Oquab, Timothée Darcet, Théo Moutakanni et al., arXiv 2023.

Segmentation: the grounding layer

A box is the wrong output for manipulation. A box around a mug also contains the table behind it, and a grasp is planned on a surface region, not on a rectangle, so per-pixel labellingsemantic segmentationLabelling every pixel of an image with a class, as opposed to drawing a box around each object. For manipulation the per-pixel form is what matters, because a grasp is planned on a region of a surface rather than on a rectangle: a box around a mug also contains the table behind it, and a mask does not. is what a downstream planner can actually use.

Segment Anything (SAM, 2023) makes image segmentation promptable. Its default ViT-H model was trained on SA-1B, which the paper reports as 1.1 billion automatically generated masks from 11 million images. Those are dataset counts: the training recipe samples and filters masks rather than establishing that every mask is used. Points, boxes and mask prompts condition the decoder; an ambiguous point can refer to a part or a whole object, so SAM can return multiple candidate masks Kirillov 2023↓Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi et al., arXiv 2023.

The paper evaluates zero-shot transfer on new datasets and tasks. In its single-point test across 23 datasets, the default prompt is the ground-truth mask's center and the default output is SAM's most confident mask. SAM has higher mIoU than RITM on 16 of those 23 datasets, not all of them; the separate oracle result selects the best mask using ground truth. Free-form text is a separately trained, CLIP-conditioned proof of concept, not evidence that the ordinary geometric-prompt model has the same text capability. Fine structures and boundaries remain failure cases. The reported roughly 50 ms prompt processing on a browser CPU assumes a precomputed image embedding; the paper explicitly says a heavy image encoder prevents overall real-time performance Kirillov 2023↓Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi et al., arXiv 2023.

SAM 2 extends point, box and mask prompting across video with a streaming memory of earlier predictions and prompted frames. The paper's v2, dated 28 October 2024, states that its reported results use the improved model called SAM 2.1. In the interactive video experiment, the authors report better average segmentation accuracy with more than three times fewer interactions than their SAM+XMem++ and SAM+Cutie baselines. This is a simulated comparison on nine densely annotated video datasets, using Hiera-B+ at resolution 1024, three clicks per interacted frame and one to eight interacted frames, not a measured reduction in every user's work. Offline evaluation repeatedly prompts the worst-IoU frame; online evaluation adds corrections when IoU falls below 0.75 and changes only subsequent frames. Both use ground truth to select corrections. Section 6.1 describes feeding tracker masks to SAM as mask prompts; Appendix F.1.4 instead reconstructs masks with clicks until IoU exceeds 0.8, then adds correction clicks. Those baseline descriptions do not fully agree Ravi 2024↓SAM 2: Segment Anything in Images and VideosNikhila Ravi, Valentin Gabeur, Yuan-Ting Hu et al., arXiv 2024.

The image-speed comparison is different. Tables 5 and 15 compare SA-1B-only SAM 2 with a Hiera-B+ encoder against SAM with ViT-H: 58.9 versus 58.1 one-click mIoU on the 23-dataset SA-23 aggregate, and 130.1 versus 21.7 images per second. That is the paper's roughly sixfold throughput result on one A100, with image batches of 10, PyTorch 2.3.1, CUDA 12.1, bfloat16 mixed precision and compiled image encoders for both models. It is not single-image latency, video-tracking speed or robot control frequency. The image evaluation spans 37 datasets, including video frames treated as individual images; the separate first-frame-prompt video-object-segmentation comparison spans 17 datasets Ravi 2024↓SAM 2: Segment Anything in Images and VideosNikhila Ravi, Valentin Gabeur, Yuan-Ting Hu et al., arXiv 2024.

These are model- and protocol-specific results. Objects are inferred independently, which limits inter-object modeling. The paper still reports failures after shot changes or long occlusions, in crowded scenes and on thin, fast-moving details. Its mixed-data SA-23 prose says 61.4 while Tables 5 and 15 say 61.9; that unresolved source inconsistency is not part of the SA-1B-only comparison above. Appendix F.3.2 also warns that OVIS is not strictly zero-shot for the mixed-data model because its videos occur in MOSE training data Ravi 2024↓SAM 2: Segment Anything in Images and VideosNikhila Ravi, Valentin Gabeur, Yuan-Ting Hu et al., arXiv 2024.

Promptable segmentationpromptable segmentationSegmentation that predicts a mask from an image and prompts such as foreground/background points, a box or a mask. SAM was trained on SA-1B, reported as 1.1 billion automatically generated masks from 11 million images. The paper evaluates zero-shot transfer to new datasets, but an ambiguous prompt can produce multiple candidate masks and performance is not guaranteed on every image distribution. SAM 2 extends geometric prompting across video frames with streaming memory of predictions and prompted frames; masks can be refined with further prompts, but tracking can fail after shot changes, occlusion or confusion between similar objects. In the SAM instance-segmentation experiment, the box prompt comes from a separate ViTDet detector. supplies object masks in MOKA and ReKep, but the keypoint interfaces do not all use the same perception stack.

MOKA uses GroundedSAM masks to propose marked candidate points, then prompts a pretrained VLM to select grasp, function and target keypoints where applicable, along with waypoint regions and motion attributes. Calibrated depth lifts the surface points into the robot frame; a separate grasp sampler and motion pipeline turn the selections into actions. The prompted method does not train a robot policy, although the paper separately studies policy distillation from successful trajectories Liu 2024↓MOKA: Open-World Robotic Manipulation through Mark-Based Visual PromptingFangchen Liu, Kuan Fang, Pieter Abbeel et al., 2024.

ReKep proposes semantic keypoints from DINOv2 features within SAM masks and calibrated RGB-D geometry. GPT-4o writes Python sub-goal and path constraints from a marked image and a language instruction; a separate solver repeatedly plans end-effector motion from tracked keypoints. Its implementation penalizes constraint violations rather than guaranteeing physical feasibility. Rigidity assumptions and tracking failures remain limitations, and some evaluations use human annotations or disable tracking Huang 2024↓ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic ManipulationWenlong Huang, Chen Wang, Yunzhu Li et al., 2024.

RoboPoint instead instruction-tunes a VLM to predict image-space affordance points, mixing synthetic object and free-space reference data with real-image VQA and LVIS detection data. Its point predictor does not require an external detector at test time, but the reported robot pipeline still uses depth, an end-effector offset and a motion planner. Predicting a point is not by itself a grasp pose or a complete control policy Yuan 2024↓RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for RoboticsWentao Yuan, Jiafei Duan, Valts Blukis et al., 2024.

Dense Object Nets learns a descriptor vector for each RGB pixel, using matches from static RGB-D reconstructions as self-supervision. With object-centric training techniques, the authors demonstrate correspondence across viewpoints and moderately deformable object configurations; they do not guarantee that every physical point has an identical descriptor. Their class-consistent training mode generalizes across sufficiently similar hats, shoes and mugs, while instance-specific training distinguishes objects Florence 2018↓Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic ManipulationPeter R. Florence, Lucas Manuelli, Russ Tedrake, CoRL 2018.

The paper's 20-minute estimate is for learning a new object, including collection of a handful of scenes. Appendix D reports 3500 optimization steps, about 13 minutes, on a single Nvidia 1080 Ti or Titan Xp; Appendix B reports about 70 seconds for one static-scene scan. The demonstrated setup uses a calibrated Kuka IIWA LBR arm, a wrist-mounted Primesense Carmine 1.09 RGB-D sensor and a Schunk two-finger gripper. Humans often rearranged objects and still switched and identified different objects during multi-object collection. A matched reference pixel is mapped into depth geometry for grasp planning; the descriptor does not itself specify the gripper's 6-DoF orientation Florence 2018↓Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic ManipulationPeter R. Florence, Lucas Manuelli, Russ Tedrake, CoRL 2018.

Dex-Net 2.0 instead learns to score candidate parallel-jaw grasps. The paper reports 6.7 million synthetic training datapoints pairing grasps with depth observations and analytic grasp-quality labels, generated from 1500 object models. Its training images are grasp-aligned crops, not 6.7 million measured robot successes or necessarily that many distinct scene views; Figure 3 describes the population as “over 6.7 million grasp images.” GQ-CNN takes an aligned depth crop and the gripper's depth relative to the camera. Its labels use thresholded robust epsilon quality and collision constraints; physical grasp success is evaluated separately Mahler 2017↓Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp MetricsJeffrey Mahler, Jacky Liang, Sherdil Niyaz et al., RSS 2017.

The Dex-Net planner samples and ranks antipodal candidates, checks reachability and table collision, and operates on single-view depth for an isolated object on a planar worksurface. Training assumes rigid objects and known camera intrinsics and gripper geometry. It avoids registering a target-specific object model at execution, not geometric assumptions or synthetic object models during training. Missing depth on thin parts and collisions remain failure modes; its pile-handling demonstration separates objects before grasping Mahler 2017↓Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp MetricsJeffrey Mahler, Jacky Liang, Sherdil Niyaz et al., RSS 2017.

6-DoF pose estimation, and how to read a claim about it

Pose estimationpose estimationRecovering an object's full rigid position and orientation, six degrees of freedom, from sensor data, rather than only where it is in the image. Methods divide by what they assume: instance-level ones such as PoseCNN are trained per object and need that exact object's model, while model-free methods such as FoundationPose take a CAD model or a few reference images at test time and handle objects they never saw in training. It is the step that turns a detection into something a grasp planner can use, because force closure is computed against a known object pose. is the stage that turns a mask into a rigid body with an orientation. The methods divide by what they assume they already know about the object, which is also the division that decides whether you can use one.

Instance-level methods are trained for specific objects. PoseCNN localises each object's centre, predicts its distance, and regresses a quaternion for rotation, with a loss designed to handle symmetric objects; it shipped with the YCB-Video dataset, 21 objects across 92 videos and 133,827 frames Xiang 2018↓PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered ScenesYu Xiang, Tanner Schmidt, Venkatraman Narayanan et al., RSS 2018. Accurate on the objects it knows, useless on the object that arrived this morning.

Model-free and novel-object methods removed that constraint. MegaPose estimates the pose of novel objects by render-and-compare against a CAD model supplied at test time Labbé 2022↓MegaPose: 6D Pose Estimation of Novel Objects via Render and CompareYann Labbé, Lucas Manuelli, Arsalan Mousavian et al., CoRL 2022, and FoundationPose unifies model-based and model-free operation in one network: give it a CAD model, or give it a handful of reference images, and it runs on an object it never trained on, reaching results comparable to instance-level methods despite assuming far less Wen 2024↓FoundationPose: Unified 6D Pose Estimation and Tracking of Novel ObjectsBowen Wen, Wei Yang, Jan Kautz et al., CVPR 2024.

To read a pose-estimation result, check its evaluation protocol. Hinterstoisser and colleagues' LINEMOD-based paper evaluates a model $\mathcal M$ using its ground-truth rotation and translation $(R,T)$ and its estimated pose $(\widetilde R,\widetilde T)$. Equation (1) averages the distances between corresponding model points after the two transforms
Hinterstoisser 2013↓Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered ScenesStefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013):

$$m = \operatorname{avg}_{x \in \mathcal M} \left\lVert (Rx+T)-(\widetilde R x+\widetilde T) \right\rVert$$

The paper counts the detection and pose estimate as correct when $m \leq k_m d$, where $d$ is the model's diameter and $k_m$ is a chosen coefficient. Equality counts as correct. Table 1 uses $k_m=0.1$; that is this evaluation's setting, not a universal pose-accuracy threshold
Hinterstoisser 2013↓Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered ScenesStefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013).

For the symmetry-aware pose errorsymmetry-aware pose error (ADD-S)A model-point pose-error rule that replaces fixed point correspondences with nearest-neighbour matches. In Hinterstoisser and colleagues' LINEMOD-based paper, Equation (1) averages distances between corresponding model points transformed by the ground-truth and estimated poses. Equation (2) instead matches each ground-truth-transformed point to the nearest estimated-pose-transformed model point before averaging. The paper names cup, bowl, box and glue as ambiguous objects, including ambiguity from a subset of views. Its correctness rule is inclusive: the average distance must be at most a chosen coefficient times the model diameter; Table 1 uses 0.1. This is a point-set score and a chosen evaluation threshold, not a guarantee that every visually indistinguishable rotation has zero error., Equation (2) changes the correspondence rule: each ground-truth-transformed model point is matched to its nearest estimated-pose-transformed model point, and those distances are averaged
Hinterstoisser 2013↓Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered ScenesStefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013):

$$m = \operatorname{avg}_{x_1 \in \mathcal M} \min_{x_2 \in \mathcal M} \left\lVert (Rx_1+T)-(\widetilde R x_2+\widetilde T) \right\rVert$$

The paper uses this rule for objects that are ambiguous, or appear ambiguous from a subset of views, naming “cup”, “bowl”, “box” and “glue”
Hinterstoisser 2013↓Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered ScenesStefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013). Read that as a rule over the supplied model points, not a guarantee that every indistinguishable rotation of any bowl has zero error or passes the chosen threshold.

The BOP Challenge 2023 report uses a different aggregate from ADD-S. For 6-DoF localisation, each dataset score first averages three metric recalls: VSD, MSSD and MSPD. Each recall is averaged over multiple correctness thresholds, with a strict correctness test $e < \theta_e$; VSD also varies its misalignment tolerance. $AR_C$ then averages the dataset scores equally across seven core datasets: LM-O, T-LESS, ITODD, HB, YCB-V, TUD-L and IC-BIN. The report displays this dimensionless recall score on a 0 to 100 scale, not ADD-S distance, millimetres or robot success. It does not use the object-first averaging of the report's 2D-detection $AP_C$. VSD considers the visible object surface; MSSD and MSPD use pre-identified object symmetries. The report delegates numerical threshold grids and further symmetry details to its cited 2020 report. Seen and unseen localisation use the same evaluation methodology, with object identities and instance counts supplied at test time
Hodan 2024↓BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid ObjectsTomas Hodan, Martin Sundermeyer, Yann Labbé et al., arXiv preprint, 2024.

In the report’s retrospective 2017 to 2023 comparison, seen-object accuracy rises from 56.9 to 85.6 $AR_C$, a relative improvement of more than 50 percent. The endpoints are Vidal’s point-pair-feature method and GPose2023, not a universal gain in robot performance. The dates and channels need care: Figure 1 labels the selected methods RGB-D and associates Vidal with 2017, while Table 2 labels the Vidal-Sensors18 entry’s submission year as 2019 and its test input as depth. That entry uses PPF and ICP without training images; GPose2023 uses object-specific networks, custom detections, coordinate-guided refinement and RGB-D training/testing with PBR plus real training images
Hodan 2024↓BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid ObjectsTomas Hodan, Martin Sundermeyer, Yann Labbé et al., arXiv preprint, 2024.

For unseen objects, GenFlow-MultiHypo16 leads Task 4 with 67.4 $AR_C$. The best 2020 seen-object entry, CosyPose-ECCV20-SYNT+REAL-ICP, scores 69.8 in Task 1. Section 4.4 calls these results comparable, not identical. Their reported average image-processing times are 34.58 and 13.74 seconds, respectively: reported timings, not a matched-hardware speed ratio or a robot control frequency. Both entries test on RGB-D images. GenFlow uses one network across datasets, CNOS-fastSAM, recurrent-flow refinement and PBR training; CosyPose uses dataset-specific networks, default synthetic-plus-real detections, DeepIM plus ICP refinement and RGB training with PBR plus real images. Task 4 still supplies 3D mesh models of the test objects for onboarding, capped at 5 minutes per object on one GPU; no real images of those objects may be used for onboarding. This is a comparison between two benchmark entries with different training and onboarding conditions, not evidence that the seen-versus-unseen gap disappeared. Figure 4 still shows GenFlow failures under heavy object occlusion
Hodan 2024↓BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid ObjectsTomas Hodan, Martin Sundermeyer, Yann Labbé et al., arXiv preprint, 2024.

Visual servoing: image-based and position-based control

Image-based visual control defines a desired configuration of image features and uses their measured error to guide motion. Espiau, Chaumette and Rives’s 1992 paper applies a task-function framework to this problem. The task function is designed for regulation and need not be the raw feature difference. Their interaction matrix relates feature rates to the camera’s translational and rotational velocity relative to the scene, expressed in the camera frame. Convergence depends on the task and interaction-model conditions; neglecting target motion can leave a tracking error Espiau 1992↓A new approach to visual servoing in roboticsB. Espiau, F. Chaumette, P. Rives, IEEE Trans. Robotics and Automation, 1992.

For the fixed goal and motionless target considered in Chaumette and Hutchinson’s Part I, the relation is $\dot{s} = L_s v_c$: $v_c$ is the camera’s six-component spatial velocity, not a vector of joint rates Chaumette 2006↓Visual Servo Control, Part I: Basic ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006.

Visual servoingvisual servoingUsing computer vision data to control a robot's motion. Image-based visual servoing (IBVS) uses image features in its error; position-based visual servoing (PBVS) uses estimated pose parameters. Espiau, Chaumette and Rives applied a task-function framework to visual control in their 1992 paper. Its camera-motion model uses translational and rotational velocity relative to the scene, expressed in the camera frame. IBVS is not universally pose- or calibration-free: its point-feature interaction matrix uses depth and camera intrinsics. Under the tutorial's local stability conditions, coarse estimates can perturb a convergent camera path without changing the final pose reached, but poor estimates can cause instability and large displacements can encounter singularities or local minima. PBVS pose errors can also affect final accuracy. uses vision in the control loop; image-based visual servoing (IBVS) and position-based visual servoing (PBVS) define different errors. IBVS uses image features, while PBVS uses estimated pose parameters. This does not make IBVS universally pose- or calibration-free: the point-feature interaction matrix uses depth and camera intrinsics, and its estimate may use current depths or depths at the desired pose
Chaumette 2006↓Visual Servo Control, Part I: Basic ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006.

For a fixed goal and motionless target, the tutorial gives local asymptotic stability conditions for redundant-feature IBVS: the interaction matrices must have full rank, and the estimated pseudoinverse multiplied by the true matrix must satisfy a positivity condition. The approximations must not be too coarse. Within a convergent IBVS motion, coarse estimates can change the camera trajectory without changing the final pose reached; poor estimates can instead make the system unstable. Large displacements can encounter local minima or singularities, and the camera path can be unpredictable and suboptimal. This is not a guarantee under arbitrary calibration error or image noise. In PBVS, errors in the estimated pose can also affect final accuracy
Chaumette 2006↓Visual Servo Control, Part I: Basic ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006.

Chaumette and Hutchinson’s Part I, Basic Approaches (2006), presents IBVS and PBVS and analyzes their performance and stability
Chaumette 2006↓Visual Servo Control, Part I: Basic ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006. Part II, Advanced Approaches (2007), extends the treatment to interaction-matrix estimation, hybrid and partitioned control, optimization, switching, trajectory planning, target tracking, and joint-space formulations for eye-in-hand and eye-to-hand systems
Chaumette 2007↓Visual Servo Control, Part II: Advanced ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2007.

Visibility requirements depend on the chosen measurements. The tutorial’s stereo point feature is observed in both images
Chaumette 2006↓Visual Servo Control, Part I: Basic ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006, while its epipolar estimate uses matches between the current and desired images. Part II describes switching and trajectory-planning methods that address target visibility, occlusions and joint limits
Chaumette 2007↓Visual Servo Control, Part II: Advanced ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2007. Visibility is not automatic: Part I gives PBVS configurations in which points leave the camera’s field of view
Chaumette 2006↓Visual Servo Control, Part I: Basic ApproachesFrançois Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006.

Separate model error from geometric error

Evaluate a perception model in its native output first, then evaluate the geometry that consumes it. A detector can have stable image coordinates while the robot misses because the camera transform drifted. A depth model can improve pixel metrics while a grasp degrades because invalid values cluster on the target material. Mixing the stages into one task-success number makes both failures harder to localize.

Maintain a small calibration target set and a separate task object set. Track reprojection, depth residual, pose residual, age and task success as a chain, preserving sample IDs between stages. Re-run the geometric checks after camera remounts, lens changes, time-source changes and software updates that alter resizing or intrinsics. The dedicated calibration guide defines those checks; the ROS 2 guide shows where images, transforms and timestamps meet at runtime.

What the end-to-end models replaced

This is the argument the module exists to make. Every stage above is a place where a real system carries an interface, a calibration file, a failure mode and an engineer who owns it. Calibration residuals, depth invalidation on the wrong surface, a detector's class list, a pose estimate's symmetry handling: six subsystems you can debug one at a time, composing into one error budget you can write down.

The explicit pipeline is exactly what end-to-end vision-language-action policies replaced. A policy trained on images and actions does not compute a pose, because nothing in its objective asks it to, and the geometric reasoning that calibration and pose estimation made explicit gets absorbed into weights where it is no longer a stage you can inspect or calibrate. That trade runs in both directions. What you buy is a stack that no longer breaks when an object arrives that no detector was trained on and no CAD model exists for. What you sell is the error budget: when an end-to-end policy misses a grasp by 8 mm, there is no hand-eye residual to re-solve, because there is no hand-eye transform.

The keypoint interfaces sat in the honest middle of that trade, and what happened to them is worth noting. They kept geometry as an explicit intermediate representation while letting a learned model produce it, and by the time frontier systems arrived the geometry had moved out of the runtime pipeline and into the training mixture as an auxiliary objective. Naming this pipeline is what makes that history legible instead of mystical. "End to end" is a claim about which of these six stages stopped being a stage, and you cannot evaluate the claim without knowing what the six were.