Perception for Manipulation

Perception for manipulation turns camera and depth data into object poses in the robot's own frame, through calibration, depth sensing, detection, segmentation and pose estimation, and grasp planning computes grasps against those poses. DOPE's pose network was accurate enough for a real robot to grasp known household objects in clutter^{Source 28}28 Tremblay et al. 2018, Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. Jonathan Tremblay, Thang To, Balakumar Sundaralingam et al., Conference on Robot Learning (CoRL) 2018.

Photo: Marc Auledas / Wikimedia Commons. Licence: CC BY-SA 4.0.

An illustrative positioning budget

Try the transparent bottle under "Adjust more": the depth term then dominates the budget.

The instrument combines authored hand-eye, depth and pose magnitudes by root-sum-of-squares, with depth fixed at a 0.5 m standoff and the hand-eye term growing with working distance up to 1.5 m. Its 15 mm and 30 mm bands are teaching thresholds: a 90 mm opening around a 60 mm part leaves $(90-60)/2=15$ mm per side, and one reported assembly task had only 0.5 mm contact clearance^{Source 27}27 Lum et al. 2026, Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?. Tyler Ga Wei Lum, Kushal Kedia, C. Karen Liu et al., CoRL 2026.

Calibration: what a pixel means

A camera maps a 3D point onto a 2D sensor, and its intrinsicscamera intrinsicsCamera intrinsics are the camera's own parameters, independent of where it is mounted: focal length in pixels along each axis, the principal point where the optical axis meets the sensor, and lens distortion. Zhang's method recovers them from a few views of a planar target at unknown orientations, so robots calibrate with a printed checkerboard instead of a metrology rig. Without them a pixel is only a direction in an unknown parameterisation, and no image measurement converts into a metric ray. describe that mapping: focal length in pixels on each axis, the principal point and lens distortion. Zhang's 2000 method shows the camera a planar target at several unmeasured orientations. It models radial distortion and gives a closed-form solution followed by maximum-likelihood nonlinear refinement. Zhang contrasts it with classical techniques that need expensive equipment such as two or three orthogonal planes^{Source 1}1 Zhang 2000, A flexible new technique for camera calibration. Z. Zhang, IEEE Trans. Pattern Analysis and Machine Intelligence, 2000.

Hand-eye calibration: what a pixel means to the robot

Extrinsics place the camera relative to something else. Hand-eye calibrationhand-eye calibrationHand-eye calibration is estimating the rigid transform between a camera and the robot frame it reports into. That is gripper to camera for a wrist-mounted camera, and the robot base to the camera for a fixed one. Tsai and Lenz posed it as solving AX = XB from robot motions and the camera's observed motion of a static target, the formulation most implementations still use. Translation error is a constant offset; rotation error is an angle, so its cost in millimetres grows with target distance. finds the flange-to-camera transform for a wrist camera, or base-to-camera for a fixed one. Shiu and Ahmad cast it as $AX = XB$, where $A$ is a robot motion, $B$ the camera's observed motion of a static target and $X$ the unknown transform^{Source 29}29 Shiu et al. 1989, Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB. Y. C. Shiu, S. Ahmad, IEEE Trans. Robotics and Automation, 1989. Tsai and Lenz gave an efficient closed-form technique that solves rotation first, then translation^{Source 2}2 Tsai et al. 1989, A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. R. Y. Tsai, R. K. Lenz, IEEE Trans. Robotics and Automation, 1989.

Park and Martin derived closed-form exact and least-squares solutions with Lie theory^{Source 30}30 Park et al. 1994, Robot sensor calibration: solving AX=XB on the Euclidean group. F. C. Park, B. J. Martin, IEEE Trans. Robotics and Automation, 1994. OpenCV's hand-eye calibration defaults to the Tsai and Lenz method, needs at least two motions with non-parallel rotation axes, so three poses, and strongly recommends many more^{Source 31}31 OpenCV 2026, Camera Calibration and 3D Reconstruction: calibrateHandEye. OpenCV, OpenCV 4.x Documentation, as of 2026-10-04.

Tsai and Lenz's error analysis shows the lever arm: a 3 mrad rotation error at a 5 in camera-to-target distance gives a 15 mil translation error^{Source 2}2 Tsai et al. 1989, A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. R. Y. Tsai, R. K. Lenz, IEEE Trans. Robotics and Automation, 1989. The instrument's hand-eye term is a ray-to-plane offset at separation $d$ for a ray inclined by $\theta$ to the plane's normal, about 1.7 mm at 10 cm and 17 mm at 1 m for one degree:

Depth sensing: what fails, and on what

Stereo triangulates features matched between two images, so a featureless surface gives it nothing to match. Intel's RGBD depth cameras add an active texture projector to make matching unambiguous^{Source 3}3 Keselman et al. 2017, Intel RealSense Stereoscopic Depth Cameras. Leonid Keselman, John Iselin Woodfill, Anders Grunnet-Jepsen et al., CVPR Workshops 2017.

The March 2026 RealSense D400 datasheet lists ±2% Z-accuracy up to 2 m for D410/D415 and D43x over an 80% region of interest at HD resolution. Accuracy compares valid pixels with ground truth over 80% of the depth field of view, and factory KPIs reflect typical conditions; these models are tested on a texture-less white target at the default 150 mW laser power and auto exposure^{Source 4}4 RealSense 2026, RealSense Product Family D400 Series Datasheet. RealSense, RealSense, as of 2026-08-22.

RealSense's D415/D435 tuning guide says D400 depth error scales with the square of distance. It advises moving closer while staying outside the minimum operating distance, MinZ. It describes resolution and range trade-offs when changing MinZ, and notes that focus and differing left and right views limit very-near matching^{Source 5}5 Grunnet-Jepsen et al. 2026, Tuning depth cameras for best performance. Anders Grunnet-Jepsen, John N. Sweetser, John Woodfill, RealSense documentation, as of 2026-08-22.

Learned models now estimate depth from images, alone or with a sparse sensor: Depth Pro predicts metric monocular depth without camera intrinsics, a 2.25-megapixel map in 0.3 seconds on a standard GPU^{Source 33}33 Bochkovskii et al. 2024, Depth Pro: Sharp Monocular Metric Depth in Less Than a Second. Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain et al., arXiv preprint (ICLR 2025), 2024.

For industrial 3D scanning, the PhoXi 3D Scanner L page lists 0.200 mm calibration accuracy and 0.190 mm temporal noise (both 1 σ), an 870 to 2150 mm scanning range and 250 to 2750 ms scanning time. These are separate specification fields for one model^{Source 7}7 Photoneo 2026, PhoXi 3D Scanner L. Photoneo, Photoneo, as of 2026-08-22.

Time of flight casts its own modulated near-infrared light and measures its round trip per pixel. Multi-path interference lets one pixel integrate light that arrived by several routes. The Azure Kinect DK documentation lists five causes of invalid depth: outside the IR illumination mask, saturated IR signal, low IR signal, filter outlier and multi-path interference. Invalid pixels carry depth zero. Ambiguous depth arises when light reflects from one wall onto another in a corner, and where foreground and background mix at object edges. Fast motion can increase edge invalidation^{Source 6}6 Microsoft 2019, Azure Kinect DK depth camera. Microsoft, Microsoft Learn, 2019; accessed 2026-08-22. Microsoft announced the end of Azure Kinect DK production in August 2023, and support for its Sensor SDK ended in August 2024^{Source 44}44 Microsoft 2024, Microsoft Support for Azure Kinect DK Sensor SDK. Microsoft, Azure Kinect Sensor SDK repository, GitHub, 2024.

Photo: Evan-Amos / Wikimedia Commons. Licence: Public domain.

The failure taxonomy that decides how a cell is built

• ClearGrasp starts from the observation that transparent objects "often appear as noisy or distorted approximations of the surfaces that lie behind them". It infers surface normals, transparent-surface masks and occlusion boundaries, then replaces the returned depth^{Source 8}8 Sajjan et al. 2020, ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation. Shreeyak Sajjan, Matthew Moore, Mike Pan et al., ICRA 2020.
• Specular reflections can saturate D400 images, and the D400f column says "Saturation mitigated"^{Source 4}4 RealSense 2026, RealSense Product Family D400 Series Datasheet. RealSense, RealSense, as of 2026-08-22. Azure Kinect invalidates saturated IR pixels^{Source 6}6 Microsoft 2019, Azure Kinect DK depth camera. Microsoft, Microsoft Learn, 2019; accessed 2026-08-22.
• Dark surfaces and low light weaken the signal. Azure Kinect invalidates a pixel whose IR signal is too weak to generate depth^{Source 6}6 Microsoft 2019, Azure Kinect DK depth camera. Microsoft, Microsoft Learn, 2019; accessed 2026-08-22. RealSense ties D4xx depth quality to the input images and warns of grainy low-light images, underexposure and overexposure. For its projected-pattern example, reducing background light with the projector on can raise the pattern's signal-to-noise ratio^{Source 5}5 Grunnet-Jepsen et al. 2026, Tuning depth cameras for best performance. Anders Grunnet-Jepsen, John N. Sweetser, John Woodfill, RealSense documentation, as of 2026-08-22.
• RealSense names fences and wire grids as scenes where left-right correspondence can become ambiguous. Its D4 processor compares the best and second-best match scores, and raising DSSecondPeakThreshold rejects more matches as zero depth^{Source 5}5 Grunnet-Jepsen et al. 2026, Tuning depth cameras for best performance. Anders Grunnet-Jepsen, John N. Sweetser, John Woodfill, RealSense documentation, as of 2026-08-22.
• Azure Kinect notes that a surface invalidated by multi-path from one camera view can reappear from another^{Source 6}6 Microsoft 2019, Azure Kinect DK depth camera. Microsoft, Microsoft Learn, 2019; accessed 2026-08-22.

The 2% figure is measured on a texture-less white target with the projector at default power^{Source 4}4 RealSense 2026, RealSense Product Family D400 Series Datasheet. RealSense, RealSense, as of 2026-08-22, and the PhoXi page publishes no target condition for its 0.200 mm figure^{Source 7}7 Photoneo 2026, PhoXi 3D Scanner L. Photoneo, Photoneo, as of 2026-08-22. Accuracy for transparent or specular surfaces is not disclosed in either datasheet.

Depth yields a point cloudpoint cloudA point cloud is a set of 3D points, usually with no ordering and no connectivity, as produced by back-projecting depth-camera or lidar measurements through the camera intrinsics. A network reading it must be permutation invariant. PointNet uses a shared per-point encoder and symmetric pooling; PointNet++ recursively applies local PointNets to metric-space neighbourhoods for multi-scale features. Pooling summarizes, so some geometry can be lost; density-adaptive grouping addresses sparse neighbourhoods., an unordered set that a network must process without depending on input order. PointNet uses a shared per-point encoder and a symmetric pooling function^{Source 13}13 Qi et al. 2017, PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Charles R. Qi, Hao Su, Kaichun Mo et al., CVPR 2017. PointNet++ applies PointNet within radius neighbourhoods around sampled centroids, in coordinates relative to each centroid, building local features at growing scales. Density-adaptive variants combine scales where small neighbourhoods hold too few samples^{Source 14}14 Qi et al. 2017, PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Charles R. Qi, Li Yi, Hao Su et al., NeurIPS 2017.

Detection: from pixels to candidate objects

Classical detectors return boxes over a class list fixed at training time. Mask R-CNN extends Faster R-CNN with a branch that predicts an object mask in parallel with box recognition^{Source 34}34 He et al. 2017, Mask R-CNN. Kaiming He, Georgia Gkioxari, Piotr Dollár et al., arXiv preprint (ICCV 2017).

Open-vocabulary detection removes the fixed list. Grounding DINO adds language-vision fusion to the DINO detector's feature enhancer, query selection and decoder and accepts category names or referring expressions. It performs weakly on referring expressions without REC training data. Its Swin-L model reports 52.5 AP on COCO 2017 validation after pretraining on O365, OpenImage and GoldG without COCO training images^{Source 9}9 Liu et al. 2024, Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. Shilong Liu, Zhaoyang Zeng, Tianhe Ren et al., arXiv 2024.

DINOv2 learns image and patch features by self-supervised pretraining on the curated LVD-142M dataset. Its transfer evaluations freeze the encoder and train linear heads for classification and segmentation, and linear or DPT heads for depth^{Source 10}10 Oquab et al. 2023, DINOv2: Learning Robust Visual Features without Supervision. Maxime Oquab, Timothée Darcet, Théo Moutakanni et al., arXiv 2023.

Segmentation: the grounding layer

A box around a mug also contains the table behind it, so a grasp planner needs the surface region that per-pixel labellingsemantic segmentationSemantic segmentation is labelling every pixel of an image with a class, instead of boxing each object. Manipulation needs the per-pixel form because grasps are planned on surface regions. A box around a mug also contains the table behind it; a mask does not. provides. Segment Anything (SAM, 2023) makes segmentation promptable; its default ViT-H model was trained on SA-1B, reported as 1.1 billion automatically generated masks on 11 million images. Point, box and mask prompts condition the decoder, and an ambiguous point can return several candidate masks for a part or a whole object. Fine structures and boundaries remain failure cases, and the heavy image encoder prevents overall real-time performance^{Source 11}11 Kirillov et al. 2023, Segment Anything. Alexander Kirillov, Eric Mintun, Nikhila Ravi et al., arXiv 2023.

SAM 2 extends prompting to video with a streaming memory of earlier predictions and prompted frames. Objects are inferred independently, and failures persist after shot changes or long occlusions, in crowded scenes and on thin, fast-moving details^{Source 12}12 Ravi et al. 2024, SAM 2: Segment Anything in Images and Videos. Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu et al., arXiv 2024.

Grounded SAM feeds Grounding DINO's open-set boxes to SAM^{Source 35}35 Ren et al. 2024, Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. Tianhe Ren, Shilong Liu, Ailing Zeng et al., 2024, and CNOS segments unseen objects by matching DINOv2 descriptors of SAM proposals against references built from CAD models^{Source 36}36 Nguyen et al. 2023, CNOS: A Strong Baseline for CAD-based Novel Object Segmentation. Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin et al., ICCV 2023 R6D Workshop.

Promptable segmentationpromptable segmentationPromptable segmentation is predicting a mask from an image plus prompts such as foreground/background points, a box or a mask. SAM trained on SA-1B, reported as 1.1 billion automatically generated masks from 11 million images. Its paper evaluates zero-shot transfer to new datasets; an ambiguous prompt can produce multiple candidate masks, and performance is not guaranteed on every image distribution. SAM 2 extends geometric prompting across video with streaming memory of predictions and prompted frames; further prompts refine masks. Tracking can fail after shot changes, occlusion or similar-object confusion. In SAM's instance-segmentation experiment, a separate ViTDet detector supplies the box prompt. supplies object masks to MOKA and ReKep, two keypoint interfaces with different perception stacks. MOKA proposes marked candidate points from GroundedSAM masks, then prompts a pretrained VLM to select grasp, function and target keypoints, waypoint regions and motion attributes. Calibrated depth lifts the points into the robot frame, and a separate grasp sampler and motion pipeline produce actions; the prompted method trains no robot policy^{Source 24}24 Liu et al. 2024, MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting. Fangchen Liu, Kuan Fang, Pieter Abbeel et al., 2024.

ReKep proposes semantic keypoints from DINOv2 features within SAM masks and calibrated RGB-D geometry. A large vision-language model writes Python sub-goal and path constraints from a marked image and an instruction, and a solver repeatedly plans end-effector motion from tracked keypoints. The solver penalizes constraint violations without guaranteeing physical feasibility, and rigidity assumptions and tracking failures remain limitations. Some evaluations use human annotations or disable tracking^{Source 25}25 Huang et al. 2024, ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation. Wenlong Huang, Chen Wang, Yunzhu Li et al., 2024.

RoboPoint instruction-tunes a VLM to predict image-space affordance points from synthetic object and free-space data mixed with real-image VQA and LVIS detection data. It needs no external detector at test time, though its robot pipeline adds depth, an end-effector offset and a motion planner^{Source 26}26 Yuan et al. 2024, RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics. Wentao Yuan, Jiafei Duan, Valts Blukis et al., 2024.

Dense Object Nets learns a descriptor for each RGB pixel, self-supervised by matches from static RGB-D reconstructions, and demonstrates correspondence across viewpoints and moderately deformable configurations. Class-consistent training generalizes across sufficiently similar hats, shoes and mugs, while instance-specific training distinguishes objects; for grasping, the gripper's 6-DoF orientation needs further information^{Source 15}15 Florence et al. 2018, Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. Peter R. Florence, Lucas Manuelli, Russ Tedrake, CoRL 2018.

6-DoF pose estimation, and how to read a claim about it

Pose estimationpose estimationPose estimation is recovering an object's full rigid position and orientation, six degrees of freedom, from sensor data. Instance-level methods such as PoseCNN train per object and need that object's exact model. Model-free methods such as FoundationPose take a CAD model or a few reference images at test time and handle unseen objects. Pose estimation turns a detection into grasp-planner input, because force closure is computed against a known pose. turns a mask into a rigid body with a position and orientation, and methods divide by what they assume about the object. Instance-level methods train on specific objects: PoseCNN localizes each object's centre, predicts its distance and regresses a rotation quaternion, with a loss that handles symmetric objects^{Source 16}16 Xiang et al. 2018, PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Yu Xiang, Tanner Schmidt, Venkatraman Narayanan et al., RSS 2018. DOPE, trained only on synthetic domain-randomized and photorealistic data, performs competitively with a network trained on real and synthetic data^{Source 28}28 Tremblay et al. 2018, Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. Jonathan Tremblay, Thang To, Balakumar Sundaralingam et al., Conference on Robot Learning (CoRL) 2018.

Category-level methods such as NOCS estimate the pose and size of unseen instances in known categories with no exact CAD model at training or test time^{Source 37}37 Wang et al. 2019, Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation. He Wang, Srinath Sridhar, Jingwei Huang et al., arXiv preprint (CVPR 2019). Novel-object methods drop per-object training. MegaPose uses render-and-compare against a CAD model supplied at test time^{Source 18}18 Labbé et al. 2022, MegaPose: 6D Pose Estimation of Novel Objects via Render and Compare. Yann Labbé, Lucas Manuelli, Arsalan Mousavian et al., CoRL 2022, and FoundationPose works from a CAD model or a few reference images and reports results comparable to instance-level methods^{Source 17}17 Wen et al. 2024, FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. Bowen Wen, Wei Yang, Jan Kautz et al., CVPR 2024.

To read a pose result, check its evaluation protocol. Hinterstoisser and colleagues evaluate a model $\mathcal M$ with ground-truth pose $(R,T)$ and estimate $(\tilde R,\tilde T)$ by averaging distances between corresponding transformed model points^{Source 20}20 Hinterstoisser et al. 2013, Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes. Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013):

An estimate is correct when $m \leq k_m d$, where $d$ is the model's diameter and $k_m$ a chosen coefficient; their LINEMOD evaluation uses $k_m=0.1$^{Source 20}20 Hinterstoisser et al. 2013, Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes. Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013). For the symmetry-aware pose errorsymmetry-aware pose error (ADD-S)Symmetry-aware pose error (ADD-S) is a pose-error rule using nearest-neighbour matches instead of fixed model-point correspondences. In Hinterstoisser and colleagues' LINEMOD-based paper, the plain error averages distances between corresponding model points under ground-truth and estimated poses; ADD-S instead matches each ground-truth-transformed point to the nearest estimated-pose-transformed model point before averaging. The paper names cup, bowl, box and glue as ambiguous, including from a subset of views. Its correctness rule is inclusive: the average distance must be at most a chosen coefficient times the model diameter, and their LINEMOD evaluation uses 0.1. The score and threshold are evaluation choices, so a visually indistinguishable rotation can still score a nonzero error., each ground-truth-transformed point is matched to its nearest estimated-pose point before averaging^{Source 20}20 Hinterstoisser et al. 2013, Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes. Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013):

They apply this rule to objects that are ambiguous, or look ambiguous from some views, naming "cup", "bowl", "box" and "glue"^{Source 20}20 Hinterstoisser et al. 2013, Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes. Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic et al., Computer Vision – ACCV 2012 (LNCS, published 2013).

The BOP Challenge 2023 report scores 6-DoF localization differently: each dataset score averages the VSD, MSSD and MSPD recalls, each over several thresholds with a strict test $e < \theta_e$, and $AR_C$ averages seven core datasets (LM-O, T-LESS, ITODD, HB, YCB-V, TUD-L, IC-BIN) on a 0 to 100 scale. Seen and unseen tracks share the method, with object identities and instance counts supplied at test time, and the 2D-detection $AP_C$ averages over objects first^{Source 19}19 Hodan et al. 2024, BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects. Tomas Hodan, Martin Sundermeyer, Yann Labbé et al., arXiv preprint, 2024.

From 2017 to 2023, seen-object accuracy rose from 56.9 to 85.6 $AR_C$, a relative gain of over 50 percent. The endpoints are Vidal's point-pair-feature method with ICP and no training images, and GPose2023 with object-specific networks trained on PBR and real images^{Source 19}19 Hodan et al. 2024, BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects. Tomas Hodan, Martin Sundermeyer, Yann Labbé et al., arXiv preprint, 2024.

For unseen objects, GenFlow-MultiHypo16 led 2023's Task 4 with 67.4 $AR_C$, against 69.8 for the best 2020 seen-object entry, CosyPose, and the report calls these comparable across distinct tracks; average image times were 34.58 and 13.74 seconds on unmatched hardware. Task 4 still supplies mesh models for onboarding, capped at 5 minutes per object on one GPU with no real images of the objects, and GenFlow fails under heavy occlusion^{Source 19}19 Hodan et al. 2024, BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects. Tomas Hodan, Martin Sundermeyer, Yann Labbé et al., arXiv preprint, 2024.

In the BOP 2024 report, the best unseen-object method, FreeZeV2.1, is 22% more accurate on BOP-Classic-Core than GenFlow and 4% behind GPose2023, but takes 24.9 against 2.7 seconds per image, and the report names 2D detection as the main bottleneck for unseen objects^{Source 38}38 Nguyen et al. 2025, BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimation. Van Nguyen Nguyen, Stephen Tyree, Andrew Guo et al., 2025.

Visual servoing: image-based and position-based control

Visual servoingvisual servoingVisual servoing is controlling a robot's motion with computer vision data. Image-based visual servoing (IBVS) builds its error from image features; position-based (PBVS) from estimated pose. Espiau, Chaumette and Rives applied a task-function framework to visual control in 1992, modelling translational and rotational camera velocity relative to the scene, expressed in the camera frame. IBVS's point-feature interaction matrix still needs depth and camera intrinsics. Under the tutorial's local stability conditions, coarse estimates can perturb a convergent camera path without changing the final pose; poor estimates can cause instability. Large displacements can meet singularities or local minima. PBVS pose errors can also affect final accuracy. uses vision inside the control loop, and Hutchinson, Hager and Corke's 1996 tutorial divides such systems into position-based and image-based classes^{Source 39}39 Hutchinson et al. 1996, A tutorial on visual servo control. S. Hutchinson, G. D. Hager, P. I. Corke, IEEE Transactions on Robotics and Automation, 1996. Espiau, Chaumette and Rives's 1992 paper applies a task-function framework to image-based control, with a task function designed for regulation that need not be the raw feature difference. Its interaction matrix relates feature rates to the camera's translational and rotational velocity in the camera frame. Convergence depends on task and model conditions, and neglecting target motion can leave a tracking error^{Source 21}21 Espiau et al. 1992, A new approach to visual servoing in robotics. B. Espiau, F. Chaumette, P. Rives, IEEE Trans. Robotics and Automation, 1992.

For a fixed goal and motionless target, Chaumette and Hutchinson's Part I writes $\dot{s} = L_s v_c$, with $v_c$ the camera's six-component spatial velocity. Image-based visual servoing (IBVS) uses image features, and position-based visual servoing (PBVS) uses estimated pose. The IBVS point-feature interaction matrix still uses depth and camera intrinsics, estimated from current depths or depths at the desired pose^{Source 22}22 Chaumette et al. 2006, Visual Servo Control, Part I: Basic Approaches. François Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006.

Redundant-feature IBVS is locally asymptotically stable when the interaction matrices have full rank and the estimated pseudoinverse times the true matrix meets a positivity condition; coarse estimates can change the camera path, while poor ones can make the system unstable. Large displacements can meet local minima or singularities, PBVS points can leave the field of view, and in PBVS pose-estimation errors affect final accuracy^{Source 22}22 Chaumette et al. 2006, Visual Servo Control, Part I: Basic Approaches. François Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2006.

Part II (2007) adds switching and planning methods that address visibility, occlusions and joint limits, for eye-in-hand and eye-to-hand systems^{Source 23}23 Chaumette et al. 2007, Visual Servo Control, Part II: Advanced Approaches. François Chaumette, Seth Hutchinson, IEEE Robotics and Automation Magazine, 2007. Bateux and colleagues fine-tuned a CNN to estimate the relative pose between current and desired images and kept a pose-based control law^{Source 40}40 Bateux et al. 2018, Training Deep Neural Networks for Visual Servoing. Quentin Bateux, Eric Marchand, Jürgen Leitner et al., 2018 IEEE International Conference on Robotics and Automation (ICRA).

Learned servoing can also aim the camera. EyeRobot 2.0, from Ken Goldberg (UC Berkeley) and colleagues (October 2026), swivels the two eye viewpoints of a single stereo camera to fixate a 3D point, with a gaze-servoing policy and a fixation selector both trained with RL on real data. It beat passive stereo by 40% in real trials and, when a grasped object hid the wrist views, more than doubled the success of policies that had them, 48% against 22%^{Source 43}43 Hari et al. 2026, EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras. Kush Hari, Justin Kerr, Nidhya Shivakumar et al., 2026.

Separate model error from geometric error

Hand-eye accuracy depends on noise and motion range, so Enebuse and colleagues advise accounting for both when benchmarking or calibrating^{Source 32}32 Enebuse et al. 2022, Accuracy evaluation of hand-eye calibration techniques for vision-guided robots. Ikenna Enebuse, Babul K. S. M. Kader Ibrahim, Mathias Foo et al., PLOS ONE, 2022. A detector can keep stable image coordinates while the robot misses because the camera transform drifted, so each stage is checked in its own output first. The calibration guide defines the checks to rerun after a camera remount, and the ROS 2 guide shows where images, transforms and timestamps meet at runtime.

What the end-to-end models replaced

End-to-end vision-language-action (VLA) policies absorb the explicit pipeline into weights. Gemini Robotics is a VLA model that controls robots directly, built on an embodied-reasoning model that can detect objects, point, and predict trajectories, grasps and 3D bounding boxes^{Source 41}41 Gemini Robotics Team et al. 2025, Gemini Robotics: Bringing AI into the Physical World. Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie et al., 2025. Such a policy needs no detector class or CAD model for a new object, and it has no hand-eye residual to re-solve when it misses a grasp by 8 mm.

Some frontier systems keep geometric intermediates explicit, as the keypoint interfaces did while a learned model produced the geometry: MolmoAct encodes observations into depth-aware perception tokens and generates editable trajectory traces before predicting low-level actions^{Source 42}42 Lee et al. 2025, MolmoAct: Action Reasoning Models that can Reason in Space. Jason Lee, Jiafei Duan, Haoquan Fang et al., 2025.