EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras

Fine bimanual manipulation usually needs a camera close to the hand to see contact detail — but the moment a grasped object blocks that lens, the camera is useless. EyeRobot 2.0 looks to human eyes instead: rather than adding another camera, it uses a pair of steering eyes that keep the most sensitive fixation point on whatever needs scrutiny most.

Background

Fine bimanual manipulation usually leans on wrist cameras: a close-up view of the hand that keeps contact detail visible under occlusion. But wrist cameras add cost, cabling and a fragile point, and they fail outright the moment a grasped object blocks the lens. Human eyes do not solve this by adding another camera; they solve it with active gaze — swiveling the eyeball so the fovea, the most sensitive region, lands on whatever point needs the most scrutiny.

Core Innovation

EyeRobot 2.0 gives the robot a pair of steering eyes. It physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze, and then processes the resulting images foveally, allocating more visual tokens to the image centers so computation concentrates on task-relevant features. The hard part is coordinating this Active Visual Fixation (AVF) during execution. The authors solve it hierarchically: first a low-level gaze servoing policy conditioned on a goal object, then a target selector that emits fixation goals based on task progress. Both are trained with reinforcement learning on real-world data — the servo with a dense geometric reward, the selector co-trained with the BC gripper policy so it discovers fixation sequences that resemble a human’s. The system also canonicalizes gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution.

Results

• 7 real-world and 6 simulated tasks; over 1000 physical and 1800 simulated trials.
• Remove the wrist camera and standard policies suffer: with only passive stereo, real-world success drops from 52% to 27%.
• EyeRobot 2.0 closes that gap using stereo alone, outperforming passive stereo by 40% in real and 20% in sim.
• It matches ego plus wrist policies when their wrist views are clear (69% vs 64%).
• It more than doubles their success when grasped objects occlude the wrist cameras (48% vs 22%).

Limitations

Active gaze requires a steerable two-viewpoint mechanism, and those extra mechanical degrees of freedom are themselves a new hardware and calibration burden; the abstract gives no cost, power or long-term reliability figures. Both the gaze servo and the target selector are trained with RL on real-world data, which is not cheap to collect or train. Evaluation centers on seven real tasks, so cross-task and cross-scene generalization is not quantified. The foveal token-allocation scheme’s dependence on model size and deployment compute is also left open.

Industry Implications

The trade is stark: a pair of moving eyes in place of cameras permanently mounted on the hands. What a vendor removes is a camera, its wiring and a fragile point on every arm; what it adds is a precision gaze mechanism that must be calibrated, and a real mitigation of the classic failure where the lens is blocked once the hand grabs something. The most commercially meaningful line in the results is that success more than doubles under wrist occlusion — precisely where wrist cameras are weakest, active gaze’s relative advantage is largest. Any team struggling with hand occlusion in multi-finger or large-object grasping should study this route, while folding the added gaze freedom into its cost and reliability accounting.