EGOWILD2DEX: Learning Dexterous Robotic
Manipulation from In-the-Wild Human Experience

Learning from in-the-wild ego-human data

Large-scale ego-human demonstrations offer a scalable alternative to costly robot teleoperation. Beyond scale, they capture environmental dynamics, contact-induced object affordances, and dexterous multi-finger coordination.

Plenty more work has shown that ego-human data can provide useful priors for robot manipulation, but often relies on massive non-public datasets, costly paired human–robot collections, or specially constructed scenes. Learning from uncontrolled in-the-wild egocentric data remains challenging because of cluttered backgrounds and dynamic headset views. Moreover, much of that progress uses gripper action spaces rather than the multi-finger coordination needed for long-horizon dexterous tasks.

EgoWild2Dex studies how robots can learn from naturally occurring in-the-wild egocentric human experience. Two practical challenges follow. First, dynamic head-mounted observations differ from the fixed robot-camera views used at deployment. Second, broad human experience is hard to directly transfer to reliable robot execution.

To address these problems, EgoWild2Dex offers three contributions to the community.

• GeoFormer. A moving head-mounted camera and a fixed robot camera can make the same action look significantly different. We introduce a lightweight geometric network that learns this viewpoint gap directly and warps ego observations toward the robot view, without relying on calibration-heavy 3D reconstruction and inpainting.
• Progressive Human–Robot Training Pipeline. Matching the camera view only solves part of the transfer problem: human motion still has to become reliable robot action. We map ego, glove, and robot demonstrations into one robot-native action space, then train progressively—from broad human experience, to task-specific co-training, to robot execution and recovery.
• EgoWild Dataset. Dexterity in the real world happens in cluttered, changing places. We collect 538.9 hours of unscripted ego-human behavior in homes, factories, pharmacies, parcel stations, and other real sites, preserving the dynamic viewpoints, long-tail object categories, and complete workflows that make everyday manipulation challenging.

GeoFormer

In the collection of in-the-wild egocentric human data, the camera moves as the human head moves. A robot, however, watches the workspace from a mostly fixed viewpoint. As a result, the same manipulation can look very different in the two domains. Feeding both types of frames directly into a single policy requires it to jointly disentangle motion, perspective, and action. Existing methods first project the scene onto the robot camera's viewpoint and then inpaint missing regions, but they depend on precise calibration and incur substantial computational overhead.

We built GeoFormer to bridge this gap. It learns from unpaired ego and robot images, progressively refining a lightweight projective warp that captures their viewpoint difference. During policy training, we freeze GeoFormer and use the inverse warp to transform each ego frame to match the robot view. This simple but effective alignment runs 21.9× faster than the Project+Inpaint baseline while improving similarity to the robot-view image.

Progressive Human–Robot Training

Ego

Glove

Robot

Aligning the viewpoints still does not tell a robot how to reproduce human motion with its own arms and hands. We therefore draw on three complementary sources of experience.

Ego data itself plays two distinct roles: open-ended EgoWild demonstrations build a broad prior, while task-specific ego demonstrations ground that prior in the downstream tasks the robot must execute. Glove demonstrations bring the workflow into the robot-camera view while preserving accurate wrist and finger motion. Robot demonstrations supply the physical contact and dynamics, but they are expensive to collect.

Since no source is sufficient on its own, we train progressively: learn broadly, ground the target task with human data, and refine execution with robot demonstrations and recoveries.

To train this progression as one policy, we align the action space from different sources. We calibrate a shared reference, map wrist motion through arm inverse kinematics, and retarget finger motion into hand-motor commands. This converts ego, glove, and robot demonstrations into the same robot-native action representation, allowing every stage to keep the same architecture and training objective.

EgoWild

In-the-wild data is useful as it contains more variation. As backgrounds, objects, viewpoints, and execution styles change, the model has fewer opportunities to rely on shortcuts tied to a single staged setup. Instead, it can learn manipulation patterns that persist across environments—the kind of experience we expect to support generalization to unfamiliar scenes and objects. To capture this variation, we created EgoWild in homes, supermarkets, parcel stations, factories, pharmacies, and offices. We leave each site untouched and let people complete real tasks without scripted action sequences, preserving the decisions, mistakes, and recoveries.

The resulting 538.9-hour dataset contains 179,049 episodes, 125,961 unique task descriptions, and 1,282 object categories. Each sequence combines 2048 × 1536 first-person video, tracked bimanual motion, and timestamped language for atomic actions, segments, and complete skills. Calibrated hand trackers reduce median fingertip error from 4.21 cm to 0.66 cm. These natural settings produce dense, dynamic observations: each frame contains 34.78 foreground instances on average, while cumulative head rotation reaches 15.93° per second.

Results

We evaluate EgoWild2Dex in three long-horizon tasks.

Citation

Citing this research

@article{lin2026egowild2dex,
  author = {Kunyang Lin and Xutao Wen and Jingxi Lin and Lanyong Lin and Jiaming Liu and Tianshuo Yang and Xianchi Chen and Yue Han and Yiduo Li and Zhanpeng Zhang and Ping Luo},
  title  = {EGOWILD2DEX: Learning Dexterous Robotic Manipulation from In-the-Wild Human Experience},
  year   = {2026},
  month  = {September},
  url    = {https://mmlab.hk/egowild2dex},
}