EgoVerse: An Open-Source Recipe for Human-to-Robot Transfer

Leveraging human demonstration data offers a more scalable path for real-world Physical AI development than robot teleoperation. However, the training and data recipe needed for sustainable robot improvement has been unclear.

Scale AI has partnered with leading academic and industry labs to introduce EgoVerse, a large-scale dataset and training framework for training Physical AI from egocentric data. EgoVerse both releases high-quality training data and open-sources the learning pipeline for sustainable improvement across robots.

The work is a joint effort across leading academic institutions including Georgia Tech, Stanford, ETH Zurich, MIT, UC San Diego, and Meta among others.

An Open-Source Framework for Co-Training Any Robot with Egocentric Data

The EgoVerse study provides one of the first large, cross-lab validations that combining human and robot data improves robot performance. The EgoVerse architecture uses high-fidelity human data to unlock co-training: Human data and robot data are part of a unified action space the model learns from.

Key findings include:

• Egocentric data can use a single recipe to adapt to multiple robots and end effectors, including humanoid, 6-DOF and 7-DOF arms
• Co-training on human and robot data significantly improves performance, with gains of up to ~30% across tasks and environments
• Diversity is critical to generalization: In particular, real-world scene diversity drives generalization to new environments

These results were replicated across multiple robot platforms and independent labs, underscoring that the findings are not vendor-, environment- or system-specific. This points to a broader shift in how robot learning systems should be built: scale alone is insufficient without the right abstractions and infrastructure to make that data usable.

Scale’s Approach: High-Quality, Diverse Data for Policy Co-Training

“Scale’s diverse egocentric data offers an ideal dataset to train Physical AI models. The diversity, dense language annotations, and accurate 3D hand tracking have enabled strong learning signals and consistent improvements across a wide variety of challenging tasks in our evaluations.” – Danfei Xu, Professor at Georgia Tech

Scale’s Egocentric Dexterity data captures high-fidelity data using stereo cameras and IMUs which can be used in co-training together with robot data. Through a proprietary machine learning pipeline, we deliver:

• Egocentric RGB video from head-mounted devices
• Accurate 3D hand pose tracking including full 21-keypoint hand positions, not just wrist position
• Camera pose estimates accurate 6-DoF camera trajectories to reconstruct the spatial context of each demonstration
• Dense language annotations with natural-language descriptions of what's happening at both the demonstration and subgoal (1-2s) level

Scale contributed high-volume, diverse hours of human demonstration data, along with infrastructure and engineering support to enable large-scale data aggregation and usability across the consortium.

The EgoVerse dataset, framework, and research findings are available now at https://egoverse.ai/. Learn more about Scale AI is developing the next generation of robots here: https://scale.com/physical-ai