RoboColosseum tests robot AI at 124 real-world sites; no fine-tuned model completes a Unitree G1 humanoid task
RoboColosseum is a new benchmark that tests robot AI models on real robots in offices, homes, parks, libraries and factories instead of a single lab. Its founding members are NUS's MAGIC Lab, Hugging Face, the Singapore Embodied AI Centre and BitRobot, according to its arXiv preprint. Unitree said on X it is excited to support the project, which uses G1s as its main humanoid.
The first study, run Sept. 17 to Oct. 28, logged 5,699 blinded head-to-head comparisons at 124 sites, covering five vision-language-action models and three frontier AI agents. The agents rivaled the specialized robot models. After fine-tuning on the same 100 demonstrations per task, NVIDIA's GR00T N1.7 led on SO-101 arms and pi0.5 on Franka and bimanual YAM.
On five G1 tasks, including hanging keys and loading a washing machine, none of the five fine-tuned robot models succeeded once (the AI agents weren't tested on G1); average progress was 13.3% to 23.2%. The paper isn't peer-reviewed yet.
More on Unitree, Nvidia and robot ai.
About the footage: Official RoboColosseum project video: a real Unitree G1 working at a table in a home setting. The clip doesn't say which model, if any, was controlling the robot or whether it was teleoperated.
The source post
Post by Unitree on X.
View the post on X
Sources
- arXiv: RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild
- RoboColosseum project site
- RoboColosseum on X: Introducing RoboColosseum
- Unitree on X
- Hugging Face Papers: RoboColosseum
New to robots? Read our guide: How to tell if a robot video is real: CGI, sped up, or teleoperated?.
Summarized by Servospace from the sources above, drafted with AI help and checked against those sources before publishing. Claims are the companies' or outlets' unless we say otherwise. How we work.