Runway Launches Praxis-1, a New AI Model for Robot Control

Runway, a company primarily known for its generative video models, is making a significant move into the robotics sector. The organization has officially released Praxis-1, a 'World Action Model' that enables robots to perform specific tasks based on environmental perception and video data. The project is fully open-source, with its code available under the Apache 2.0 license.

How Praxis-1 Works

The primary advantage of Praxis-1 is its ability to learn the laws of physics from simple video footage. Runway initially trained the model on general videos to understand object movement and interaction, subsequently refining it with data reflecting human and robot behavior. This approach could reduce the future need for collecting expensive, robot-specific datasets.

The system processes camera imagery, text instructions, and information regarding the robot's current position in real time. By processing this data, the model predicts environmental changes and plans the robot's movements accordingly. According to the company, the model instantly adjusts the control process as new camera frames arrive.

Test Results and Real-World Capabilities

In tests conducted by Runway, Praxis-1 demonstrated impressive results. On the Nvidia RoboLab-120 leaderboard, which evaluates a robot's ability to sort and move objects, Praxis-1 recorded a 46.2% success rate, making it a leader in its category. In real-world conditions, when tested on a Franka robotic arm, the model successfully completed 88 out of 100 trials, including tasks such as closing a drawer or placing a banana on a plate.

The company also tested the model's adaptability to new platforms. On robotic systems previously developed by Google, Praxis-1 achieved an 80% success rate, significantly outperforming the original RT-1 model, which recorded 52.4%. It is worth noting that these tests have so far been conducted only by Runway, and independent verification has not yet taken place.

Technical Requirements and Limitations

Despite its success, Praxis-1 is still a work in progress. Researchers note that the model sometimes struggles to distinguish between visually similar objects and may have difficulty understanding ambiguous instructions. Furthermore, it cannot yet plan complex, long-term actions independently, meaning difficult tasks must be broken down into individual stages.

Technically, the system is resource-intensive. The standard version requires a GPU with 80 GB of memory, though a compressed version is available, which is expected to run on an RTX 5090 with 32 GB of VRAM. Developers can download the model from the Hugging Face platform, and the code is hosted on GitHub.

For Runway, Praxis-1 represents a first step into robotics, bridging video intelligence with the physical world. In the future, the company plans to integrate the model with other general-purpose AI systems, which would make robots more autonomous and versatile.