OpenAI has disbanded its stand-alone robotics team, moving its researchers onto other teams.
OpenAI had a robotics team in its early days but wound it down in 2020, with cofounder Wojciech Zaremba later citing limited robot data and a more efficient path toward artificial general intelligence (AGI)—the human-like AI that has been OpenAI’s goal since its founding—through other methods. In particular, OpenAI became increasingly devoted to training ultra-large language models on vast datasets of human-written text. This is the approach that eventually yielded the company’s breakthrough product ChatGPT in late 2022.
This line of thinking—that general AI models can be applied directly to controlling robots without the need to create specialized models with “spatial intelligence”—would seem to challenge the assumptions underpinning many of today’s robotics startups, which have raised billions on the premise that robots need brains of their own.
But Russ Tedrake, an MIT professor and cofounder and CEO of Walden Robotics, said robotics companies still have an advantage the AI labs don’t: Their robots are already gaining work experience in real places, such as on factory floors. Such “deployment” experience—and the data these robots gather—helps robots adapt to specific workplaces and tasks. Better foundational models make that site-specific data more valuable because robots can learn from it faster, Tedrake said.
Recent experiments have tested how much work a general-purpose model can do without specialized training on robotic data. Researchers gave Astra camera images, robot information and a control interface that allowed the model to command the robot to take certain actions. In an Nvidia study, an Astra-powered agent achieved a 74.8% success rate on a simulated manipulation benchmark—where the model commands a simulated robot in a digital simulation, as opposed to a real robot—compared with 51.0% reported by the best specialized robotic models on this leaderboard.
Shi said advances in general models’ spatial and reasoning capabilities already let them handle tasks that previously required dedicated training. They can cope when a bottle is in a different position or the lighting changes, and reason about the sequence of a task or which step went wrong, he said.
Meanwhile, large language models are powerful but slow and need to be hosted in the cloud, meaning the robot or device needs a constant network connection. Shi still sees a need for smaller, specialized models that can run on a robot’s onboard computer without needing a constant internet connection.
“There may be different starting points for how to grow intelligence,” Pete Florence, cofounder of Generalist, a startup training general-purpose AI models for robots, told Fortune. He said it was unclear that any existing model would be the one that dominates robotics in the long-run.
Back at booth #208 at CoRL, a few former members of the once-unified, stand-alone OpenAI robotics team milled about. They still drew a crowd, despite having no display or demo. After all, one of the conference’s hottest questions was how much robotics expertise would remain valuable as general-purpose models grew more capable—and the experience of those former OpenAI robotics researchers, now dispatched to other teams, offered one answer.