In a recent demonstration, a Google engineer showed a visually-equipped robotic arm three figurines on a table: a dinosaur, a lion, and a whale. The engineer told the arm to “pick up the extinct animal.” The arm reached across and plucked up the dinosaur, even though the robot had not been programmed to know what a dinosaur is.

By uniting robots with large language model artificial intelligence, Google is giving robots new powers of independent action that requires no additional programming by humans.

“We’ve had to reconsider our entire research program as a result of” the opportunity to combine AI and robotics, Vincent Vanhoucke, Google DeepMind’s robotics chief, told The New York Times. “A lot of the things we were working on before have been entirely invalidated.”

Equipping robots with an understanding of language and the fount of information that is the Internet gives them the ability to reason and improvise that represents a “breakthrough,” robotics professor Ken Goldberg at the University of California Berkeley said to the NYT.

Typically, robots are programmed with step-by-step instructions for a specific task. If the task changes even slightly, the robot has to be given a new set of instructions. Each set of instructions has to be fine-tuned and tested until they guide the robot perfectly every time. The human time involved can amount to weeks.

More recently, DeepMind decided to try giving robots access to the Internet to see if they could learn new skills on their own, without humans needing to tell them what to do.

Early experiments enabled the robots to learn instructions but they were unable to process images, so they were “blind” and unable to operate in the physical world.

The newest version, dubbed RT-2, is what Google calls a “vision-language-action” AI model: it can see and understand its surroundings, then tell a robot how to move to accomplish a goal.

The model understands several languages and can relate abstract concepts. In a demonstration, a robot was to pick up a soccer ball. Instead, it was instructed to “pick up Lionel Messi,” the international soccer star. The robot made the association and picked up the ball without hesitation.

“This really opens up using robots in environments where people are,” Vanhoucke said. They could be used as medical assistants, warehouse pickers, office gofers, or household help to fold laundry or put away groceries.

Google says it has programmed its AI with a range of safeguards, such as not picking up a cup with liquid in it that might spill.
“After years in the wilderness, hardware robots are back,” the NYT said, “and they have their chatbot brains to thank.”