Robotics has spent decades being good at repetition. Put a machine in a controlled setting, define the task carefully, and it can execute the same motion with impressive consistency. The trouble begins when the world changes: an object is moved, a human gives an instruction in slightly different words, a workspace is unfamiliar, or the intended task is not captured by a rigid script. That gap between repeatable automation and useful general-purpose assistance is where much of the current excitement around generative AI, large foundational models, and robotics is concentrated.
In 2024, Google’s DeepMind Robotics researchers are among many teams exploring what happens when models that can interpret language and images are connected to physical machines. The potential applications extend from learning and product design to environments where the robot must respond to circumstances rather than merely follow a prewritten sequence. There is a great deal of anticipation surrounding the possibilities of training robotics with DeepMind, but the important question is less whether robots can perform demonstrations and more whether they can cope with variation.
Table of Contents
From scripted machines to adaptable systems
DeepMind’s current emphasis is on giving training robotics a better understanding of what humans expect from them. That sounds simple, but it is a fundamental shift in how robots are designed to operate. Instead of repeating the same task over and over, a robot needs to recognize changes in its environment or mission parameters and react appropriately.
In a factory, that could mean handling an item that is not in its usual position. In a hospital, it could mean navigating instructions and surroundings that cannot be reduced to a fixed production-line routine. In a home, the variation is even more pronounced: people describe goals differently, objects are moved constantly, and the definition of success can depend on context. A robot that only works when every condition has been arranged in advance is useful automation; a robot that can interpret a changing scene begins to look more broadly applicable.
That does not mean language models alone solve robotics. A machine can produce a plausible verbal interpretation of a request without being able to safely carry out the physical action. Robotics still has to contend with perception, movement, object interaction, and the fact that physical errors have real consequences. The appeal of large foundational models is that they may provide a more flexible layer between a person’s intent and a robot’s actions, rather than requiring every variation of an instruction to be hard coded.
AutoRT connects language, vision, and robot fleets
The DeepMind team designed AutoRT to harness large foundational models for several different ends. The system uses a Visual Language Model, or VLM, for improved situational awareness. In practical terms, combining visual and language capabilities matters because a robot must relate words to what it sees. A command is not enough if the system cannot identify relevant objects or make sense of the environment in which it is operating.
AutoRT also enables a fleet of robots to work in tandem, using cameras to map their environment and identify objects. Fleet-level operation is a meaningful part of the project. Individual robot demonstrations can show that a particular task is possible, but developing training and operational methods that extend across multiple machines is a different challenge. Different devices may encounter different scenes, and a useful system needs a way to turn that spread of experience into learning rather than treating each robot as an isolated experiment.
The hardware can accomplish tasks suggested by a large language model, or LLM. LLMs are widely believed to be key to helping robots understand more natural-language commands and reducing the need for hard coding skills. The wording matters here: natural language is a more accessible way for people to express goals, but it is not a substitute for control. The real value lies in translating broad human requests into actions that remain grounded in visual information and physical constraints.
AutoRT has been extensively tested over the past seven months and can manage up to 20 robots and 52 devices simultaneously. DeepMind has conducted 77,000 trials and completed over 6,000 tasks. Those figures point to a research effort that is trying to move beyond a handful of polished examples. Repeated trials are especially important in robotics because a system that succeeds once can still fail when object positions, lighting, instructions, or movement paths change.
There is also a useful distinction between managing a collection of devices and delivering a finished all-purpose robot. AutoRT’s reported scale is evidence of a training and experimentation approach, not a promise that robots are ready to handle every real-world assignment. That distinction is often lost in discussions of AI-powered machines. Progress can be substantial without erasing the practical difficulty of deploying robots around people and in unpredictable settings.
Why video-based learning could matter
Alongside AutoRT, DeepMind has developed RT-Trajectory, which uses video input to teach robots. Many teams are using YouTube videos to train robots on a large scale. Video is attractive because it contains an enormous range of human activity, objects, and environments. Yet ordinary video has a limitation for robot learning: it can show what happened without clearly expressing the path of a robot arm or the motion information needed to reproduce the action.
RT-Trajectory addresses that by adding a two-dimensional sketch of the arm in action over the video. The trajectories are represented as RGB images, providing practical visual cues to the model as it learns robot-control policies. This is a pragmatic idea. Rather than asking a system to infer every aspect of motion from raw footage, the visual overlay gives it a more direct representation of how an action unfolds.
DeepMind reports that RT-Trajectory training had double the success rate of RT-2 training, achieving 63% success on 41 tasks compared to 29%. The comparison is significant because it suggests that the representation of motion can be as important as the sheer quantity of training material. More data is not automatically better data. If the model cannot connect visual observations to the trajectory required for physical control, a vast video collection may have limited value for manipulation tasks.
DeepMind emphasizes that RT-Trajectory takes advantage of abundant data from robotic motion that is currently not being utilized. That is arguably the most consequential part of the work. Robotics has long faced a data problem: physical interactions are slower and more expensive to collect than text or images, while useful motion data can be difficult to format for learning systems. Finding ways to reuse existing motion information could make experimentation more efficient and allow researchers to extract more value from datasets already available.
RT-Trajectory takes another step toward constructing robots that can move with efficient accuracy in new scenarios while unlocking knowledge from existing datasets. The result should be viewed as progress toward adaptability, not a declaration that the problem has been solved. Still, the combination of AutoRT’s language-and-vision-driven coordination and RT-Trajectory’s focus on motion learning shows where DeepMind sees the next gains: systems that do not simply recognize instructions or imitate isolated actions, but connect perception, intent, and movement in environments that refuse to stay still.
