AI Insight
MIT researchers have developed VLASH, a new method that enables robots to plan their next movements while simultaneously executing current actions by predicting their future position and environmental state. Unlike conventional approaches that cause robots to pause between action sequences, this technique allows AI models to use forecasted robot states rather than current positions for motion planning. The method doubled robot speed in pick-and-place tasks and improved performance in dynamic activities like table tennis, without adding computational overhead.
Why it matters
This advancement could significantly improve robots operating in time-critical environments such as emergency response and search-and-rescue operations, where fast, fluid movements are essential. The technique is hardware-agnostic and adds no extra computational cost, making it broadly applicable across different robotic systems and potentially accelerating the deployment of more responsive autonomous robots in real-world settings.
Understand the Science
A new method developed by MIT researchers makes robots better at thinking ahead while they are acting, leading to smoother motions and quicker reactions.
This technique enables the artificial intelligence model that plans a robot’s motion to forecast its future position. The model uses this prediction to seamlessly transition current movements into the next actions.
Many existing methods cause a robot to stop and think about what it needs to do next, leading to slow and jerky motions. By basing its calculations on the future state of the robot, rather than its current position, the MIT method helps robots operate much faster.
Importantly, the technique does not add any computational overhead to the planning process and can be applied to varied robotic hardware.
This new method doubled the speed of robots performing activities like pick-and-place tasks, while significantly reducing lag time between motions. It also boosted the performance of robotic arms in highly dynamic activities, such as playing table tennis and Whack-a-Mole.
The system could be especially useful for robots that perform fast and agile maneuvers in challenging real-world environments, like emergency response or search-and-rescue. It could also allow robots to react more quickly when recovering from mistakes.
“This work sets up a good foundation for efficient, fast, accelerated, and low-cost robotics applications. We look forward to expanding our work into the latest world action models, so it has even stronger capabilities as we keep pushing to make physical AI faster,” says Song Han, an associate professor in the MIT Department of Electrical Engineering and Computer Science (EECS), member of the Research Laboratory of Electronics, and lead author of a paper on this method.
Han is joined on the paper by co-lead authors Jiaming Tang, an MIT EECS graduate student, and Yufei Sun, a student at Tsinghua University; as well as others at Nvidia, the University of California at Berkeley, the University of California at San Diego, and Caltech. The research will be presented at the Intelligent Robots and Systems Conference.
Forecasting the future
In state-of-the-art robotics applications, generative AI systems called vision-language-action (VLA) models act as the brain of a robot, planning its next moves and executing those actions.
A VLA model takes environmental observations from the robot’s camera and instructions about its task, outputs the next few motions as one chunk of actions, then executes those actions on the robotic hardware.
But VLA inference — the real-time procedure during which the model processes visual inputs, reasons about the task, and outputs actions — is computationally demanding, so the robot can experience substantial pauses while planning its next actions. These pauses disrupt the fluidity of its motions and make it slower to react to changes in the environment.
“Our motivation was to overlap the thinking process with the execution process to make the reaction speed faster,” Tang says.
The MIT researchers developed a new system called VLASH that enables a VLA to predict the future state of the robot and its environment. It uses this information to plan the next set of motions while the robot is completing the current action chunk.
This solves a major hurdle faced by many other methods, which use the current state of the robot to predict its next moves.
“Since the environment will change after the robot moves, if we plan based on stale observations of the current environment, there will be a misalignment that causes very unstable control,” Tang explains.
VLASH avoids this misalignment due to a key insight by the researchers. Although the model doesn’t know exactly what the environment will look like in the future, it does know the robot’s current position and how it will move to perform the actions it is about to take.
The framework uses this information to predict the state of the robot after it completes its current chunk of actions. It uses that estimation to plan the next motions.
“In this way, we give the robot awareness of its future state,” Tang says.
Augmenting acceleration
On its own, this technique speeds up the robot’s motions by eliminating lag time that usually occurs between action chunks, accelerating reaction speeds more than 30-fold.
But to make their approach even faster, the MIT researchers generate coarser chunks of actions, so the robot executes a few larger steps that follow the same trajectory. This technique is called action quantization.
While action quantization led to a slight dip in accuracy, it enables a robot to complete the overall task two to three times faster.
However, the researchers found that simply feeding future robot states to the VLA during deployment is not enough to enable accurate and stable control of the robot.
They developed a training-augmentation method that groups training data in such a way that the VLA learns to use future state information instead of current observations.
By reusing some training data, this fine-tuning method accelerated training fivefold with no additional computational overhead.
“Even though there is a very large model working in the background, VLASH lets the robot react and execute its actions very fast, much more like a human would. This could help to make robots for all sorts of dynamic tasks more effective,” Tang says.
When compared with baseline methods in simulation, VLASH consistently performed faster while maintaining the accuracy of robotic maneuvers. The system also outpaced these methods on real hardware in pick-and-place, stacking, and sorting tasks.
For instance, VLASH placed cubes in a box while sorting them by color twice as fast as these methods, while achieving the same 90 percent accuracy as the best baseline. The system can also perform highly dynamic tasks like playing ping-pong and whack-a-mole.
In the future, the researchers want to combine VLASH with more powerful generative AI systems called world models that can predict the robot’s actual environmental observations, in an effort to boost performance and open new applications.
This work is supported, in part, by the MIT-IBM Computing Research Lab, Amazon, the National Science Foundation, and Nvidia.