“`html
A person clearing clutter from an attic can frequently infer the contents of a box merely by lifting it and giving it a shake, without needing to look inside. Scholars from MIT, Amazon Robotics, and the University of British Columbia have trained robots to perform a similar function.
They devised a method that permits robots to utilize solely internal sensors to gather information about an object’s weight, softness, or contents by grasping it and subtly shaking it. Using this approach, which does not necessitate external measurement devices or cameras, the robot can accurately estimate parameters such as an object’s mass within seconds.
This economical method could prove particularly advantageous in scenarios where cameras may be less effective, for example, sorting items in a dim basement or clearing debris in a building that partially collapsed following an earthquake.
Central to their strategy is a simulation process that integrates models of the robot and the object to swiftly determine the characteristics of that object as the robot interacts with it.
The researchers’ technique is as adept at estimating an object’s mass as some more intricate and costly methods that incorporate computer vision. Additionally, their data-efficient approach is resilient enough to manage various types of unfamiliar situations.
“This concept is widely applicable, and I believe we are merely beginning to explore what a robot can learn through this method. My aspiration would be to have robots venture into the world, touch items and move objects in their surroundings, and discern the properties of everything they come into contact with autonomously,” remarks Peter Yichen Chen, an MIT postdoctoral researcher and chief author of a paper on this technique.
His coauthors comprise fellow MIT postdoc Chao Liu; Pingchuan Ma PhD ’25; Jack Eastman MEng ’24; Dylan Randle and Yuri Ivanov from Amazon Robotics; MIT professors of electrical engineering and computer science Daniela Rus, who directs MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL); and Wojciech Matusik, who heads the Computational Design and Fabrication Group within CSAIL. The research will be showcased at the International Conference on Robotics and Automation.
Sensing signals
The researchers’ technique employs proprioception, which is the capability of a human or robot to sense its movement or position within a space.
For example, a person lifting a dumbbell at the gym can perceive the weight of that dumbbell through their wrist and bicep, even though it is held in their hand. Similarly, a robot can “sense” the weight of an object via the various joints in its arm.
“A human doesn’t have super-precise measurements of the joint angles in our fingers or the exact amount of torque we apply to an object, but a robot does. We leverage these capabilities,” Liu explains.
As the robot lifts an object, the researchers’ system collects signals from the robot’s joint encoders, which are sensors that track the rotational position and speed of its joints during movement.
Most robots are equipped with joint encoders within the motors that operate their movable parts, Liu adds. This renders their method more cost-effective than some approaches because it does not require additional components like tactile sensors or vision-tracking systems.
To assess an object’s properties during robot-object interactions, their system relies on two models: one that simulates the robot and its movements, and another that models the dynamics of the object.
“Creating an accurate digital twin of the real-world is crucial for the success of our method,” Chen emphasizes.
Their algorithm “observes” the robot and object during a physical interaction and utilizes joint encoder data to backtrack and identify the properties of the object.
For instance, a more substantial object will move more slowly than a lighter one if the robot exerts the same force.
Differentiable simulations
They employ a method known as differentiable simulation, which enables the algorithm to forecast how minor alterations in an object’s characteristics, such as mass or softness, influence the robot’s final joint position. The researchers constructed their simulations using NVIDIA’s Warp library, an open-source development tool that supports differentiable simulations.
Once the differentiable simulation aligns with the robot’s actual movements, the system has pinpointed the accurate property. The algorithm can accomplish this in mere seconds and only requires one real-world trajectory of the robot in action to execute the calculations.
The researchers applied their method to discern the mass and softness of an object, but their technique could also ascertain properties like moment of inertia or the viscosity of a liquid within a container.
Moreover, as their algorithm does not require an extensive dataset for training like some methods based on computer vision or external sensors, it would not be as prone to failure when faced with unfamiliar environments or new objects.
In the future, the researchers aim to explore combining their method with computer vision to develop a multimodal sensing method that is even more potent.
“This work is not intended to replace computer vision. Both techniques have their advantages and disadvantages. However, here we have demonstrated that, even without a camera, we can already ascertain some of these properties,” Chen states.
They also intend to investigate applications involving more sophisticated robotic systems, such as soft robots, and more complex objects, including sloshing liquids or granular materials like sand.
In the long run, they aspire to apply this technique to enhance robot learning, enabling future robots to rapidly acquire new manipulation skills and adapt to changes in their surroundings.
“Determining the physical characteristics of items from data has long posed a challenge in robotics, particularly when only limited or noisy measurements are accessible. This work is consequential because it demonstrates that robots can accurately infer properties such as mass and softness using solely their internal joint sensors, without reliance on external cameras or specialized measurement tools,” remarks Miles Macklin, senior director of simulation technology at NVIDIA, who was not involved in this research.
This work is partially funded by Amazon and the GIST-CSAIL Research Program.
“`