Robots are getting better at seeing, moving and interacting with the world. But there is still one major challenge: data.
A robot cannot learn physical skills from images or text alone. To pick up a cup, open a drawer, fold a cloth or place an object accurately, it needs examples of how those actions happen in the real world.
This is where UMI (Universal Manipulation Interface) comes into the picture.
UMI is a handheld robot data collection system that allows people to demonstrate manipulation tasks without directly teleoperating a robot. Instead of keeping a robot arm in the loop for every demonstration, a person can use a compact, camera-equipped gripper to record how a task is performed.
The idea is simple: let humans demonstrate physical skills, turn those demonstrations into structured robotlearning data, and use that data to train robots.
What Is UMI?
UMI stands for Universal Manipulation Interface. It was introduced as a framework for collecting human demonstrations and transferring those skills to robot policies.
The system uses a handheld gripper with a camera. A person holds the device and performs a task in the real world. The system captures visual information along with the movement of the gripper and its interaction with objects.
This creates a connection between two important things:
What the human sees → What the human does
That relationship is valuable for robot learning.
The original UMI research focused on making robot teaching more portable and suitable for challenging tasks such as dynamic movements, precise manipulation, bimanual tasks and longer sequences of actions.
How Does UMI Data Collection Work?
A typical UMI workflow can be understood in four stages.
1. A human demonstrates the task
The operator performs the task naturally using the handheld UMI interface.
For example, they might pick up a bottle, arrange objects, fold clothing or perform another manipulation task.
Unlike traditional robot teleoperation, the person does not necessarily need to control the final robot during the demonstration.
2. The system records the demonstration
A camera mounted on the handheld device captures the operator’s view of the workspace.
At the same time, the system captures information about the movement and gripper state. This allows the resulting recording to contain more useful information than an ordinary video.
Instead of simply showing that a person picked up a cup, the data can represent the visual scene, movement of the end effector, gripper position and timing of the action.
3. The demonstration is processed
Raw recordings are not automatically ready for robot training.
The data may need tracking, synchronisation, quality checks, segmentation and formatting before it can be used by a learning system.
This step is important because a useful robot dataset needs more than a large number of recordings. The demonstrations also need to be consistent, relevant and usable.
4. The data can be used for robot learning
Once processed, the demonstrations can become training data for imitation learning and other robot-learning approaches.
The objective is for a robot policy to learn the relationship between what is happening in the environment and the action that should follow.
That is what makes UMI interesting for Physical AI.
Why Is UMI Important for Physical AI?
Traditional AI systems can learn from enormous amounts of digital information.
Robots have a different problem.
A robot needs to understand the physical world. It needs to know how objects move, how much they need to be manipulated, how actions change the environment and how one movement leads to another.
A photograph of a glass can tell an AI model what a glass looks like.
It does not tell a robot how to safely grasp it.
For that, the robot needs action-oriented data.
UMI provides one way to collect this type of information through human demonstrations. Nferent AI also focuses on this broader challenge through Physical AI data collection, where real-world human actions and robot interactions are converted into training data for intelligent machines.
UMI vs Traditional Robot Teleoperation
Robot teleoperation has been an important method for collecting manipulation data. A human operator controls a physical robot while the system records the robot’s movements and sensor information.
The limitation is that every demonstration is tied closely to the robot setup.
UMI takes a different approach.
The demonstration can be collected using a portable handheld interface, separating the data-collection environment from the final robot deployment environment.
This can make it possible to collect demonstrations in more varied real-world environments rather than requiring every person and every task to be performed next to a robot.
The original UMI framework was designed around this idea of transferring human demonstrations into deployable, hardware-agnostic robot policies.
What Kind of Data Can UMI Capture?
A UMI-style demonstration can contain several useful layers of information, including:
- Visual observations of the environment
- Human manipulation movements
- End-effector trajectory
- Gripper opening and closing
- Object interaction
- Timing of actions
- Changes in the surrounding environment
- This makes the data more useful for robot learning than a standard instructional video.
The important part is not just recording the human. It is creating a relationship between observation and action that a robot-learning system can use.
What Role Does Nferent AI Play?
Nferent AI works in the broader area of Physical AI data infrastructure.
The company’s focus is not simply on recording videos. The goal is to help turn real-world human demonstrations and robot interactions into structured datasets that can support the development of Physical AI and robot-learning systems.
For UMI-based data collection, this means Nferent AI can play a role across the data pipeline:
Human Demonstration → Data Capture → Processing → Quality Control → Structured Dataset → Robot Learning
Nferent AI’s UMI data-collection work focuses on capturing human manipulation demonstrations and preparing useful training data for robotics applications. Its published work describes UMI data as including visual observations, manipulation movements, gripper information and the relationship between what is observed and what action is performed.
This fits into Nferent AI’s larger Physical AI vision: helping robotics teams collect the real-world data required to train increasingly capable robots.
Why Data Quality Matters
Collecting thousands of demonstrations is not enough.
Imagine a dataset containing many recordings where the camera loses tracking, tasks are performed differently by every operator, or unsuccessful attempts are mixed with successful demonstrations.
More data would not automatically solve the problem.
A useful UMI dataset needs:
- Consistency — demonstrations should follow a clear collection process.
- Variation — data should include meaningful differences in objects, positions and environments.
- Quality control — poor or incomplete demonstrations should be identified.
- Metadata — information about tasks and recordings should be organised.
- Scalability — the process should work as the dataset grows.
- This is where data engineering becomes just as important as the collection hardware.
UMI and the Future of Robot Learning
UMI represents a broader shift in robotics.
Instead of asking only, “How can humans control robots?”, researchers and companies are increasingly asking:
“How can robots learn physical skills from humans?”
That change is important for Physical AI.
As robot foundation models and imitation-learning systems develop, they will require diverse real-world demonstrations covering different objects, environments, movements and tasks.
UMI is one approach to creating that data.
It does not replace every other data-collection method. Teleoperation, simulation, tactile sensing, RGB-D capture and autonomous robot interaction can all provide different types of information.
Together, these approaches can create richer datasets for training robots.
Final Thoughts
The biggest value of UMI is not the handheld device itself.
It is the idea behind it.
A human can demonstrate a physical skill naturally. That demonstration can be captured, processed and converted into data that a robot-learning system can use.
For Physical AI, this creates an important bridge between human experience and robot capability.
As robotics moves toward systems that can learn more complex real-world skills, scalable data collection will become increasingly important.
And that is where platforms and data infrastructure providers such as Nferent AI can play a role helping transform real world demonstrations into structured, usable data for the next generation of intelligent robots.
