Robot learning has a data problem. Modern imitation-learning policies, especially the diffusion and transformer-based ones that power today’s most capable manipulation systems, are hungry for large, diverse demonstration datasets. Collecting that data has traditionally meant one of two things: expensive teleoperation sessions with a physical robot arm, or scraping unstructured human videos from the internet that don’t cleanly map to robot actions. The UMI gripper  short for Universal Manipulation Interface offers a third path, and it’s built around a strikingly simple piece of hardware: a handheld, 3D-printed gripper with a GoPro strapped to it.

Developed by researchers at Stanford, Columbia, and the Toyota Research Institute and presented at Robotics: Science and Systems (RSS) 2024, the UMI gripper reframes robot data collection as something a human can do anywhere, with no robot in the loop at all. That reframing is what makes it worth understanding in detail  how it works, how the data is used, and how it’s already being adopted commercially.

What Is a UMI Gripper?

The UMI gripper is a triggeractivated, spring loaded parallel jaw gripper with soft, compliant fingers, worn or held like a hand tool. Mounted on it is a single GoPro camera with a wide fisheye lens and that camera is, deliberately, the only sensor on the device. There are no extra IMUs bolted on for tracking, no external motion capture rig, no tethered cables. Everything the system needs to reconstruct a demonstration is pulled from that one video stream.

The central insight behind the UMI gripper is that a human hand can demonstrate almost any manipulation task a robot arm might eventually perform folding a shirt, tossing an object into a bin, wiping a counter, washing a dish. The bottleneck isn’t human dexterity; it’s capturing that demonstration in a form a robot policy can actually learn from. The UMI gripper solves this without requiring a robot to be physically present, and without losing the precise, metric-scale motion data that passive video capture typically discards.

What Does UMI Gripper Data Collection Actually Capture?

Each demonstration recorded with a UMI gripper produces a single, self contained MP4 file, and that simplicity is a design choice, not a limitation. From this video, the UMI pipeline extracts several layers of structured data:

Egocentric visual observations. The wide fisheye lens gives a first-person, wrist level view of the workspace  close enough to see fine contact details, wide enough to preserve surrounding context.

6-DoF end-effector trajectory. Using visual-inertial simultaneous localization and mapping (SLAM), built on top of open-source GoPro camera IMU calibration and tracking tools, the UMI gripper recovers the precise 3D position and orientation of the gripper throughout the demonstration, at real metric scale rather than an arbitrary unit.

Gripper width and trigger state. A simple mechanical linkage on the handheld device records how open or closed the jaws are at every moment, giving the policy the analog gripper action signal it needs, not just a binary open/close.

Because all of this is derived from onboard sensing rather than an external tracking setup, a demonstrator using a UMI gripper can walk into a kitchen, a warehouse aisle, an office, or a park and start recording immediately. There’s no calibration rig to set up and no dedicated lab space required which is exactly the “in-the-wild” data collection the framework’s full name promises.

The UMI Gripper Data Collection Workflow

UMI gripper data collection is designed for close to zero setup time. The demonstrator typically follows a short, repeatable protocol:

1. Time synchronization (optional). For bimanual tasks using two UMI grippers, the internal clocks of the two GoPro cameras are synchronized by scanning a rolling QR code, achieving roughly ±1/60 second accuracy.
2. Gripper calibration (optional). The minimum and maximum width between the gripper’s fingers is calibrated by recording a short video of opening and closing the gripper a few times.
3. Demonstration recording. The operator performs the task naturally, holding the UMI gripper the way they would hold any tool, while the GoPro captures egocentric video throughout.
4. Post-processing. SLAM based tracking, gripper state extraction, and validity checks against target robot kinematic constraints convert the raw video into a structured, policy ready dataset.

This lightweight protocol is what lets UMI gripper datasets scale in a way teleoperation simply can’t.

From Human Hand to Robot Action

Recording the demonstration is only half the story. UMI’s second major contribution is a policy interface designed specifically to make UMI gripper data transferable to real robot hardware, despite the obvious embodiment gap between a human hand and a robot gripper.

Two design choices do most of the work here. First, UMI uses a relative-trajectory action representation rather than absolute positions actions are expressed relative to the current gripper pose, which generalizes far better across environments and starting conditions than fixed world-frame coordinates. Second, the system incorporates inference-time latency matching, which accounts for the fact that a real robot’s control loop has different timing characteristics than the handheld UMI gripper used for data collection, so the learned policy behaves consistently once deployed.

The payoff is that policies trained purely on UMI gripper data turn out to be largely hardware agnostic. The same dataset can be used to control different robot arms, because the representation is built around the end-effector’s motion and gripper state rather than any particular robot’s joint configuration. Researchers have demonstrated this transfer on tasks like bimanual sweater folding, dynamic object tossing and sorting, and even washing a dirty dish tasks that are dynamic, contact rich, or require two coordinated grippers, all traditionally hard to teach through conventional teleoperation.

Why UMI Gripper Data Collection Matters for Scaling Robot Learning

The appeal of the UMI gripper isn’t just clever engineering  it’s what that engineering unlocks organizationally. Because the entire demonstration lives inside a standard MP4 file, UMI gripper data can be collected by many people in many places and simply shared over the internet, the same way any video file would be. That turns data collection from a single-lab, single robot bottleneck into something that resembles crowdsourcing: a large, geographically distributed pool of nonexpert demonstrators, each contributing a handful of episodes, can in principle add up to a dataset that no individual lab could collect through teleoperation alone.

This addresses a real structural problem in robot learning. Teleoperated datasets tend to be small, because each episode requires a robot, an operator, and often significant setup time. In the wild human videos are abundant but lack the precise, robot-relevant action labels a policy needs. The UMI gripper sits in between: it’s nearly as portable and cheap as filming a video, but it captures the metric, structured action information that teleoperated data provides.

Industry Adoption: UMI Gripper Data Collection in Commercial Pipelines

The UMI gripper started as an academic research framework, but it has already moved into commercial data infrastructure. Nferent AI, an India based startup building what it calls the “data layer for Physical AI,” lists UMI and mobile manipulation devices as one of its core hardware categories alongside industrial robot arms, grippers, vision systems, and human motion-capture rigs. The company position Universal Manipulation Interface systems for scalable, real-world data collection as a key tool in its pipeline for capturing human skill demonstrations from factories, workshops, and industrial environments across India.

In this kind of commercial workflow, the UMI gripper is used the same way it was originally designed for: an operator demonstrates a task by hand, without needing a robot on-site, and the resulting egocentric video, end-effector trajectory, and gripper-width data becomes the raw input to a larger pipeline of skill labeling, object tagging, action segmentation, and dataset delivery. Vendors like this are essentially offering UMI gripper data collection as an outsourced service, so robotics and humanoid AI companies can acquire production ready manipulation datasets without running their own in-house capture operations. This is a good sign for the framework’s staying power: what started as a research artifact at Stanford is now recognizable enough, and reliable enough, to sit on a commercial data vendor’s hardware list next to industrial arms and motion capture suits.