Frontier modalities
Egocentric video versus teleoperation versus UMI data
Separate observations of human activity from robot actions and gripper-based demonstrations before choosing training data.
- Topic
- Frontier modalities
- For
- Buyers and data owners
- Reading time
- 5 min
- Updated
Egocentric video records a first-person observation of activity. Robot teleoperation records a person controlling a robot, potentially with paired states and commands. UMI is a particular handheld-gripper demonstration and policy framework. They are not interchangeable labels for people manipulating objects.
Start with what the model must learn and which native signals are actually available. A video showing a successful action is not automatically an executable robot-action trajectory.
| Question | Evidence or decision |
|---|---|
| Required learning signal | Observation, commanded action, measured state or gripper trajectory |
| Source evidence | Capture system, schema, clocks and processing |
| Compatibility | Robot or policy interface and a tested sample |
| Quantity | Accepted episodes and distinct task coverage |
Human video provides observations, not implicit robot commands
An egocentric recording may show task order, objects, failures and context. Camera placement, continuity and action visibility affect usefulness. Report pose or depth generated later as derived annotations, separate from what the source recorded.
Do not infer robot joint states, gripper opening or force values from a descriptive video label. If those are missing, say so. The buyer may use the footage for a different objective, but that does not make missing control signals available.
Teleoperation needs the robot-side record
Ask for robot platform, control mode, observation timestamps, state and action schema, units, control frequency and resets. A commanded action and the measured resulting state are distinct signals. Identify latency and any resampling rather than treating row alignment as physical simultaneity.
Document operator interventions, failed attempts, episode boundaries and safety constraints. Keep derivatives linked to native logs. Different robots or control interfaces may require adaptation before data can support a target policy.
UMI is a framework, not every wrist video
The original UMI project combines handheld grippers with a designed interface and relative-trajectory policy representation. A claimed UMI package should identify its capture and processing implementation, trajectories, gripper information and applicable calibration or timing records.
Ask what is native to the device and what is reconstructed or estimated. UMI-style hardware variations need their own documentation; the research project does not certify a commercial supplier’s implementation or the suitability of a particular dataset.
Evaluate the package in the intended workflow
Request a pilot that imports into the actual training or evaluation pipeline. LeRobot documentation illustrates a format separating visual data, tabular signals and metadata; a format label alone does not guarantee all required features are populated.
Compare unique task coverage, accepted episodes, actual action signals, permissions and adaptation work. Do not compare video hours with robot-action rows as if they were the same quantity. Fiund confirms availability and scope after review, without implying that all three products are in stock.
Sources and further reading
Frequently asked questions
Is head-camera footage teleoperation?
No. Teleoperation involves controlling a robot; the delivered action and state record must be specified.
Does LeRobot format prove a dataset has robot actions?
No. Inspect the actual schema and source of each field.
Have a specific collection requirement?
Send your requirements →Related resources
6 min read
Head and wrist cameras: what each view adds
Plan complementary egocentric viewpoints without mistaking moving head/wrist cameras for a fixed stereo pair.
5 min read
Hand-pose annotations: measured versus estimated labels
Inspect reference systems, coordinates, uncertainty and processing before treating hand pose as ground truth.
5 min read
Egocentric data collection pay by program and country
Check dated official program terms, eligibility and billing units without turning unrelated job rates into footage prices.
5 min read
Recording workplaces for AI: permissions and preparation
Review site access, workers, bystanders, confidential material and intended use before recording a workplace for a dataset.
5 min read
Egocentric data providers: an evidence-based comparison
Compare catalog claims, custom sourcing, closed-loop programs and technical evidence rather than ranking unverified inventory.
5 min read
Off-the-shelf versus custom egocentric collection
Choose an existing subset, new recording or a hybrid only after reviewing task coverage, missing signals and pilot cost.