Frontier modalities
Narrated egocentric data: collect useful task descriptions
Keep spoken narration, transcripts, action segments and generated descriptions distinct and linked to the original timeline.
- Topic
- Frontier modalities
- For
- Buyers and data owners
- Reading time
- 5 min
- Updated
Narration can explain a task while first-person video shows it, but the spoken account may lag, anticipate or omit the visible action. Decide whether the project needs concurrent speech, a later explanation, instructions or an annotation generated by another process.
The downloadable paired video/text demo is an original synthetic illustration of timeline structure, not human capture, observed timing quality or Fiund inventory. Use it to inspect file relationships; obtain an appropriate real pilot before a collection commitment.
| Question | Evidence or decision |
|---|---|
| Narration | Concurrent or retrospective, original recording reference |
| Transcript | Human or automatic origin, conventions and review |
| Action label | Visible evidence, time span and uncertainty |
| Generated description | Model/version and distinction from recorded speech |
Define the language product
Concurrent narration is recorded while performing the task. Retrospective narration is recorded later. A transcript represents speech; an action description summarizes behavior. Keep these labels distinct instead of presenting all text as what the operator said at the exact event time.
Agree language and variety, level of detail, treatment of hesitations and whether narration is optional. Avoid collecting unnecessary personal or workplace information in spoken explanations. Permissions for the recording and intended use need the applicable review.
Preserve timing and source references
Keep source audio, video and available timestamps. Document synchronization, pauses, edits and transcript timing conventions. Do not align an explanation to a contact event just because the text mentions it. Record uncertainty when boundaries are approximate.
An illustrative annotation might say action segment 2.0–4.0 seconds: place block in tray; narration segment 1.5–3.5 seconds: I am putting the block away. Those invented values demonstrate different time spans, not measured alignment.
Identify who produced each label
Store narration method, transcript source, review status and any generated-description model and version. Keep a reviewed transcript separate from a machine draft. If someone annotates the task after watching video, do not attribute that sentence to the recorded operator.
Review whether a description is supported by the visible evidence. Avoid adding intent, force, contact or success labels that the recording does not establish. Let the schema represent unknown and ambiguous events.
Test the complete pilot
Review task visibility, speech intelligibility, timing and language accuracy across representative sessions. Narration can change behavior or distract an operator, so plan safe tasks and evaluate the effect instead of assuming speech always improves the source.
Deliver a manifest linking episode, video, audio and text versions. The synthetic demo demonstrates packaging only; it does not establish speech quality, natural behavior or buyer acceptance. Scope any transcription and review work separately before scaling.
Frequently asked questions
Is a generated caption a transcript?
No. A transcript represents recorded speech; a generated caption is a derived description.
Is the downloadable demo a real captured human session?
No. It is explicitly synthetic and only illustrates the file/timeline structure.
Have a specific collection requirement?
Send your requirements →Related resources
Related reading
Download synthetic paired demo
6 min read
Low-resource language data: pricing and collection planning
Scope speech and text collection by named language, dialect, native review, accepted units and participant compensation.
5 min read
A pilot acceptance plan for robotics data
Write acceptance, source-aware split boundaries and rework rules before expanding a robotics collection.
5 min read
Egocentric video versus teleoperation versus UMI data
Separate observations of human activity from robot actions and gripper-based demonstrations before choosing training data.
5 min read
Egocentric data collection pay by program and country
Check dated official program terms, eligibility and billing units without turning unrelated job rates into footage prices.
5 min read
Recording workplaces for AI: permissions and preparation
Review site access, workers, bystanders, confidential material and intended use before recording a workplace for a dataset.