Resources
Agentic trajectory and computer-use datasets, explained
Agents learn from recorded task sequences: what was on screen and what the user did. Here is what a trajectory contains, and why consent and PII decide whether you can use it.
Published 2026-07-22 · 6 min read
Key takeaways
- A trajectory pairs each observation (the screen) with a bounded action (click, type, scroll) toward a goal.
- Coverage is a choice: the task distribution you train on is the task distribution the agent can handle.
- Real demonstrations capture recovery and edge states that synthetic flows miss; most programs mix both.
- Captured screens are dense with PII; consent, redaction, and staged accounts are prerequisites.
- Source to a brief so real trajectories arrive with rights and PII handling settled.
An agentic trajectory is a record of a task being done. It captures the sequence of states and actions: what was on the screen, where the cursor moved, what was clicked or typed, and how the task ended. Train on enough of them and a model learns to operate software the way a person does.
Public benchmarks show the shape of the data. Mind2Web collected over 2,000 tasks across 137 websites in 31 domains, with human action sequences. OSWorld built 369 tasks in real Ubuntu, Windows, and macOS environments, with execution-based scoring. These measure whether an agent can act. They also make clear how much of the useful signal lives in real, human-recorded workflows.
What a trajectory contains
A trajectory is more than a screen recording. Each step pairs an observation with an action. The observation is the screen, often with the accessibility tree or the underlying page structure. The action is a bounded event: a click at a location, a keystroke, a scroll, a key combination. A goal ties the steps together, and an outcome marks success or failure.
Quality comes from completeness. Missing intermediate steps, unlabeled goals, or actions that do not map to what the screen showed all reduce the value. A clean trajectory can be replayed and checked, which is why execution-based benchmarks like OSWorld score the end state rather than the path.
Web, desktop, and mobile
Coverage splits by surface. Web agents learn from browser tasks; Mind2Web and WebArena are reference points. Desktop agents need operating-system tasks across applications and files, which OSWorld targets. Mobile is its own domain, with on-screen gestures and app flows.
A buyer’s brief should name the surfaces and the applications. An agent trained only on shopping sites will not file an expense report. The distribution of tasks in the training data becomes the distribution of tasks the agent can handle.
Real versus synthetic trajectories
Synthetic trajectories are generated: scripted flows, or paths produced by an existing agent in a simulator. They scale cheaply and cover rare states on demand. They also inherit the quirks of the generator and can miss the messy recoveries that real users perform.
Real trajectories are recorded from people doing genuine tasks. They capture hesitation, error correction, and the odd interface states that scripts do not anticipate. Most programs use a mix: real demonstrations to anchor behavior, synthetic data to fill gaps. The real portion is the harder to source, because it comes from people and their screens.
Consent and PII in captured workflows
A screen recording of real work is dense with personal data. Names, emails, account numbers, messages, documents, and internal tools all pass through the frame. Capture a workflow and you capture whatever was on screen, including third parties who never agreed to appear.
This is why consent and handling matter more here than almost anywhere. Contributors have to agree to record, and to what happens to the recording. Sensitive fields need redaction. Some tasks should run in staged accounts with synthetic data rather than live systems. Without those controls, a trajectory set is a data-protection problem wearing a training-data label.
Sourcing trajectories to a brief
A brief defines the surfaces, the applications, the task list, and the handling rules. It states how PII is treated, which environments are staged, and what a completed trajectory must contain to count.
Sourced-to-brief capture returns trajectories recorded by consenting contributors, with redaction applied and provenance attached. Contributors keep ownership and license the use for AI training. For agent programs, that combination, real human task sequences with the rights and PII handling settled, is what turns a benchmark score into a model you can deploy.
Trajectory capture spec
- Surfaces and applications named (web, desktop, mobile)
- Task list with clear goals and success criteria
- Observation format: screen plus structure or accessibility tree
- Action schema: click, type, scroll, key combinations
- PII handling: redaction rules and staged-account use
- Consent per contributor and provenance per trajectory
Sources
Frequently asked questions
What is the difference between a trajectory and a screen recording?
A screen recording is pixels over time. A trajectory adds structure: each step links the screen state to the specific action taken and to the task goal, so the sequence can be replayed, scored, and trained on.
Are synthetic trajectories good enough on their own?
They help with scale and rare states, but they inherit the generator’s blind spots and often miss real error recovery. Most teams anchor on real human demonstrations and use synthetic data to fill coverage gaps.
How do you keep personal data out of the training set?
Through consent, redaction of sensitive fields, and staging tasks in accounts with synthetic data. Provenance records what was captured and how it was handled, which is what diligence checks.
Related resources
Want data that clears this in diligence?
Whether you're building a model or sitting on an archive, the first conversation is short and specific.
Send a brief