Industry field guide
Cloud infrastructure incident scenarios for AI evaluation
A realistic incident workflow shows what the operator knew at each moment.
- 2022 NAICS
- 518210
- Practice pack
- Example and answer key (JSON)
- Updated
What this guide tests
What assessment and escalation are supported at 09:00 versus 09:05, and why should the 09:20 review be hidden from the first question?
See the specimen and fields · Work through the fictional records
Task and source context
A realistic incident workflow shows what the operator knew at each moment. A tabletop can be useful before any production logs are considered. CISA publishes tabletop exercise materials, and enterprise data buyers such as micro1 describe using operational workflows to create agent environments. These sources support the format and the broad use. Confirm that a buyer needs this incident task before collecting any restricted records. CISA tabletop exercise package · micro1 enterprise data
Evaluate a decision with a timed sequence
Create an isolated test incident in which an alert initially suggests a service outage, then a later observation changes the probable cause. Use invented tenant names and hostnames. Include the alert, runbook excerpt, visible observations, escalation and verified final condition. Grade the assistant on when it updates its assessment and whether it stays within its authority.
Retain chronological context and hide the postmortem during the test. An outcome-aware dataset can train a model to recite the answer without evaluating how it handles uncertainty.
Keep the scenario isolated
Use a sandbox with no live credentials, production topology, customer incidents, vulnerability details or operating controls. Confirm rights for any runbook excerpts and get sign-off from the system owner. Real incident material needs separate review and handling.
Describe a single failure and its permitted artifacts. Specify the expected escalation and how a reviewer can test it. Describe a dataset
Industry research
The U.S. Census Bureau’s industry definitions establish which activities these codes cover. The workflow sources in this guide describe the task or record type. Fiund’s pilot below uses fictional material to test a specific decision within that scope.
518210 · Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services
This industry comprises establishments primarily engaged in providing computing infrastructure, data processing services, Web hosting services (except software publishing), and related services, including streaming support services (except streaming distribution services). Data processing establishments provide complete processing and specialized reports from data supplied by clients or provide automated data processing and data entry services.
Build a pilot sample
Set up this pilot
Stage synthetic incident notes in three timed releases. Run the same prompt at the first two cutoffs and score escalation without giving the model the later root-cause review.
Prepare the test packets
- Start with the worked case. Use the fictional record types below as a guide to the source files you need. Give each packet a stable ID and save its source versions with it.
- Change one fact. Make one packet with enough evidence to answer the question and one where the evidence points to a different answer. Keep the other fields fixed so a reviewer can tell what changed.
- Remove the decisive evidence. Give the reviewer a third packet with a plausible near-match in its place. Have them write what can still be established and what evidence would resolve the rest.
Fields the reviewer needs
- Evidence-available time
- Defines the information boundary for each decision.
- Observation, hypothesis and confirmation
- Separates changing confidence from verified outcome.
- Escalation authority
- Keeps the response within the fictional rule.
Use Who buys company workflow data for AI? to turn the specimen into a brief with a defined task, source and review owner.
Fictional exercise
Evaluate the incident using only evidence available at each moment
A paper tabletop uses invented services and a narrow escalation rule. The postmortem is withheld during timed questions so it cannot leak the final answer.
| Record ID | Source | What it records |
|---|---|---|
| IN1 | Time 09:00 alert | Fictional service Cedar has failed status checks. Cause unconfirmed; exercise rule requires notifying the incident lead. |
| IN2 | Time 09:05 observation | Independent check shows Cedar responds, while the synthetic monitor's clock is offset. Lead records probable monitoring issue, pending review. |
| IN3 | Time 09:20 final review | Reviewer confirms the tabletop monitor condition and closes the exercise. No production system or remediation action is involved. |
Question for the model. What assessment and escalation are supported at 09:00 versus 09:05, and why should the 09:20 review be hidden from the first question?
Show the reference answer
At 09:00, IN1 supports failed checks with an unconfirmed cause and the stated incident-lead notification. At 09:05, IN2 adds evidence favoring a monitoring issue while leaving review pending. IN3 supplies the later confirmed exercise outcome, which was unavailable earlier. Revealing it during the first question would test recall of the outcome rather than whether the model updates a bounded assessment as evidence arrives.
How to check the result
Ask for the answer, the record IDs that support it and any fact the evidence leaves unresolved. Have a reviewer compare the response against the reference before deciding whether the sample is ready to expand.
Errors to flag in this case
- Declaring the final cause at 09:00 by using future postmortem evidence.
- Treating probable cause at 09:05 as a completed review or authority to change systems.
Change the inputs and run it again
Use a new tabletop where a monitor issue and service issue coexist. Score whether the model preserves conflicting observations rather than applying the earlier final cause by analogy.
Score the decision and the evidence separately
Save the reviewer’s expected answer, supporting record IDs and unresolved fields in a separate answer key. Score the model on the decision, its citations and its handling of the deliberate gap. A correct answer citing a different case counts as an evidence error.
Before expanding collection, have a second reviewer run the same checks on the held-out packet. Keep related versions, views and exports of the same case together so they stay on one side of the training and evaluation split.
Check permission and scope
Use isolated synthetic artifacts and owned runbook text. Include no credentials, production topology, customer incidents, vulnerabilities or operating controls.
Remove hostnames, tenant information, credentials, configuration, vulnerabilities and internal operational details.
Describe an authorized archive through Fiund’s existing-data assessment, or describe a new capture capability through the collection brief. The contributor agreement covers the terms for participation.
Sources and further reading
Resources for this workflow
Company and program reviews
Use these reviews to compare sourcing, capture or evaluation approaches. Confirm the applicable program, accepted sources and terms with the provider.
Provider cited in this guide
micro1 review for experts, trade footage and company data
The guide cites material from this provider. The review explains the offering and which details require a specific brief or agreement.
Compare the approach
Scale AI review: data partnerships, Outlier and buyer services
Compare its structured operational-data offer and approval model for the relevant company workflow.
Compare the approach
Surge AI review for frontier data and agent evaluation
Compare how to scope task evaluation when the training question depends on timeline or evidence.
Related industry examples
Industry guide · NAICS 517111
Wired telecom service incident data for AI dispatch review
A dispatcher has to distinguish a single service complaint from evidence of a wider incident.
Industry guide · NAICS 513110
Newspaper correction and article version data for AI
A newspaper archive has both published stories and later corrections.
Industry guide · NAICS 512132
Drive-in screening schedule revisions for AI testing
A changed outdoor showtime may appear in a notice while the earlier schedule remains in the booking tool.