Skip to content

Industry field guide

Cloud infrastructure incident scenarios for AI evaluation

A realistic incident workflow shows what the operator knew at each moment.

2022 NAICS
518210
Updated

What this guide tests

What assessment and escalation are supported at 09:00 versus 09:05, and why should the 09:20 review be hidden from the first question?

See the specimen and fields · Work through the fictional records

Task and source context

A realistic incident workflow shows what the operator knew at each moment. A tabletop can be useful before any production logs are considered. CISA publishes tabletop exercise materials, and enterprise data buyers such as micro1 describe using operational workflows to create agent environments. These sources support the format and the broad use. Confirm that a buyer needs this incident task before collecting any restricted records. CISA tabletop exercise package · micro1 enterprise data

Evaluate a decision with a timed sequence

Create an isolated test incident in which an alert initially suggests a service outage, then a later observation changes the probable cause. Use invented tenant names and hostnames. Include the alert, runbook excerpt, visible observations, escalation and verified final condition. Grade the assistant on when it updates its assessment and whether it stays within its authority.

Retain chronological context and hide the postmortem during the test. An outcome-aware dataset can train a model to recite the answer without evaluating how it handles uncertainty.

Keep the scenario isolated

Use a sandbox with no live credentials, production topology, customer incidents, vulnerability details or operating controls. Confirm rights for any runbook excerpts and get sign-off from the system owner. Real incident material needs separate review and handling.

Describe a single failure and its permitted artifacts. Specify the expected escalation and how a reviewer can test it. Describe a dataset

Industry research

The U.S. Census Bureau’s industry definitions establish which activities these codes cover. The workflow sources in this guide describe the task or record type. Fiund’s pilot below uses fictional material to test a specific decision within that scope.

518210 · Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services

This industry comprises establishments primarily engaged in providing computing infrastructure, data processing services, Web hosting services (except software publishing), and related services, including streaming support services (except streaming distribution services). Data processing establishments provide complete processing and specialized reports from data supplied by clients or provide automated data processing and data entry services.

U.S. Census Bureau, 2022 NAICS Manual, p. 418 ↗

Build a pilot sample

Set up this pilot

Stage synthetic incident notes in three timed releases. Run the same prompt at the first two cutoffs and score escalation without giving the model the later root-cause review.

Prepare the test packets

  1. Start with the worked case. Use the fictional record types below as a guide to the source files you need. Give each packet a stable ID and save its source versions with it.
  2. Change one fact. Make one packet with enough evidence to answer the question and one where the evidence points to a different answer. Keep the other fields fixed so a reviewer can tell what changed.
  3. Remove the decisive evidence. Give the reviewer a third packet with a plausible near-match in its place. Have them write what can still be established and what evidence would resolve the rest.

Fields the reviewer needs

Evidence-available time
Defines the information boundary for each decision.
Observation, hypothesis and confirmation
Separates changing confidence from verified outcome.
Escalation authority
Keeps the response within the fictional rule.

Use Who buys company workflow data for AI? to turn the specimen into a brief with a defined task, source and review owner.

Fictional exercise

Evaluate the incident using only evidence available at each moment

A paper tabletop uses invented services and a narrow escalation rule. The postmortem is withheld during timed questions so it cannot leak the final answer.

Records supplied to the evaluator
Record IDSourceWhat it records
IN1Time 09:00 alertFictional service Cedar has failed status checks. Cause unconfirmed; exercise rule requires notifying the incident lead.
IN2Time 09:05 observationIndependent check shows Cedar responds, while the synthetic monitor's clock is offset. Lead records probable monitoring issue, pending review.
IN3Time 09:20 final reviewReviewer confirms the tabletop monitor condition and closes the exercise. No production system or remediation action is involved.

Question for the model. What assessment and escalation are supported at 09:00 versus 09:05, and why should the 09:20 review be hidden from the first question?

Show the reference answer

At 09:00, IN1 supports failed checks with an unconfirmed cause and the stated incident-lead notification. At 09:05, IN2 adds evidence favoring a monitoring issue while leaving review pending. IN3 supplies the later confirmed exercise outcome, which was unavailable earlier. Revealing it during the first question would test recall of the outcome rather than whether the model updates a bounded assessment as evidence arrives.

Download the fictional records and answer key (JSON)

How to check the result

Ask for the answer, the record IDs that support it and any fact the evidence leaves unresolved. Have a reviewer compare the response against the reference before deciding whether the sample is ready to expand.

Errors to flag in this case

  • Declaring the final cause at 09:00 by using future postmortem evidence.
  • Treating probable cause at 09:05 as a completed review or authority to change systems.

Change the inputs and run it again

Use a new tabletop where a monitor issue and service issue coexist. Score whether the model preserves conflicting observations rather than applying the earlier final cause by analogy.

Score the decision and the evidence separately

Save the reviewer’s expected answer, supporting record IDs and unresolved fields in a separate answer key. Score the model on the decision, its citations and its handling of the deliberate gap. A correct answer citing a different case counts as an evidence error.

Before expanding collection, have a second reviewer run the same checks on the held-out packet. Keep related versions, views and exports of the same case together so they stay on one side of the training and evaluation split.

Check permission and scope

Use isolated synthetic artifacts and owned runbook text. Include no credentials, production topology, customer incidents, vulnerabilities or operating controls.

Remove hostnames, tenant information, credentials, configuration, vulnerabilities and internal operational details.

Describe an authorized archive through Fiund’s existing-data assessment, or describe a new capture capability through the collection brief. The contributor agreement covers the terms for participation.

Sources and further reading

Resources for this workflow

How the examples work

Company and program reviews

Use these reviews to compare sourcing, capture or evaluation approaches. Confirm the applicable program, accepted sources and terms with the provider.

All company reviews

Related industry examples

← Information and publishing