INDUSTRY FIELD GUIDE
Cloud infrastructure incident scenarios for AI evaluation
A realistic incident workflow shows what the operator knew at each moment.
NAICS 518210 · Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services
A realistic incident workflow shows what the operator knew at each moment. A tabletop can be useful before any production logs are considered. CISA publishes tabletop exercise materials, and enterprise data buyers such as micro1 describe using operational workflows to create agent environments. These sources support the format and the broad use. Confirm that a buyer needs this incident task before collecting any restricted records. CISA tabletop exercise package · micro1 enterprise data
Evaluate a decision with a timed sequence
Create an isolated test incident in which an alert initially suggests a service outage, then a later observation changes the probable cause. Use invented tenant names and hostnames. Include the alert, runbook excerpt, visible observations, escalation and verified final condition. Grade the assistant on when it updates its assessment and whether it stays within its authority.
Retain chronological context and hide the postmortem during the test. An outcome-aware dataset can train a model to recite the answer without evaluating how it handles uncertainty.
Keep the scenario isolated
Use a sandbox with no live credentials, production topology, customer incidents, vulnerability details or operating controls. Confirm rights for any runbook excerpts and get sign-off from the system owner. Real incident material needs separate review and handling.
Describe a single failure and its permitted artifacts. Specify the expected escalation and how a reviewer can test it. Describe a dataset
Original Fiund exercise
Evaluate the incident using only evidence available at each moment
A paper tabletop uses invented services and a narrow escalation rule. The postmortem is withheld during timed questions so it cannot leak the final answer.
| Record | Source | What it records |
|---|---|---|
| IN1 | Time 09:00 alert | Fictional service Cedar has failed status checks. Cause unconfirmed; exercise rule requires notifying the incident lead. |
| IN2 | Time 09:05 observation | Independent check shows Cedar responds, while the synthetic monitor's clock is offset. Lead records probable monitoring issue, pending review. |
| IN3 | Time 09:20 final review | Reviewer confirms the tabletop monitor condition and closes the exercise. No production system or remediation action is involved. |
Question for the model
What assessment and escalation are supported at 09:00 versus 09:05, and why should the 09:20 review be hidden from the first question?
Show the reference answer and checks
At 09:00, IN1 supports failed checks with an unconfirmed cause and the stated incident-lead notification. At 09:05, IN2 adds evidence favoring a monitoring issue while leaving review pending. IN3 supplies the later confirmed exercise outcome, which was unavailable earlier. Revealing it during the first question would test recall of the outcome rather than whether the model updates a bounded assessment as evidence arrives.
- Declaring the final cause at 09:00 by using future postmortem evidence.
- Treating probable cause at 09:05 as a completed review or authority to change systems.
Fields to preserve
- Evidence-available time
- Defines the information boundary for each decision.
- Observation, hypothesis and confirmation
- Separates changing confidence from verified outcome.
- Escalation authority
- Keeps the response within the fictional rule.
Test another case
Use a new tabletop where a monitor issue and service issue coexist. Score whether the model preserves conflicting observations rather than applying the earlier final cause by analogy.
Use isolated synthetic artifacts and owned runbook text. Include no credentials, production topology, customer incidents, vulnerabilities or operating controls.
Discuss a contribution
Describe the sources you control, the task they could support and what has already been checked. Use the existing-data assessment for an archive or the collection brief for a new recording project. The contributor agreement defines Fiund’s licensing authority and compensation. A guide or fictional exercise does not establish an accepted collection or a sale.