Industry field guide
Municipal plan-review examples for evaluating AI document intake
A plan intake assistant can be evaluated on whether it detects a missing required item and points the applicant to the correct checklist.
- Sector
- Public administration
- 2022 NAICS
- 925120
- Practice pack
- Example and answer key (JSON)
- Updated
What this guide tests
Which intake evidence is missing for X under the dated checklist, and why does Y's packet or old C1 not establish X's completeness?
See the specimen and fields · Work through the fictional records
Task and source context
A plan intake assistant can be evaluated on whether it detects a missing required item and points the applicant to the correct checklist. Decisions about compliance or permits remain with authorized officials. Phoenix and Tempe publish plan-review steps that illustrate the kinds of documented requirements a fictional evaluation can use. These public processes establish a practical task. They do not establish a city’s intent to license applicant files. Phoenix construction plan review · Tempe plan review
Build a versioned intake case
Create a fictional application, simplified drawing and municipal checklist reference. Label which document is missing, cite the dated requirement and specify the point at which a human reviewer must take over. Include a case where the same material is acceptable under one application type and insufficient under another. Keep the official current rule separate from your fictional application.
Confirm reuse rights even when material is accessible
Public access to a record and permission for commercial AI use are different questions. Federal government works have a specific rule under U.S. copyright law. Local government records and materials supplied by private applicants or contractors require their own review. Keep exact property addresses, security-sensitive plans and applicant information out of the example. 17 U.S.C. §105
Share the task and a fabricated intake packet. Specify the jurisdiction, checklist version, required evidence and human review boundary. Describe a dataset
Industry research
The U.S. Census Bureau’s industry definitions establish which activities these codes cover. The workflow sources in this guide describe the task or record type. Fiund’s pilot below uses fictional material to test a specific decision within that scope.
925120 · Administration of Urban Planning and Community and Rural Development
This industry comprises government establishments primarily engaged in the administration and planning of the development of urban and rural areas. Included in this industry are government zoning boards and commissions.
Build a pilot sample
Set up this pilot
Write a fictional dated intake checklist and two application packets. Move one item between revisions, then grade missing evidence under the checklist applicable to the received date.
Prepare the test packets
- Start with the worked case. Use the fictional record types below as a guide to the source files you need. Give each packet a stable ID and save its source versions with it.
- Change one fact. Make one packet with enough evidence to answer the question and one where the evidence points to a different answer. Keep the other fields fixed so a reviewer can tell what changed.
- Remove the decisive evidence. Give the reviewer a third packet with a plausible near-match in its place. Have them write what can still be established and what evidence would resolve the rest.
Fields the reviewer needs
- Application type and received date
- Selects the relevant practice requirements.
- Checklist revision and item list
- Makes the missing evidence precise.
- Present, absent and exception states
- Preserves the human-review handoff.
Use Build a public-sector AI evaluation from versioned records to turn the specimen into a brief with a defined task, source and review owner.
Fictional exercise
The right application type determines which intake item is missing
A fictional municipality has an independently authored intake checklist and abstract drawings. It illustrates document completeness, not a permit or compliance decision for any actual property.
| Record ID | Source | What it records |
|---|---|---|
| MI1 | Practice checklist revision C2 | Type A requires form F and diagram D. Type B requires F, D and owner-authority card O. C2 replaces C1 from day 10. |
| MI2 | Mock application X | Type B received day 12; contains F and D, with O absent. |
| MI3 | Comparator and old checklist | Application Y is Type A with F and D. Older C1 omitted O for Type B; no exception authorizes C1 for X. |
Question for the model. Which intake evidence is missing for X under the dated checklist, and why does Y's packet or old C1 not establish X's completeness?
Show the reference answer
MI1 makes C2 applicable after day 10, and MI2 places X on day 12 as Type B. X therefore lacks owner-authority card O under the exercise checklist. Y's F-and-D packet addresses Type A, which has a different requirement, while superseded C1 supplies no exception for X. Identify the missing item and route to a human intake reviewer without approving compliance, issuing a permit or inferring rights to real applicant materials.
How to check the result
Ask for the answer, the record IDs that support it and any fact the evidence leaves unresolved. Have a reviewer compare the response against the reference before deciding whether the sample is ready to expand.
Errors to flag in this case
- Applying Type A completeness requirements to a Type B application.
- Using an obsolete checklist because it yields a complete result.
Change the inputs and run it again
Provide O but omit D in a new Type B case. Require a fresh item-by-item check instead of memorizing O as every application's missing document.
Score the decision and the evidence separately
Save the reviewer’s expected answer, supporting record IDs and unresolved fields in a separate answer key. Score the model on the decision, its citations and its handling of the deliberate gap. A correct answer citing a different case counts as an evidence error.
Before expanding collection, have a second reviewer run the same checks on the held-out packet. Keep related versions, views and exports of the same case together so they stay on one side of the training and evaluation split.
Check permission and scope
Use fabricated intake records and independent diagrams. Exclude addresses, applicants, security-sensitive plans and actual permit decisions.
Use public excerpts or invented addresses. Exclude private tenant information, site access and individual housing eligibility.
Describe an authorized archive through Fiund’s existing-data assessment, or describe a new capture capability through the collection brief. The contributor agreement covers the terms for participation.
Sources and further reading
Resources for this workflow
Includes worksheet
Build a public-sector AI evaluation from versioned records
Build a public or invented versioned record example and distinguish proposal, issue and outcome.
Includes worksheet
Build an AI evaluation from professional work
Write a rubric for the supported administrative decision and an abstention case.
Company and program reviews
Use these reviews to compare sourcing, capture or evaluation approaches. Confirm the applicable program, accepted sources and terms with the provider.
Compare the approach
Surge AI review for frontier data and agent evaluation
Compare how a grader and expert review should handle ambiguous document evidence.
Compare the approach
Sapien review: dataset quality review and expert checks
Read the quality review questions for source scope, reference judgments and unresolved cases.
Related industry examples
Industry guide · NAICS 925110
Housing authority plan adoption and revision data for AI
A housing authority may publish a plan draft and later adopt a version that addresses comments.
Industry guide · NAICS 921140
Combined government ordinance approval and publication data for AI
A local government can test whether software attributes adoption, executive approval and publication to the right roles.
Industry guide · NAICS 923140
Veterans program form and instruction version data for AI
An archived instruction may reference a form revision that differs from the current listing.