Skip to content

Industry field guide

Municipal plan-review examples for evaluating AI document intake

A plan intake assistant can be evaluated on whether it detects a missing required item and points the applicant to the correct checklist.

2022 NAICS
925120
Updated

What this guide tests

Which intake evidence is missing for X under the dated checklist, and why does Y's packet or old C1 not establish X's completeness?

See the specimen and fields · Work through the fictional records

Task and source context

A plan intake assistant can be evaluated on whether it detects a missing required item and points the applicant to the correct checklist. Decisions about compliance or permits remain with authorized officials. Phoenix and Tempe publish plan-review steps that illustrate the kinds of documented requirements a fictional evaluation can use. These public processes establish a practical task. They do not establish a city’s intent to license applicant files. Phoenix construction plan review · Tempe plan review

Build a versioned intake case

Create a fictional application, simplified drawing and municipal checklist reference. Label which document is missing, cite the dated requirement and specify the point at which a human reviewer must take over. Include a case where the same material is acceptable under one application type and insufficient under another. Keep the official current rule separate from your fictional application.

Confirm reuse rights even when material is accessible

Public access to a record and permission for commercial AI use are different questions. Federal government works have a specific rule under U.S. copyright law. Local government records and materials supplied by private applicants or contractors require their own review. Keep exact property addresses, security-sensitive plans and applicant information out of the example. 17 U.S.C. §105

Share the task and a fabricated intake packet. Specify the jurisdiction, checklist version, required evidence and human review boundary. Describe a dataset

Industry research

The U.S. Census Bureau’s industry definitions establish which activities these codes cover. The workflow sources in this guide describe the task or record type. Fiund’s pilot below uses fictional material to test a specific decision within that scope.

925120 · Administration of Urban Planning and Community and Rural Development

This industry comprises government establishments primarily engaged in the administration and planning of the development of urban and rural areas. Included in this industry are government zoning boards and commissions.

U.S. Census Bureau, 2022 NAICS Manual, p. 597 ↗

Build a pilot sample

Set up this pilot

Write a fictional dated intake checklist and two application packets. Move one item between revisions, then grade missing evidence under the checklist applicable to the received date.

Prepare the test packets

  1. Start with the worked case. Use the fictional record types below as a guide to the source files you need. Give each packet a stable ID and save its source versions with it.
  2. Change one fact. Make one packet with enough evidence to answer the question and one where the evidence points to a different answer. Keep the other fields fixed so a reviewer can tell what changed.
  3. Remove the decisive evidence. Give the reviewer a third packet with a plausible near-match in its place. Have them write what can still be established and what evidence would resolve the rest.

Fields the reviewer needs

Application type and received date
Selects the relevant practice requirements.
Checklist revision and item list
Makes the missing evidence precise.
Present, absent and exception states
Preserves the human-review handoff.

Use Build a public-sector AI evaluation from versioned records to turn the specimen into a brief with a defined task, source and review owner.

Fictional exercise

The right application type determines which intake item is missing

A fictional municipality has an independently authored intake checklist and abstract drawings. It illustrates document completeness, not a permit or compliance decision for any actual property.

Records supplied to the evaluator
Record IDSourceWhat it records
MI1Practice checklist revision C2Type A requires form F and diagram D. Type B requires F, D and owner-authority card O. C2 replaces C1 from day 10.
MI2Mock application XType B received day 12; contains F and D, with O absent.
MI3Comparator and old checklistApplication Y is Type A with F and D. Older C1 omitted O for Type B; no exception authorizes C1 for X.

Question for the model. Which intake evidence is missing for X under the dated checklist, and why does Y's packet or old C1 not establish X's completeness?

Show the reference answer

MI1 makes C2 applicable after day 10, and MI2 places X on day 12 as Type B. X therefore lacks owner-authority card O under the exercise checklist. Y's F-and-D packet addresses Type A, which has a different requirement, while superseded C1 supplies no exception for X. Identify the missing item and route to a human intake reviewer without approving compliance, issuing a permit or inferring rights to real applicant materials.

Download the fictional records and answer key (JSON)

How to check the result

Ask for the answer, the record IDs that support it and any fact the evidence leaves unresolved. Have a reviewer compare the response against the reference before deciding whether the sample is ready to expand.

Errors to flag in this case

  • Applying Type A completeness requirements to a Type B application.
  • Using an obsolete checklist because it yields a complete result.

Change the inputs and run it again

Provide O but omit D in a new Type B case. Require a fresh item-by-item check instead of memorizing O as every application's missing document.

Score the decision and the evidence separately

Save the reviewer’s expected answer, supporting record IDs and unresolved fields in a separate answer key. Score the model on the decision, its citations and its handling of the deliberate gap. A correct answer citing a different case counts as an evidence error.

Before expanding collection, have a second reviewer run the same checks on the held-out packet. Keep related versions, views and exports of the same case together so they stay on one side of the training and evaluation split.

Check permission and scope

Use fabricated intake records and independent diagrams. Exclude addresses, applicants, security-sensitive plans and actual permit decisions.

Use public excerpts or invented addresses. Exclude private tenant information, site access and individual housing eligibility.

Describe an authorized archive through Fiund’s existing-data assessment, or describe a new capture capability through the collection brief. The contributor agreement covers the terms for participation.

Sources and further reading

Resources for this workflow

How the examples work

Company and program reviews

Use these reviews to compare sourcing, capture or evaluation approaches. Confirm the applicable program, accepted sources and terms with the provider.

All company reviews

Related industry examples

← Public administration