INDUSTRY FIELD GUIDE
Social science codebooks and annotation revision data for AI
A theme code can change meaning between annotation rounds.
NAICS 541720 · Research and Development in the Social Sciences and Humanities
A theme code can change meaning between annotation rounds. AAPOR disclosure standards emphasize documenting collection and analysis methods, and HHS publishes informed-consent resources for covered research. A social scientist can use original fictional text to test annotation provenance. AAPOR disclosure standards · HHS informed consent guidance
Write two short fictional excerpts about commuting options. Split one theme into two in a second codebook, and record an adjudicator’s reason for assigning a label to only one excerpt. Ask the model to cite the version and mark the ambiguous excerpt unresolved. Score both the label and reasoning.
Exclude real interviews, identifiable responses and participant records. Use original writing and keep consent for actual research and later AI reuse under separate review.
Describe an original codebook evaluation sample
Original Fiund exercise
A split theme requires the codebook version and an adjudication trail
Two independently written sentences describe imaginary commuting preferences. They are authored examples, not human-subject responses or a representation of an actual study.
| Record | Source | What it records |
|---|---|---|
| SC1 | Codebook versions | Version 1 code ACCESS covers both fare and route availability. Version 2 replaces it with COST and ROUTE; ambiguous text may remain UNRESOLVED. |
| SC2 | Fictional excerpts | Excerpt A: 'The fare exceeds my daily budget.' Excerpt B: 'Getting there is difficult.' Both received ACCESS in round 1. |
| SC3 | Round 2 adjudication | A assigned COST under v2 because it explicitly mentions fare. B has no completed adjudication and no stated reason for its difficulty. |
Question for the model
Which v2 label is supported for A, and can B's old ACCESS label be automatically converted to COST or ROUTE?
Show the reference answer and checks
SC3 supports COST for A under version 2, with the explicit fare wording in SC2 as the reason. B's version-1 ACCESS annotation spans both new themes and its sentence does not disambiguate them. Preserve B's historical ACCESS label with version 1 and mark its v2 assignment unresolved. Neither inheritance from the old label nor assumptions about commuting difficulty supply the missing adjudication.
- Displaying ACCESS as if it retained the same definition after the version-2 split.
- Guessing an ambiguous excerpt's cause from a general commuting stereotype.
Fields to preserve
- Excerpt and annotation round
- Preserves the text linked to each decision.
- Codebook version and definition
- Makes the label interpretable.
- Adjudicator reason and unresolved state
- Distinguishes reviewed assignments from inferred ones.
Test another case
Change B to 'There is no bus route to that stop' and supply a new v2 adjudication. Require an updated ROUTE label with the new evidence, without rewriting round-1 history.
Use original fictional excerpts. Real participant data, consent scope and later AI reuse require separate authorization and review.
Discuss a contribution
Describe the sources you control, the task they could support and what has already been checked. Use the existing-data assessment for an archive or the collection brief for a new recording project. The contributor agreement defines Fiund’s licensing authority and compensation. A guide or fictional exercise does not establish an accepted collection or a sale.