INDUSTRY FIELD GUIDE
Software issue histories for testing AI coding agents
An issue thread is useful when it can be tied to a reproducible starting state and a checkable fix.
NAICS 513210 · Software Publishers
An issue thread is useful when it can be tied to a reproducible starting state and a checkable fix. SWE-bench demonstrated an evaluation built around real repository issues. Later evaluations have identified flaws and contamination in public benchmark cases. A fresh, authorized set of software tasks can be useful if the acceptance test actually proves the repair. SWE-bench research · OpenAI on SWE-bench Verified limitations
Package a task as a reproducible case
Provide a problem statement, isolated code snapshot, steps to reproduce, dependency lockfile and expected behavior. Keep the reference fix separate. Include tests that fail before the fix and pass after it, along with regression tests for related behavior. Have another maintainer verify that tests accept legitimate fixes and reject shortcuts.
Use task families or products for the train and evaluation split. If adjacent tickets expose the fix, keep them together. Mark cases that require unavailable services or credentials and replace them with a safe local fixture.
Clear the rights to the source and discussion
Code ownership can differ from issue comments, customer logs and third-party dependencies. Strip credentials, deployment URLs and real customer traces. Check licenses, contractor assignments and client terms. Start with a self-contained example a team owns or an approved open-source case under its actual terms.
Describe the testable workflow before exporting a repository. State how a buyer can reproduce the issue, prove a correct fix and review source authority. Describe a dataset
Original Fiund exercise
An issue's first passing test does not prove a general repair
An original local task defines sum_sizes as the sum of nonnegative integer list entries. Dummy snapshots, fixtures and candidate results expose a shortcut without relying on a public benchmark or customer repository.
| Record | Source | What it records |
|---|---|---|
| SO1 | Reproducible starting fixture | sum_sizes([2,3]) returns 2 but expected 5. Empty input returns expected 0. Snapshot S1 and fixed dependency list supplied. |
| SO2 | Candidate output log | Candidate C returns 5 for [2,3], 0 for [], and 5 for [0,4]. Last expected output is 4. |
| SO3 | Independent acceptance key | Task requires all valid lists, not just the first fixture. Candidate patch and reference solution kept separate from model input. |
Question for the model
Which regression does candidate C fix in the supplied checks, and what evidence prevents calling the issue fully repaired?
Show the reference answer and checks
SO2 shows C passing the original [2,3] reproduction and retaining the empty-list result. It fails the independent [0,4] check by returning 5 instead of 4. Under SO3's stated task, that is evidence against a general repair despite the first test passing. Report each input, expected value and observed value; do not label the issue resolved or infer a correct implementation from one successful case.
- Accepting a hard-coded shortcut after only the original reproduction passes.
- Leaking the reference solution or adjacent fix discussion into the evaluation input.
Fields to preserve
- Snapshot and dependency fixture
- Makes the starting failure reproducible.
- Input, expected and observed output
- Shows exactly which behavior a check proves.
- Held-out acceptance case
- Rejects a shortcut that fits only the first example.
- Task-family split
- Keeps closely related fix-revealing cases together.
Test another case
Use a separately authored list family with three or more entries and varied zeros. Have an independent maintainer verify legitimate implementations and reject constant-output shortcuts before accepting the task pack.
Use original self-contained code and synthetic fixtures. Exclude credentials, deployment URLs, customer traces, protected issue comments and unlicensed dependencies.
Discuss a contribution
Describe the sources you control, the task they could support and what has already been checked. Use the existing-data assessment for an archive or the collection brief for a new recording project. The contributor agreement defines Fiund’s licensing authority and compensation. A guide or fictional exercise does not establish an accepted collection or a sale.