INDUSTRY FIELD GUIDE
Custom software issue, patch and regression test data for AI
A software firm can demonstrate whether a patch fixes the reported behavior while preserving an unrelated case.
NAICS 541511 · Custom Computer Programming Services
A software firm can demonstrate whether a patch fixes the reported behavior while preserving an unrelated case. OpenAI’s SWE-bench Verified work describes expert screening of software tasks. Its later analysis documents benchmark contamination and test problems, reasons to keep original evaluation cases separate from training material. SWE-bench Verified methods · OpenAI coding benchmark analysis
Author a tiny repository with an input parsing bug. Package the issue, reproducible baseline and expected behavior. Keep a hidden test for a boundary input and a test that preserves the normal path. Ask a reviewer to compare actual outcomes and record the code revision and environment.
Use code your firm owns and clear dependencies, client terms and any contributor rights. Exclude production repositories, issue history, secrets and private logs.
Describe an original software regression example
Original Fiund exercise
A patch can fix a parsing example while breaking the ordinary path
An independently authored local task defines parse_count as reading a nonnegative decimal integer with optional surrounding spaces. The packet describes dummy snapshots and observable outputs, with no production repository or private issue history.
| Record | Source | What it records |
|---|---|---|
| SR1 | Original task fixture | Input ' 3 ' should return 3 but baseline revision R1 returns an error. Input '12' correctly returns 12. |
| SR2 | Candidate revision R2 outputs | ' 3 ' returns 3; '12' returns 1; boundary input '0' returns 0. Environment fixture E unchanged. |
| SR3 | Acceptance rule | Whitespace handling and full multi-digit parsing both required. Reference patch and additional boundary inputs stored separately from model inputs. |
Question for the model
Which reported behavior improves in R2, which regression remains, and can passing the original reproduction establish acceptance?
Show the reference answer and checks
SR2 fixes the supplied whitespace example by returning 3 for ' 3 ', but regresses the ordinary multi-digit path by returning 1 instead of 12 for '12'. Its zero result does not resolve that failure. SR3 requires both behaviors, so the original reproduction passing is insufficient for acceptance. Preserve code revision, environment and each expected/observed output, while keeping reference fixes and hidden cases outside the input.
- Accepting a patch after the reported failing example passes without checking an existing normal path.
- Leaking the reference patch into evaluation inputs or attributing a regression to a changed environment absent evidence.
Fields to preserve
- Revision and environment fixture
- Makes output attribution reproducible.
- Input and expected/observed result
- Defines the exact repaired or regressed behavior.
- Hidden-test boundary
- Separates evaluation evidence from training leakage.
Test another case
Use separately held-out multi-digit values and whitespace variants, with an independent reviewer checking that legitimate fixes pass and shortcuts fail. These described fixtures are a task-design example, not a claim that a repository was executed here.
Use owned original code and cleared dependencies for any implementation. Exclude production repositories, secrets, private logs and client issue histories.
Discuss a contribution
Describe the sources you control, the task they could support and what has already been checked. Use the existing-data assessment for an archive or the collection brief for a new recording project. The contributor agreement defines Fiund’s licensing authority and compensation. A guide or fictional exercise does not establish an accepted collection or a sale.