PRACTICAL RESOURCE
Build an AI evaluation from professional work
An expert can contribute a valuable evaluation by writing a realistic task, specifying the allowed evidence and showing how to grade an answer.
An expert can contribute a valuable evaluation by writing a realistic task, specifying the allowed evidence and showing how to grade an answer. A company can also license approved operating records. Those two routes call for different materials and permissions.
Evidence for professional workflow data
| Public evidence | What it establishes | What to ask |
|---|---|---|
| Scale Labs' PRBench includes 1,100 expert-authored law and finance tasks and 19,356 grading criteria. | Public research that evaluates models against professional judgment and multiple criteria. | Which tasks need fresh authorship, an independent reviewer, and a held-out test? |
| OpenAI and Ironclad developed 11 tasks across legal, commercial and procurement workflows, with 8 to 50 evaluation criteria per task. | A named research collaboration using a hosted practice environment and synthetic training tasks drawn from representative workflows. | Can the workflow run in an isolated environment? What final state and boundary cases should the evaluator inspect? |
| Scale's Data Partnerships program lists law firms, scientific research and business services, and asks for real operating workflows. | Public intake for authorized operating data. | Which data, partners, scope and approvals does the current brief require? |
| Troveo offers a commercial licensing process for scoped, anonymized company exports. | A public supplier program that describes a manifest, anonymization and negotiated terms. | What data is transferred, who checks the export and what do sale, reuse, downstream access and payout terms say? |
Scale Labs PRBench · OpenAI and Ironclad · Scale AI Data Partnerships · Troveo business data
Work through one original example
Write a fictional supplier onboarding policy. Rule V2 says a certificate must be valid on the activation date. For supplier ORCHID, the review occurred June 3, activation was scheduled June 10, and the provided certificate expires June 7. Another record says that a replacement was requested without providing it.
Ask for a short status, the controlling date, a source citation and the missing artifact. The expected answer keeps activation pending because the document does not cover June 10 and no replacement has arrived. Score the status, governing rule, evidence dates and unresolved artifact independently. For a held-out case, change the rule and the record order, then use an expert to verify the new key. Keep the grader and held-out cases separate from the training inputs.
Clear the material
Use independently written prompts, screenshots and data. Confirm authorship, licensing and ownership of each reference. Professional confidentiality, client contracts, trade secrets and privileges require review before sharing real records. Removing names is one step. Confirm what authorization allows the remaining content to leave its original context.
Try the law firm contract exercise, engineering verification exercise and software regression exercise. Download the Professional Workflow Evaluation Worksheet · Describe a professional workflow to Fiund
Make the exception and the revised rule independently testable
Extend the ORCHID exercise with policy V3, which uses a different document-validity date, and a second supplier whose replacement certificate is actually supplied. Write a new reference key before asking the model. Changing only names would not test the date rule that makes the original case distinctive.
Keep the applicable policy, allowed attachments, task prompt, reference answer and grading criteria as separate artifacts. Score an unsupported conclusion even when its prose sounds professional. An expert-authored evaluation can be original work; it does not require importing a client’s contract or confidential operating history. A real archive proposal needs its own source and permitted-use review.