Workflow Design Brief

Item This lesson
Decision Define what “good enough” means before a model, prompt, or vendor is selected.
Output A versioned evaluation pack and release gate.
Practice data Use your own safely redacted records or Harborline’s synthetic records.

Workflow Problem

A demo usually answers: “Can the system produce a plausible result?” An evaluation set answers a harder question: “Does it behave well enough across the cases that matter to operate?”

Without a fixed set, teams test whatever example is convenient. Easy cases receive attention, failures disappear between prompt revisions, and improvement becomes a matter of impression.

Evaluation Pack Components

A useful small evaluation pack contains:

  1. Representative inputs from normal workflow volume.
  2. Boundary cases near label, policy, or permission limits.
  3. Adversarial cases designed to trigger unsafe or unsupported behavior.
  4. Expected behavior, including valid abstention and escalation.
  5. Scoring rules tied to business consequences.
  6. Release thresholds and named approvers.
  7. Version information for sources, prompts, models, rules, and test data.

Build the Case Mix

Do not sample only the most common cases. A small set should deliberately include:

Case type Purpose
Routine Measures whether the workflow handles normal volume efficiently
Ambiguous Tests uncertainty, overlap, and missing information
High consequence Tests whether controls override fluency or confidence
Out of scope Tests abstention and routing
Stale or conflicting source Tests knowledge governance
Changed format Tests robustness to realistic input variation

A 20-case set can be more useful than 2,000 unreviewed examples if each case has a clear business reason.

Write Expected Behavior, Not Ideal Prose

Expected behavior should describe the decision contract:

  • required structured fields;
  • facts that must be preserved;
  • claims that require evidence;
  • actions that are prohibited;
  • acceptable uncertainty;
  • correct escalation or abstention.

For generative outputs, there may be several acceptable wordings. Score fidelity, evidence, coverage, and control behavior instead of exact text matching.

Score by Consequence

Not every error has equal cost. A useful rubric separates:

  • critical failures: sensitive-data exposure, unauthorized action, unsupported financial or legal commitment;
  • material failures: wrong routing, missing exception, incorrect field used in a decision;
  • quality failures: awkward wording, minor omission, avoidable reviewer editing.

A system with high average quality can still be unacceptable if it fails one critical control case.

Example Evaluation Record

Field Example
Case ID H-003
Input Vendor requests a bank-detail change before payment run
Expected behavior Create high-risk exception; do not update payment details; route to finance verification
Observed behavior Record after each run
Critical checks No autonomous update; evidence retained; reviewer named
Result Pass / fail / investigate
Reviewer note Why the observed behavior is acceptable or not

Release Gate

Define thresholds before running the final test. For example:

  • zero critical failures;
  • at least 90% correct first-pass routing on representative cases;
  • 100% escalation on defined high-risk cases;
  • citations present for every policy answer;
  • reviewer effort below the agreed pilot limit.

Thresholds should reflect the workflow, not a universal benchmark.

Practice: Build the First Evaluation Pack

Create 15–25 cases for one workflow. Include at least:

  • eight routine cases;
  • four ambiguous or incomplete cases;
  • three high-consequence cases;
  • two out-of-scope cases;
  • one source or policy conflict.

For each case, record expected behavior, critical checks, scoring rule, and business consequence. Freeze version 1 before comparing prompts or models.

Continue Learning