Evaluation Brief
| Item | This lesson |
|---|---|
| Decision | Judge what the prototype proves, what remains unproven, and whether a governed pilot is justified. |
| Primary audience | finance professional, technical product manager |
| Estimated time | 25 minutes |
| Output | A prototype evidence report covering field accuracy, source evidence, exceptions, review load, and production gaps. |
| Practice data | Use your own safely redacted workflow or the Harborline Services running case. |
Learning outcomes
By the end of the lesson, you should be able to:
- explain the operating decision in plain business language;
- identify the evidence, ownership, and failure boundaries that matter;
- produce the stated output well enough for another person to review or implement.
Extraction demos are often high-conversion because the workflow value is immediately visible: turn messy documents into structured fields, reduce manual entry, and surface exceptions faster. But a strong demo should still be framed carefully. It proves that AI can help structure messy inputs. It does not prove that full automation is already safe.
Why This Demo Exists
This demo exists to show that:
- documents often contain repeated field patterns,
- AI can help identify and normalize those fields,
- structured outputs can feed downstream workflows,
- and humans can focus more on uncertain or exceptional cases.
That is compelling to clients because it connects directly to real operational pain.
What This Demo Proves
A responsible extraction demo can prove that:
- AI can extract useful fields from varied documents,
- a structured output can be shown clearly in the interface,
- the workflow can distinguish between confident and uncertain fields,
- manual data-entry burden can plausibly be reduced.
If done well, it also proves that an otherwise messy document workflow can be made more systematic.
What This Demo Does Not Prove
It does not prove that:
- extraction is accurate enough for all production cases,
- validation rules are complete,
- downstream posting or routing is safe,
- every document template will behave the same way,
- exception handling is production-ready,
- the system can skip review for sensitive or high-value cases.
These are exactly the questions that must be addressed before full deployment.
Which Client Type Should Care
This demo is especially relevant for:
- finance and accounts-payable teams,
- operations groups dealing with forms or inbound documents,
- shared-services teams,
- procurement or vendor-onboarding workflows,
- clients with repetitive data-entry or document-capture pain.
It is less relevant where documents are rare, highly bespoke, or not central to workflow volume.
How to Evaluate It Responsibly
A responsible evaluation should ask:
- are the right fields extracted for the actual workflow?
- what happens when a field is missing or unclear?
- are validation rules visible?
- does the demo separate easy fields from risky fields?
- would the output really reduce work in a live process?
This matters more than whether the demo produces a nice-looking JSON or table.
Evaluation Criteria
| Criterion | What to check |
|---|---|
| Field usefulness | Are the extracted fields actually the ones the client needs? |
| Clarity of uncertainty | Does the demo show which fields are uncertain or invalid? |
| Validation fit | Are there obvious rule checks or quality controls? |
| Exception visibility | Can users see what would need review? |
| Workflow value | Would this actually reduce operational effort in the real process? |
What Would Be Needed for Production
A production-grade extraction workflow would usually need:
- clearly defined schema,
- deterministic validation rules,
- confidence thresholds,
- exception queues,
- review UI for uncertain fields,
- template variability handling,
- export into downstream systems,
- audit logging,
- maintenance for changing document formats.
Without those, the system remains a promising proof rather than a dependable workflow tool.
Operational Interpretation
Before the demo:
A user sees messy PDF invoices or forms and imagines hours of manual reading, copying, and correction.
After the demo:
The user sees how structured extraction could reduce manual capture, surface missing fields, and feed the next workflow step. But a responsible buyer also sees that production requires schema discipline, validation, exception review, and downstream integration.
Evaluation Traps
- showing only clean, easy documents,
- hiding uncertain or missing fields,
- implying that extraction equals approval,
- failing to connect extracted fields to a real business action,
- ignoring validation and exception design.
Responsible Positioning
A strong way to describe this demo:
This is a controlled proof that AI can extract useful fields from messy documents and reduce manual capture effort. It is not yet a full automation workflow, because production use would require schema definition, validation, confidence thresholds, exception handling, and downstream integration.
Prototype Review
- Which exact fields and downstream actions does the demo support?
- Does the demo show uncertainty rather than fake confidence?
- What validation or rule logic would production require?
- Which clients actually have high-volume extraction pain?
- Is the demo being positioned as structured intake support rather than full automation?
Representative Test Set
Use the same cases whenever the prototype changes. This prevents a polished new example from replacing evidence about whether the system actually improved.
| Test case | Expected behavior | Observed result |
|---|---|---|
| Clean invoice | Extract required fields with source locations and deterministic validation. | Record observed result |
| Duplicate or near-duplicate invoice | Create an exception rather than treating extraction as payment approval. | Record observed result |
| Unknown vendor or missing purchase order | Route to review and preserve the control failure. | Record observed result |
Record model or ruleset version, source-set version, test date, reviewer, latency, and any manual edits. An empty or undocumented result is not a pass.
Practice: Assemble the Evidence Pack
Evaluate the prototype with representative and adversarial cases, then produce A prototype evidence report covering field accuracy, source evidence, exceptions, review load, and production gaps. Record observations; do not substitute polished screenshots for evidence.
| Evidence field | What to include |
|---|---|
| Field evidence | For every extracted field, retain the source location, validation rule, and exception behavior. |
| Label design | Define mutually useful labels, boundary examples, and an explicit other/uncertain path. |
| Review design | Define the risk tier, review trigger, evidence shown, reviewer authority, and correction record. |
Definition of done
- expected and observed results are recorded separately;
- at least one failure or boundary case is included;
- the final decision is pilot, revise, or stop with reasons.
Learning Ladder
Document Processing — stage 5 of 6. Previous: Design invoice controls · Next: Design the review console