Evaluation Brief
| Item | This lesson |
|---|---|
| Decision | Judge what the prototype proves, what remains unproven, and whether a governed pilot is justified. |
| Primary audience | founder, general manager, technical product manager, builder |
| Estimated time | 25 minutes |
| Output | An exportable governed-pilot brief with scope, evidence, controls, ownership, economics, and a gate decision. |
| Practice data | Use your own safely redacted workflow or the Harborline Services running case. |
Learning outcomes
By the end of the lesson, you should be able to:
- explain the operating decision in plain business language;
- identify the evidence, ownership, and failure boundaries that matter;
- produce the stated output well enough for another person to review or implement.
This page should be read as the capstone of the demo module. A demo can create interest, but a client only buys a solution when the team can explain the gap between “impressive proof” and “usable system.” That translation is where many AI opportunities are won or lost.
Why This Page Matters
A demo proves that something can be shown. A client solution proves that it can be trusted, operated, governed, and maintained in a real environment. These are not the same thing.
This page exists to help teams answer the right follow-up questions after a demo:
- what did the demo actually prove?
- what did it not prove?
- what would be needed for production?
- which clients should care?
- how should we evaluate the next step responsibly?
That is the commercial bridge between curiosity and delivery.
What a Demo Actually Proves
A good demo may prove:
- the interface is intuitive,
- the workflow pain point is real,
- the output format is attractive,
- the concept resonates with users,
- a narrow technical path seems feasible.
That is already valuable. But it is only a starting point.
What a Demo Does Not Prove
A demo usually does not prove:
- operational reliability,
- governance adequacy,
- security or privacy readiness,
- scale readiness,
- integration readiness,
- support ownership,
- user-adoption durability,
- production economics.
If a client cannot see that distinction, the team should explain it clearly rather than oversell.
Which Client Type Should Care
Not every demo belongs in every client conversation. A good fit usually requires:
- a visible workflow pain point,
- repeated use frequency,
- clear ownership,
- willingness to change the workflow,
- enough data or content maturity,
- enough business value to justify moving beyond the demo.
The right client is not simply the one who thinks the demo looks impressive. It is the one who has a real workflow that the demo pattern can improve.
How to Evaluate a Demo Responsibly
A responsible post-demo evaluation should ask:
- what business problem did this actually illuminate?
- what part of the workflow did it make better or faster?
- where would users still hesitate to trust it?
- what controls are missing?
- what technical or operational gaps still block production?
- what metric would prove business value if we piloted it?
This shifts the conversation from novelty to delivery logic.
Manual Capstone Workspace
Use this structure when the embedded workspace is unavailable. Complete each field before recommending a governed pilot.
1. Pilot decision
| Field | Required answer |
|---|---|
| Workflow problem | Current pain, volume, baseline, and owner |
| Proposed scope | Exact inputs, outputs, users, and exclusions |
| Architecture | Rules, model, retrieval, tools, and integrations |
| Human decision | What remains reviewable, approvable, or prohibited |
| Success measure | Business and quality indicators |
| Stop condition | Failure that pauses or ends the pilot |
2. Evidence plan
| Evidence area | Required record |
|---|---|
| Test set | Normal, ambiguous, high-consequence, and out-of-scope cases |
| Expected behavior | Structured fields, evidence, escalation, and prohibited actions |
| Versioning | Model, prompt, rules, source set, and test-set version |
| Review | Named reviewer and decision authority |
| Economics | Integration, inference, review, maintenance, and change cost |
3. Control plan
| Risk | Preventive control | Detective control | Owner | Containment |
|---|---|---|---|---|
| Unsupported output | ||||
| Sensitive-data exposure | ||||
| Wrong routing or action | ||||
| Stale source | ||||
| Excessive review load |
4. Gate decision
Choose one:
- Pilot: scope, evidence, control, ownership, and economics are sufficient for a bounded trial.
- Revise: the opportunity is plausible, but one or more prerequisites are missing.
- Stop: the expected value does not justify the risk, complexity, or operating burden.
Document the reasons and the person authorized to make the gate decision.
Demo-to-Production Checklist
A useful checklist:
1) Workflow fit
- What recurring job does the demo support?
- Who owns that workflow?
- How often does it occur?
- What pain or cost is being reduced?
2) Data and source readiness
- What inputs does production require?
- Are those inputs governed, accessible, and clean enough?
- Are permissions and sensitivity clear?
3) Output design
- Is the output useful in the client’s real workflow?
- Does it need role-specific formats?
- Does it need citations, source traceability, or structure?
4) Review and governance
- What needs human review?
- What should auto-pass?
- Who escalates sensitive cases?
- What logs and retention rules are required?
5) Integration and delivery
- Does the solution need API or system integration?
- Does it require file handling, user authentication, dashboards, or exports?
- Who maintains it after launch?
6) Pilot scope
- What is the narrowest realistic pilot?
- Which users should be included?
- What success criteria define progress?
7) Production economics
- What is the cost per use or per workflow?
- What support burden should be expected?
- Is the value large enough to justify the next build stage?
This checklist is often more valuable than the demo itself.
Operational Interpretation
Before responsible translation:
A team shows a beautiful demo, the client is impressed, and both sides implicitly assume production is a short step away. Later, the project stalls on permissions, data access, review needs, unclear ownership, or weak workflow fit.
After responsible translation:
The team uses the demo to identify the real workflow value, then makes the hidden production gaps visible: data, review, governance, integration, support, and ROI. The conversation becomes more grounded, and the client can decide whether to pilot, narrow scope, or stop. This leads to better delivery decisions and more credible consulting or product work.
Production Gap Categories
A helpful way to structure the gap:
| Category | Typical hidden question |
|---|---|
| Data | Can the real inputs be accessed and governed? |
| Trust | Why should users believe or verify the output? |
| Review | Who checks the risky cases? |
| Integration | Where does the output go next? |
| Ownership | Who maintains this after launch? |
| Risk | What happens when it fails? |
| Economics | Is the value worth the implementation burden? |
If a team cannot answer these categories, it is still in demo mode.
Module-Wide Demo Evaluation Lens
Across all demos in this module, a responsible team should ask:
- what this demo proves
- what it does not prove
- what would be needed for production
- which client type should care
- how to evaluate it responsibly
These five questions help keep demo conversations honest and commercially useful.
What Would Be Needed for Production
Production readiness often requires:
- clearer scope boundaries,
- source or data governance,
- permissions,
- review and escalation logic,
- logging and monitoring,
- user support and onboarding,
- maintenance and refresh ownership,
- cost and usage planning.
The more sensitive or high-impact the workflow, the more these issues matter.
Common Mistakes
- selling the demo itself rather than the workflow value,
- treating one successful example as proof of durable reliability,
- skipping governance discussion because it feels unexciting,
- not defining the pilot narrowly,
- ignoring support and maintenance ownership,
- failing to identify the real buyer or workflow owner.
Example of Responsible Transition
A document summary demo impresses a client. Instead of promising immediate deployment, the team maps the client’s actual workflow: who uploads documents, which documents are allowed, what summary format is needed, who reviews sensitive outputs, where results must be stored, and how success will be measured. That transforms the conversation from “cool demo” to “credible pilot plan.”
Prototype Review
- What did the demo genuinely prove, and what did it not prove?
- Which client workflow and user group would actually benefit?
- What data, review, governance, and integration gaps remain?
- What is the narrowest responsible pilot scope?
- What metric will show whether the demo deserves production investment?
Capstone Workspace
Complete the six parts of the pilot decision record, choose a gate, and export the result. The workspace saves entries only in your browser; Cognaptus does not receive the data.
Open the capstone workspace in a full page
The capstone is complete only when another person can read the exported brief and identify the owner, scope boundary, evidence threshold, review path, operating cost, and gate decision without relying on the demo presentation.
Representative Test Set
Use the same cases whenever the prototype changes. This prevents a polished new example from replacing evidence about whether the system actually improved.
| Test case | Expected behavior | Observed result |
|---|---|---|
| Representative business case | Define scope, owner, baseline, evidence, and pilot gate. | Record observed result |
| High-consequence failure | Specify containment, approval, logging, and escalation. | Record observed result |
| Economic challenge | Include integration, review, maintenance, and change-management cost. | Record observed result |
Record model or ruleset version, source-set version, test date, reviewer, latency, and any manual edits. An empty or undocumented result is not a pass.
Practice: Assemble the Evidence Pack
Evaluate the prototype with representative and adversarial cases, then produce An exportable governed-pilot brief with scope, evidence, controls, ownership, economics, and a gate decision. Record observations; do not substitute polished screenshots for evidence.
| Evidence field | What to include |
|---|---|
| Business case | Record current pain, baseline volume, owner, measurable outcome, risk, and stop/go gate. |
| Workflow ownership | Map the input, output, owner, handoff, exception queue, service level, and decision right. |
| Review design | Define the risk tier, review trigger, evidence shown, reviewer authority, and correction record. |
| Operating evidence | Choose quality, volume, latency, override, and incident indicators with thresholds and owners. |
| Governance evidence | Identify the accountable owner, policy basis, approval, audit evidence, and review date. |
Definition of done
- expected and observed results are recorded separately;
- at least one failure or boundary case is included;
- the final decision is pilot, revise, or stop with reasons.