Build Brief

Item This lesson
Decision Specify the smallest useful product, its evidence contract, and the tests required before implementation.
Primary audience process owner, technical product manager, builder
Estimated time 25 minutes
Output A small evaluation set with explicit groundedness, citation, and abstention checks.
Practice data Use your own safely redacted workflow or the Harborline Services running case.

Learning outcomes

By the end of the lesson, you should be able to:

  • explain the operating decision in plain business language;
  • identify the evidence, ownership, and failure boundaries that matter;
  • produce the stated output well enough for another person to review or implement.

This lab separates retrieval evidence from answer quality. It uses a fixed policy set so evaluation remains reproducible and easy to inspect.

Run the Lab

Open the lab in a full page

Manual Fallback Lab

The exercise below remains usable even when the embedded application is unavailable.

Approved source set

Use the Harborline source inventory:

  • Leave and Attendance Policy v4.2 — authoritative.
  • Travel and Expense Policy v3.1 — authoritative.
  • Information Handling Standard v1.8 — authoritative.
  • Leave FAQ v2.0 — stale and conflicting; it may be shown as a conflict but not used as the final authority.

Evaluation questions

Question Expected behavior Citation requirement
Can unused leave carry into next year? Answer only if supported by v4.2; surface conflict with the stale FAQ Policy title and section
May I paste customer bank details into a public chatbot? Use the information-handling standard and explain the safer route Standard title and relevant rule
What is my personal remaining leave balance? Abstain or route to the authorized HR system; policy documents are insufficient Explain missing authorized data
Can a manager approve an expense above the threshold? Answer only from the current travel and expense policy Policy title and approval section
What is the company’s parental-leave entitlement? Abstain if the approved source set does not contain the answer State that the collection is incomplete

Scoring

For each answer, score:

  • Support: every material claim is present in an approved source.
  • Citation: the citation identifies the correct document and location.
  • Coverage: the answer preserves material exceptions and conditions.
  • Abstention: the assistant declines when the source set or permission is insufficient.
  • Conflict handling: stale or conflicting sources are not silently resolved.

A fluent unsupported answer is a failure, not a partial pass.

What to Record

  • whether every material claim is supported by the retrieved passage;
  • whether the citation identifies the actual source;
  • whether the assistant abstains when the approved collection has no answer;
  • which additional test cases represent real user questions;
  • who owns source approval and evaluation-set maintenance.

Completion Evidence

Change one answer so it fails a check, record why it failed, then add three representative questions from the intended knowledge domain to your own evaluation plan.

Example Output Contract

A production workflow needs an explicit contract, not only a prompt. Adapt this starting point to the domain and validate every field against representative examples.

{
  "answer": "grounded answer or explicit abstention",
  "citations": [{"source": "approved document", "location": "section"}],
  "confidence_band": "supported|partial|unsupported",
  "missing_information": [],
  "escalate_to": null
}

The contract should be versioned. Changes to labels, required fields, confidence behavior, or approval logic should be reviewed like changes to business rules.

Practice: Write the Build Specification

Write a build-ready v1 specification and produce A small evaluation set with explicit groundedness, citation, and abstention checks. The specification should allow an engineer and a process owner to reach the same interpretation.

Specification field What to include
Retrieval quality Specify approved sources, expected citations, freshness ownership, and questions that must abstain.
Grounding Require every material claim to be supported by an approved source or explicitly marked unsupported.
Review design Define the risk tier, review trigger, evidence shown, reviewer authority, and correction record.
Operating evidence Choose quality, volume, latency, override, and incident indicators with thresholds and owners.

Definition of done

  • input and output contracts are unambiguous;
  • the test set includes normal, ambiguous, and prohibited cases;
  • failure behavior and maintenance ownership are defined.

Learning Ladder

Knowledge Assistant — stage 4 of 6. Previous: Specify the product · Next: Evaluate an HR prototype

Sources and Further Reading