Build Brief
| Item | This lesson |
|---|---|
| Decision | Specify the smallest useful product, its evidence contract, and the tests required before implementation. |
| Primary audience | technical product manager, builder, process owner |
| Estimated time | 30 minutes |
| Output | A classifier blueprint containing the label schema, output contract, test set, thresholds, review path, and refresh policy. |
| Practice data | Use your own safely redacted workflow or the Harborline Services running case. |
Learning outcomes
By the end of the lesson, you should be able to:
- explain the operating decision in plain business language;
- identify the evidence, ownership, and failure boundaries that matter;
- produce the stated output well enough for another person to review or implement.
Classification is one of the most useful places to build AI because it turns messy language into structured decisions that other systems can act on. But the product design challenge is not just choosing a model. It is choosing the right label schema, deciding how certainty should work, and building a refresh cycle so the pipeline stays useful over time.
Product Decision
A classification pipeline often becomes real business infrastructure. It may route emails, tag feedback, assign priority, support CRM updates, or label documents for downstream action. That means the pipeline should be designed like a lightweight product with defined inputs, outputs, review paths, and logs—not just a prompt that returns a category.
Product Model
A useful classifier usually needs:
- a clear schema,
- representative examples,
- confidence or uncertainty handling,
- a routing rule for ambiguous cases,
- and a process for updating labels or prompts as the business changes.
If any of those pieces are weak, the pipeline may still produce labels—but they will not be reliable enough to support operations.
Reference Architecture
A sensible v1 architecture:
- input source connector,
- preprocessing layer,
- label schema and guidance,
- classification engine,
- confidence and threshold logic,
- review queue,
- logging and correction store.
This is enough for many production-like internal workflows.
Interface and Evidence Contract
Inputs
- text message, ticket, note, or document,
- optional source metadata,
- optional prior category or business context.
Outputs
- primary label,
- optional secondary label,
- confidence band,
- short reason,
- routing or action flag.
Review layer
- uncertain cases go to human review,
- new or rare categories are checked,
- reviewers can override labels,
- overrides are stored for later improvement.
Logging
- source record ID,
- label returned,
- confidence band,
- threshold decision,
- reviewer override if any,
- model or prompt version,
- timestamp.
Without logs, the pipeline is hard to improve.
Schema Design
The schema is the foundation. A good label set should be:
- operationally meaningful,
- distinct,
- not too large,
- tied to real downstream action.
A weak schema often has too many labels with fuzzy differences. A stronger schema says:
- each label exists for a reason,
- each label leads to an action,
- edge cases are documented.
For many workflows, fewer clearer labels beat a long taxonomy.
Confidence Thresholds
A practical classifier should not force every item into a confident bucket. Useful threshold logic might be:
- high confidence: label automatically used,
- medium confidence: label suggested, but reviewed in a queue,
- low confidence: hold for manual classification.
Even if the model does not produce a calibrated numeric score, the workflow can still simulate confidence bands using consistency checks, weak-signal detection, or rule-based heuristics.
Operating Workflow
Before the pipeline:
A team manually reads each incoming item, chooses categories inconsistently, and routes work based on experience or guesswork. Reporting quality suffers because labels are incomplete or unstable.
After the pipeline:
The system applies a clear schema, assigns labels, adds a confidence band, and routes ambiguous cases into review. Reviewer corrections are stored and later used to improve label definitions, prompts, or rules. The result is not perfect automation. It is a cleaner decision infrastructure.
Build vs Buy Decision
Build your own when:
- the schema is custom to your business,
- labels drive custom downstream actions,
- edge cases require internal logic,
- off-the-shelf tools do not match the taxonomy.
Buy or use a platform solution when:
- the classification task is generic,
- custom logic is limited,
- the team needs speed more than control,
- maintenance capacity is low.
The important question is whether the value comes from your custom schema or from generic categorization.
V1 vs V2 Scope
Good v1 scope
- one workflow,
- a small clear schema,
- single-label output,
- confidence thresholds,
- review queue,
- logs.
Sensible v2 scope
- multi-label output,
- more metadata-driven routing,
- better dashboards,
- reusable feedback loop,
- stronger automation after high-confidence classification,
- prompt refresh or retraining support.
Do not start with a giant taxonomy unless the workflow truly needs it.
Retraining or Prompt Refresh Cycle
Classification pipelines drift because:
- the business changes,
- new input patterns appear,
- users invent new language,
- priorities shift,
- the schema itself evolves.
A healthy refresh cycle may include:
- monthly review of overrides,
- quarterly schema review,
- prompt or rule updates when misclassification patterns repeat,
- retraining or re-benchmarking only when simpler fixes are no longer enough.
Not every pipeline needs full retraining. Many improve significantly through better schema and better prompts.
Maintenance Burden
Maintenance typically includes:
- schema clarification,
- example refresh,
- prompt changes,
- threshold tuning,
- review-queue cleanup,
- documentation of edge cases.
This is why classification should be treated as living infrastructure.
Build Sequence
- Define a small operational label schema.
- Gather representative examples, including ambiguous ones.
- Build a classifier that returns label, confidence, and reason.
- Add thresholds and review paths instead of pretending full certainty.
- Log overrides and repeated mistakes.
- Review schema and prompts on a regular cycle.
- Expand only when v1 is stable and trusted.
Reference Use Case
A support team wants to classify incoming requests into billing, technical issue, onboarding, cancellation, or partnership. The classifier returns one primary label, a confidence band, and a short reason. High-confidence billing and onboarding cases route automatically, while medium-confidence and unusual cases go to review. After one month, the team sees many overrides between “technical issue” and “onboarding.” Instead of retraining immediately, they first clarify the label definitions and update the prompt. Accuracy improves because the real problem was schema ambiguity, not model weakness.
Engineering Failure Modes
- creating too many fuzzy labels,
- forcing every case into a confident decision,
- tying labels to no real business action,
- skipping override logs,
- overengineering retraining before schema design is mature,
- ignoring drift until users stop trusting the output.
Build Readiness
- Is the label schema small, distinct, and tied to action?
- What happens at each confidence band?
- Are ambiguous cases routed for review?
- Are overrides and prompt versions logged?
- Is there a realistic refresh cycle for schema and thresholds?
Blueprint Acceptance Package
Before implementation begins, the blueprint should contain:
- a versioned label dictionary with positive and negative examples;
- a machine-readable output schema;
- a representative test set including ambiguous and high-risk cases;
- threshold rules for auto-route, human review, and abstention;
- a correction log and ownership model;
- acceptance metrics by class rather than accuracy alone;
- a refresh trigger for new labels, language, or input patterns.
The blueprint is accepted only when an engineer can implement it and an operations owner can explain how exceptions will be handled. The accompanying Classification and Routing Build Lab provides a runnable baseline for testing this package.
Example Output Contract
A production workflow needs an explicit contract, not only a prompt. Adapt this starting point to the domain and validate every field against representative examples.
{
"label": "operational_category",
"priority": "normal|high|urgent",
"confidence": 0.0,
"evidence": ["short source excerpt"],
"route_to": "named_queue",
"human_review_required": true,
"reason": "brief operational justification"
}
The contract should be versioned. Changes to labels, required fields, confidence behavior, or approval logic should be reviewed like changes to business rules.
Practice: Write the Build Specification
Write a build-ready v1 specification and produce A classifier blueprint containing the label schema, output contract, test set, thresholds, review path, and refresh policy. The specification should allow an engineer and a process owner to reach the same interpretation.
| Specification field | What to include |
|---|---|
| Label design | Define mutually useful labels, boundary examples, and an explicit other/uncertain path. |
| Routing decision | Name the destination owner, priority rule, service level, and escalation trigger. |
| Review design | Define the risk tier, review trigger, evidence shown, reviewer authority, and correction record. |
| Operating evidence | Choose quality, volume, latency, override, and incident indicators with thresholds and owners. |
Definition of done
- input and output contracts are unambiguous;
- the test set includes normal, ambiguous, and prohibited cases;
- failure behavior and maintenance ownership are defined.
Learning Ladder
Classification and Routing — stage 5 of 8. Previous: Design case routing · Next: Run the build lab