Build Brief

Item This lesson
Decision Build the minimum system needed to compare versions and operate a pilot with evidence.
Output A build specification for evaluation, release gates, monitoring, and rollback.

Product Decision

Teams often store test examples in documents, results in spreadsheets, production metrics in dashboards, and corrections in chat threads. That fragmentation makes it difficult to answer basic questions:

  • Which version passed the release gate?
  • What changed?
  • Which cases regressed?
  • What are reviewers correcting in production?
  • When should the workflow be paused?

A lightweight harness connects those records without requiring a large ML platform.

Reference Architecture

Versioned test cases
        ↓
Evaluation runner ──→ Structured result store ──→ Comparison report
        ↓                         ↓
Release gate                 Reviewer evidence
        ↓                         ↓
Pilot deployment ──→ Production indicators ──→ Alert / rollback / investigation

Core Data Contracts

Test case

{
  "case_id": "H-003",
  "input_reference": "synthetic-record",
  "expected_behavior": "escalate bank-detail change",
  "critical_checks": ["no autonomous update", "finance review required"],
  "risk_tier": "high",
  "test_set_version": "1.0"
}

Evaluation result

{
  "case_id": "H-003",
  "system_version": "prompt-7_model-A_rules-3",
  "observed_output": {},
  "check_results": [],
  "reviewer_decision": "pass|fail|investigate",
  "reviewer_note": "",
  "run_timestamp": ""
}

Production correction

{
  "workflow_id": "record-reference",
  "system_version": "",
  "original_output": {},
  "review_action": "approve|edit|reject|escalate",
  "correction_category": "",
  "business_impact": "",
  "follow_up_owner": ""
}

Release Gates

The harness should support rules such as:

  • zero critical-control failures;
  • minimum task-specific quality threshold;
  • maximum manual-review rate;
  • no regression on named boundary cases;
  • explicit approval by product and risk owners;
  • deployment record linked to the evaluated version.

A release gate is not only a dashboard. It must be able to block or clearly prevent an unapproved version from becoming the operational default.

Production Monitoring

Track indicators that connect to the workflow:

Indicator Example interpretation
Volume and case mix The system is seeing a different population than the test set
Confidence or abstention Uncertainty has shifted or sources are missing
Acceptance and correction Reviewer behavior reveals quality or usability problems
Escalation and override High-risk or unexpected cases are increasing
Latency and failure rate The technical path is degrading
Business outcome Routing, review time, backlog, or error cost is changing

Rollback and Investigation

Define:

  • who may pause the workflow;
  • which version to restore;
  • how pending records are handled;
  • which logs and source versions are preserved;
  • how affected users are notified;
  • what test must pass before resuming.

V1 Scope

A credible v1 can be a small application or repository that:

  1. stores a versioned JSON or CSV test set;
  2. executes one system version against every case;
  3. records structured outputs and rule checks;
  4. supports reviewer pass/fail decisions;
  5. compares runs and highlights regressions;
  6. exports the release record;
  7. ingests production corrections for periodic review.

Practice: Write the Build Specification

Specify:

  • repository or service boundary;
  • test-case and result schemas;
  • version identifiers;
  • automated checks and human-review fields;
  • release thresholds;
  • monitored production indicators;
  • alert and rollback triggers;
  • access roles and retention;
  • one reproducible end-to-end test.

The specification is complete when a new team member can run the same test set, identify the deployed version, and explain why it was approved.

Continue Learning