Decision Brief

Item This lesson
Decision Choose the least complex approach that can meet the quality, freshness, control, and integration requirements.
Output A method-choice record with evidence and rejected alternatives.
Practice data Use one real workflow or the Harborline Services case.

The Business Question

Teams often jump from a problem directly to a fashionable technique. A writing problem becomes “we need fine-tuning.” A policy question becomes “we need an agent.” A routing problem becomes “we need RAG.” These choices are frequently backwards.

Start with the required behavior:

  • Does the system only need clearer instructions?
  • Must it use changing company knowledge?
  • Is the task a stable prediction over repeated examples?
  • Does the process need fixed integrations and approvals?
  • Must the system choose among tools or steps dynamically?

The answer determines the architecture more reliably than the vocabulary used in a product demo.

Decision Matrix

Approach Use it when Evidence required Main limitation
Prompting The model already has the capability and the problem is instruction, format, or context Representative prompts and a fixed evaluation set Does not provide private knowledge or durable behavior by itself
RAG Answers must use current or proprietary sources Retrieval tests, citations, permission checks, abstention cases Retrieval quality and source governance become product responsibilities
Fine-tuning A repeated behavior, style, or narrow prediction cannot be achieved reliably through instructions and examples High-quality training data, held-out evaluation, maintenance plan Does not automatically make facts current or solve workflow integration
Deterministic workflow The sequence, rules, approvals, and system actions are known Process map, rule tests, exception paths, audit evidence Less flexible when the task genuinely requires open-ended choice
Agent-like system The system must choose among tools or steps under uncertainty Action tests, budgets, stop conditions, permission controls, incident plan Autonomy expands the failure surface and operating burden

A Practical Selection Sequence

  1. Try the simplest deterministic solution first. Some tasks are rules, search, templates, or calculation problems.
  2. Use prompting when the missing ingredient is instruction. Add examples and an output schema before changing the model.
  3. Add retrieval when approved knowledge must be current and traceable. Do not use fine-tuning as a document database.
  4. Use a workflow when actions and approvals are known. A model can fill bounded steps without controlling the whole process.
  5. Consider fine-tuning only after measuring a persistent capability gap. The training data and evaluation burden must be justified.
  6. Use agent-like behavior only for genuinely open-ended choices. Bound tools, permissions, cost, time, and stop conditions.

Worked Example: Employee Policy Questions

Harborline wants an assistant for leave questions.

  • Better prompting alone is insufficient because the model does not possess the current policy.
  • Fine-tuning is a poor first choice because policies change and answers need citations.
  • A RAG pattern fits the knowledge requirement.
  • A deterministic workflow handles authentication, permission checks, citation display, and escalation.
  • An open-ended agent is unnecessary because the allowed actions are narrow.

The resulting design is RAG inside a controlled workflow, not “an HR agent.”

Boundary Tests

Ask these questions before approving the method:

  • What must change when the source data changes?
  • Which claims require a citation?
  • Can a rule handle the decision more safely?
  • What side effects can the system create?
  • What happens when confidence is low or tools fail?
  • How will the team know whether the added complexity improved the outcome?

Practice: Write the Method-Choice Record

For one use case, document:

Field Required answer
Required behavior What must the system do, and what is explicitly out of scope?
Freshness Does the answer depend on current private knowledge?
Repetition Is there enough stable, labeled behavior to justify training?
Workflow Which steps, rules, integrations, and approvals are already known?
Autonomy Which choices cannot be specified in advance?
Evidence What fixed test will prove the selected method is better?
Rejected alternatives Why are simpler or more complex options not justified?

Finish with one sentence: Use ___ because ___; do not use ___ until ___ is proven.

Continue Learning