Before the Solver: Clarification Needs Its Own Readiness Gate

TL;DR for operators

A business user can ask an optimization copilot for a schedule, allocation, or planning model while leaving objectives, constraints, or policy boundaries partly unstated. Two plausible interpretations can then produce different mathematical formulations even when both look coherent. The risk is not bad algebra. It is premature formulation.

OR-Clarify, introduced in Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization,¹ evaluates whether an agent identifies formulation-critical missing information before modeling, recovers it through questioning, avoids filling gaps with unconfirmed defaults, and stops at an appropriate point.

For missing requirements that are blocking or materially change the formulation, the benchmark uses Core Exact: a run succeeds only if all such requirements are recovered exactly. Under the structured Choice protocol, InterOPT raises Core Exact from 0.506 for MC-D to 0.675, but average atomic clarification questions rise from 2.350 to 10.036.

That gain is not universal. Under open-ended clarification, ORPilot reaches 0.583 Core Exact versus 0.538 for InterOPT. InterOPT also declares itself ready in 98.8% of open-protocol runs, yet 40.6% are classified as premature.

For operators, clarification is therefore a decision-control stage before model generation: detecting unresolved requirements, resolving them, and deciding when enough has been resolved to formulate are separate capabilities.

A plausible model can still encode the wrong business

Suppose a user asks an optimization copilot to build a staffing schedule. The request gives demand estimates and available workers but says nothing about whether overtime is prohibited, merely expensive, or acceptable only for certain roles.

All three interpretations can produce a valid mathematical program. They do not produce the same program.

That distinction is the paper’s starting point. A specification is not sufficiently complete merely because an LLM can invent a reasonable continuation. It is complete only when plausible business completions no longer change the structure of the optimization problem: its objectives, constraints, decision variables, or related modeling choices.

This is where a silent assumption becomes operationally dangerous. An agent may choose a plausible overtime convention without asking. The resulting model can look internally consistent while embedding an unconfirmed business rule.

Clarification is therefore not just conversational polish. It determines whether the agent has enough authority to translate natural language into mathematical commitments.

OR-Clarify measures what happens before formulation

OR-Clarify contains 100 cases with 178 hidden formulation-critical requirements. Each tested agent receives an incomplete public business brief, while the missing facts remain private to the benchmark and simulated user.

The evaluation asks more than whether the agent eventually mentions something relevant. It measures exact recovery, silent assumptions, stopping behavior, turns, and atomic questions.

Some missing requirements are classified as P0 or P1 because they are blocking or materially affect the formulation. Core Exact counts a run as successful only when every such requirement in the case is recovered exactly.

That strictness matters. A clarification agent that recovers three of four important rules can still formulate against the wrong fourth rule. Aggregate conversational quality would hide that failure.

The benchmark accordingly changes the evaluation target. The relevant question is no longer whether the agent can formulate a model from the text it received. It is whether the agent recognizes when the text it received is not yet safe to formulate.

Persistent gap tracking improves coverage, but the bill arrives in questions

InterOPT addresses the problem in two stages.

Dynamic Gap Search repeatedly identifies unresolved formulation conditions and keeps them in a persistent ledger across turns. Gap-Guided Action Search then uses that ledger to generate targeted questions and decide whether to ask again or return READY_TO_MODEL.

The mechanism is straightforward: unresolved uncertainty should not disappear merely because the conversation has moved on.

Under the Choice protocol, that structure materially improves recovery. Core Exact rises from 0.506 for MC-D to 0.675 for InterOPT. All-Slot Exact rises from 0.474 to 0.638. Silent assumptions fall from 0.692 to 0.366 per run.

But InterOPT averages 10.036 atomic questions and 10.674 turns, compared with 2.350 questions and 3.258 turns for MC-D.

This is not a small efficiency penalty attached to a cleaner result. It changes the product decision. A system designer is choosing a point on a coverage-versus-interaction-cost curve.

The ablations help explain why. In the Choice setting, removing Dynamic Gap Search reduces Core Exact from 0.675 to 0.564; removing Gap-Guided Action Search reduces it to 0.615. Persistent gap discovery appears particularly important for coverage within this benchmark.

The paper does not show that asking more questions indefinitely is the solution. It shows that recovering more hidden requirements can require substantially more interaction when the system is explicitly engineered to search for them.

Structured clarification is part of the architecture

InterOPT’s strongest advantage appears under structured Choice interaction. That advantage weakens in the open setting.

With free-form clarification, InterOPT reaches 0.538 Core Exact and 0.492 All-Slot Exact. The adapted ORPilot baseline reaches 0.583 and 0.546 respectively, while also producing fewer silent assumptions: 0.268 per run versus InterOPT’s 0.560.

So the evidence does not support a general claim that InterOPT is the superior clarification policy.

It supports a narrower and more operationally consequential conclusion: the interface through which requirements are elicited can materially affect recovery performance.

For product teams, a multiple-choice or otherwise structured clarification layer should not be treated as presentation wrapped around the same underlying agent. When candidate business policies can be represented explicitly, the interaction protocol becomes part of the reasoning architecture.

The remaining question is which domains tolerate that structure. Real stakeholders may reject ten rounds of constrained clarification, and many business rules will not fit neatly into candidate alternatives.

A gap ledger does not tell the agent when it is safe to stop

The most consequential diagnostic comes after the agent has already been given explicit machinery for tracking uncertainty.

In the open setting, InterOPT declares readiness in 98.8% of runs. Yet 203 of 500 runs—40.6%—are classified as premature.

That result corrects an appealing assumption: once an agent has an explicit list of unresolved gaps, readiness should become mechanical.

It does not.

An empty ledger can mean that no unresolved issue remains. It can also mean that the gap-discovery stage failed to notice one. Likewise, a gap marked as asked records that a question was posed; it does not establish that the underlying requirement was correctly resolved.

The paper’s monitoring-control framing is useful here. Gap discovery monitors uncertainty. Question selection acts on it. Stopping requires another judgment: whether the remaining uncertainty is low enough that formulation should proceed.

Those are three different failure surfaces.

For operators, make readiness a governed decision

Cognaptus’ interpretation goes beyond the benchmark but follows directly from the failure mode it exposes.

An optimization-agent workflow should maintain an explicit distinction among confirmed business requirements, unresolved requirements, and defaults the system would otherwise infer. Before mathematical formulation begins, a readiness gate can inspect that state rather than relying on the agent’s conversational confidence.

For higher-risk decisions, READY_TO_MODEL should not necessarily grant autonomy by itself. Organizations can attach escalation rules: unresolved high-severity conditions, weakly supported defaults, or ambiguous policy boundaries can trigger another clarification round or human review.

The interaction budget should also be explicit. Maximum recovery and minimum questioning are competing objectives. A procurement optimizer affecting millions of dollars may justify more elicitation than an exploratory planning assistant.

Internal evaluations can reproduce the benchmark’s underlying logic even without adopting OR-Clarify directly: remove known business requirements from test cases and measure whether the agent asks for them, silently substitutes defaults, and stops too early.

What the evidence does not establish

OR-Clarify contains 100 cases and uses simulated users whose answers are restricted to source-grounded facts. Real stakeholders may answer incompletely, contradict themselves, volunteer extra information, or resist repetitive questioning.

The evaluation also stops before downstream formulation and solver execution. Recovering every hidden requirement does not guarantee that the resulting mathematical program will be correct.

And the method comparison is protocol-dependent. InterOPT’s strongest improvement occurs under Choice interaction; the free-form results do not support the same advantage.

Those boundaries do not weaken the central finding. They locate it precisely: pre-formulation clarification is a measurable system capability, and explicit gap tracking improves parts of it without solving readiness.

The deeper design problem is not simply teaching an optimization agent to ask better questions. It is deciding what evidence the agent must possess before it is allowed to stop asking and start formalizing.

Cognaptus: Automate the Present, Incubate the Future.