Asking Better Does Not Mean Specifying Better

TL;DR for operators

When a stakeholder gives a coding agent a vague software request, asking clarification questions can improve the resulting specification. But asking effective questions and accurately documenting the answers are different capabilities.

A benchmark covering 341 development tasks finds that the model with the highest proportion of effective clarification questions achieves substantially lower requirement alignment than the model with the lowest proportion. Coding-agent harnesses also improve question-level metrics while generally weakening specification quality.

For software teams, this changes how coding agents should be evaluated. Requirement elicitation, specification synthesis, and implementation deserve separate assessments. Interactive clarification improves requirement-document alignment, and better-prepared documents improve downstream coding scores in the tested configuration. Neither result guarantees that longer conversations or more elaborate agent tooling will improve production software.

The model that asks better questions may write a worse specification

A stakeholder requests a new application feature in a sentence or two. The coding agent can start implementation immediately, draft a specification from its assumptions, or ask for missing details. Asking questions seems safer, particularly when the request leaves essential behavior unspecified.

Yet the effectiveness of those questions does not reliably predict what the agent eventually writes down.

In Don’t Guess, Just Ask, Mingyu Chen and colleagues introduce ReqClarifyBench to investigate this discrepancy.¹ Across six evaluated models, Gemini-3.1-Pro elicits new requirement information through the highest proportion of its questions, while GPT-5.6-Sol produces the most accurate requirement documents.

Source: Section 5.1, Table 4. Values are averages across three simulated user personas.

The distinction matters because the two metrics describe different activities.

Effective Question Rate (EQR) measures the proportion of clarification questions that elicit at least one atomic requirement. An atomic requirement is an individually assessable expectation about the software.

Requirement Alignment (ReqAlign) instead evaluates the completed document against reference requirements. Each requirement receives a score of 1 if covered, 0 if omitted, and −1 if violated. The metric averages those scores, so it is not simply the percentage of requirements satisfied.

Gemini also leads in the average number of requirements clarified per question, at 1.702 versus GPT-5.6-Sol’s 0.650. Nevertheless, GPT-5.6-Sol achieves more than twice Gemini’s average ReqAlign.

These results establish a performance distinction, not its complete internal explanation. A model can successfully obtain information during a conversation without reproducing it accurately in the subsequent document. The benchmark makes that discrepancy measurable; it does not identify precisely which memory, reconciliation, or synthesis processes account for each model’s result.

For an organization purchasing coding-agent capabilities, question effectiveness alone is therefore an incomplete selection criterion.

A short user request can conceal hundreds of requirements

ReqClarifyBench contains 341 repository-level tasks drawn from three existing software-generation benchmarks: 100 miniapp tasks, 140 game-development tasks, and 101 website-development tasks.

The authors reconstruct detailed requirements from source artifacts, associate requirements with supporting evidence, and validate the documents and abbreviated intents using human reviewers.

The resulting contrast is substantial. An initial user intent contains approximately 13 tokens on average, while its detailed specification contains approximately 2,380 tokens and 193 atomic requirements.

This design addresses a limitation of benchmarks where coding systems receive fixed specifications. When a coding agent fails with a complete task description, implementation quality is the principal object of evaluation. When a user starts with an incomplete request, failure can also arise because necessary requirements were never elicited or were subsequently misrepresented.

The authors evaluate six LLMs under a standardized protocol permitting up to 20 clarification rounds. Their simulated users follow three behavioral profiles derived from real developer–coding-agent conversations in SWE-chat.

The benchmark thus separates the quality of information acquisition from the fidelity of its final representation, while retaining behavioral code-evaluation artifacts for assessing downstream consequences.

The user changes the clarification problem

The same coding model encounters different information conditions depending on how the stakeholder responds.

Under the Expert Nitpicker persona, users provide detailed, explicit feedback. Across the six models, average ReqAlign reaches approximately 0.609.

For Vague Requesters, who provide less informative answers, average alignment falls to 0.433. Mind Changers, who introduce or revise requirements during the conversation, produce average alignment of 0.439.

The interaction dynamics differ as well.

Expert Nitpickers generally produce shorter, information-intensive conversations. Vague Requesters can induce longer exchanges with little new information per question. Mind Changers may disclose consequential requirements later, after earlier assumptions have already shaped the developing specification.

GPT-5.6-Sol, for example, averages approximately 14–15 clarification rounds with Vague Requesters, whereas Gemini-3.1-Pro typically stops after roughly two or three rounds across personas.

Neither stopping pattern establishes an optimal conversation length. The relevant quantity is how much missing information becomes available, and how accurately the system incorporates it.

This distinction also appears when the researchers compare standalone models with their corresponding coding-agent harnesses—the surrounding environments that organize model execution, such as Codex and Claude Code.

In these configuration comparisons, harnesses improve EQR and average requirements clarified per question. Yet final document-quality metrics generally deteriorate. Under the Expert Nitpicker persona, ReqAlign falls by 15.2% for Claude and 24.0% for GPT when their respective harnesses are used.

This is a targeted harness comparison, not evidence that agent orchestration generally harms requirements engineering. It establishes that improving the process of extracting information need not improve the artifact that process produces.

The specification improves code, even before interactive clarification

The discrepancy between questioning and documentation might suggest that clarification adds complexity without dependable benefits. The downstream experiments provide a more discriminating answer.

First, across all six evaluated models, interactive clarification increases average ReqAlign from 0.212 to 0.494—a reported relative improvement of 133.1%.

Semantic similarity improves more modestly: average BERTScore F1 rises from 0.664 to 0.712.

That difference explains why atomic-requirement evaluation matters. Two documents may be reasonably similar in general meaning while differing considerably in omitted or contradicted implementation requirements.

The authors then examine whether preparing specifications changes code-generation performance.

Using Codex with GPT-5.6-Sol, they compare implementation from the initial intent, a directly generated requirement document, and a document produced after interactive clarification.

Source: Section 5.5, Table 5. Both document types are generated with GPT-5.6-Sol; results are averaged across personas. Scores use different behavioral evaluation protocols and must be compared within each benchmark, not across benchmark rows.

Two observations emerge.

Generating an explicit document improves implementation scores relative to coding directly from the initial intent, even without additional user feedback. Transforming an ambiguous request into a structured implementation artifact can therefore help the coding system.

Interactive clarification contributes additional gains across all three benchmarks. The improvement is consistent in direction, although its magnitude varies by task domain and evaluation protocol.

The experiment does not separately quantify how much of this gain comes from newly disclosed requirements, improved document organization, or other interactions between the specification and the coding system.

It nevertheless supplies downstream evidence that requirement preparation has value beyond producing an apparently comprehensive document.

Three decisions for the coding workflow

For organizations introducing coding agents into feature development, Cognaptus draws a practical implication: the workflow should distinguish three decisions rather than treating requirement understanding as a single capability.

The first is what to ask. Product stakeholders and coding assistants need a way to identify missing information. Evaluation should measure the yield of clarification questions under different stakeholder behaviors, including vague answers and changing requirements. Conversation length should be monitored as a cost, not rewarded as a performance objective.

The second is what to record. Before implementation, the system should consolidate disclosed requirements into an explicit specification. An operational quality check can examine atomic-requirement coverage, omissions, contradictions, and traceability to stakeholder responses. This check addresses a capability that question-level metrics cannot measure.

The third is when to implement. The team can define authorization conditions around specification completeness, unresolved contradictions, and stakeholder approval. The coding agent then receives the reviewed specification as its implementation input.

These are proposed workflow controls, not a production architecture experimentally validated by ReqClarifyBench. Their suitability depends on project complexity, the cost of clarification, and the reliability of the organization’s own review process.

They also suggest a more informative vendor evaluation: test model–harness combinations against representative stakeholder interactions, inspect the generated requirement artifacts, and measure resulting code behavior rather than selecting solely on conversational or coding benchmarks.

What the benchmark cannot settle

The study’s strongest evidence concerns controlled comparisons within its constructed evaluation environment.

Its users are simulated by an LLM conditioned on reference requirements, although the three personas originate from real interaction data. Human reviewers distinguished simulated personas with 95.8% average accuracy across 180 trajectories, supporting their distinguishability but not establishing that they reproduce real stakeholder behavior.

Reference requirements were reconstructed with LLM assistance and human validation. ReqAlign also depends on automated judgments, evaluated against 200 manually annotated cases. Neither process eliminates possible reference or measurement errors.

Task coverage is restricted to games, miniapps, and websites. The downstream implementation experiment uses one coding configuration rather than all six models. No production deployment comparison establishes effects on maintenance effort, developer productivity, or long-term reliability.

Consequently, the results support testing staged requirement workflows under realistic conditions. They do not establish a universally superior model, harness, or mandatory sequence of development stages.

Understanding requirements deserves its own evaluation

ReqClarifyBench identifies a consequential gap in coding-agent evaluation. Asking effective questions, producing a faithful specification, and implementing software from that specification are related capabilities, but success in one does not guarantee success in the others.

The benchmark finds measurable benefits from interactive clarification and explicit requirement documents. It also shows that user behavior and execution environments can change those benefits in ways that question-level metrics alone would miss.

For software organizations, the practical opportunity is to evaluate the entire path from stakeholder intent to implemented behavior, with explicit evidence at each transition. More capable coding agents make that discipline more valuable, not less.

Cognaptus: Automate the Present, Incubate the Future.