Rebuttal Agents, Not Rebuttal Text: Why ‘Verify‑Then‑Write’ Is the Only Scalable Future

Rebuttal is where polite language goes to be cross-examined.

A reviewer asks why the baseline is missing. Another says the theory is unclear. A third implies that the claimed novelty is, shall we say, generously interpreted. The authors have a few days to respond, and every sentence must do three jobs at once: answer the concern, avoid overclaiming, and preserve the paper’s strategic position.

This is exactly the kind of task where large language models look useful. Ask for a rebuttal, get a polished answer, save time. Wonderful — until the model invents an experiment, merges two distinct criticisms into one vague apology, or promises a revision that contradicts another response. Peer review does not reward confident fog.

The paper Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance makes a useful correction to this instinct.¹ It does not treat rebuttal writing as a better prose problem. It treats it as a decision-and-evidence organization problem. The writing comes last. Before that, the system must parse concerns, retrieve manuscript evidence, decide whether external literature is needed, build a response plan, and expose that plan for human inspection.

That is the important shift. Not “AI writes rebuttals.” We already knew AI could produce solemn paragraphs beginning with “We thank the reviewer.” The more interesting claim is that high-stakes responses should be generated only after the underlying evidence structure has been made inspectable.

In other words: verify, then write. Revolutionary, apparently, because the industry has spent several years doing the reverse.

The real failure is not bad wording; it is missing decision structure

The obvious way to automate rebuttal writing is direct-to-text generation: feed in the paper, feed in the reviews, ask the model to produce responses. The paper argues that this is structurally flawed because the model is asked to perform too many hidden operations at once.

A good author response must first identify what the reviewer actually asked. Then it must determine whether the manuscript already contains enough evidence. If not, it must distinguish between what can be clarified, what requires external support, and what requires new work. Finally, it must translate that plan into diplomatic prose without losing specificity.

Direct generation collapses all of that into a single decoding step. That is efficient in the same way that jumping from a balcony is efficient: fewer stairs, poorer control.

The authors describe two dominant existing patterns. One is supervised fine-tuning on paper-response pairs, which may teach models the style of rebuttals without giving them access to the actual factual constraints of a new manuscript. The other is interactive chat-based prompting, where powerful general-purpose models can help but only after the author manually shepherds the model through concern parsing, evidence retrieval, and revision cycles.

Both approaches have the same weakness: the intermediate reasoning is either absent or hidden. The user receives text, not an auditable map of why that text should be trusted.

RebuttalAgent is designed around the opposite premise. Its intermediate artifacts matter as much as the final draft.

The mechanism starts by breaking the review before answering it

The first move in RebuttalAgent is input structuring. The manuscript is converted into a paragraph-indexed representation and compressed into a compact form that preserves important technical claims and experimental results. This compact version is not meant to replace the paper. It is a navigation layer.

At the same time, the system extracts atomic concerns from reviewer comments. This sounds mundane, but it is one of the places where rebuttals quietly fail. Reviewers often write compound objections: the method is unclear, the baseline is missing, the novelty is overstated, and the notation is confusing. A fluent model may smooth these into one friendly paragraph. A useful assistant must split them.

The paper’s prompt templates make this explicit. Concerns should not be merged if they require different evidence. A request for “baseline X” and a request for “baseline Y” are separate concerns because they may require different experiments. A complaint about novelty and a complaint about clarity are not the same objection merely because both sound negative. This is not glamorous AI. It is administrative discipline. In rebuttal writing, that discipline is the product.

The mechanism can be summarized as:

Reviewer comments
        ↓
Atomic concerns
        ↓
Concern-conditioned manuscript context
        ↓
Internal and external evidence bundle
        ↓
Inspectable response plan
        ↓
Human revision checkpoint
        ↓
Final rebuttal draft

The key point is sequencing. The system is not allowed to write before it has built a structured representation of what must be answered.

Hybrid context is the quiet engineering trick

Long documents create a familiar problem for LLM systems. If the model sees too little of the manuscript, it misses evidence. If it sees too much, relevant details drown in a context swamp. RebuttalAgent handles this through what the authors call an atomic-concern-conditioned hybrid context.

For each concern, the system searches the compressed manuscript representation to locate relevant sections. Then it selectively expands those sections back into higher-fidelity original text while leaving the rest of the paper in compressed form.

That design matters because rebuttals often depend on small, exact details. A reviewer may ask whether an assumption is justified in Section 3, whether a result appears in Table 2, or whether an equation contradicts a later claim. A compressed summary can guide the system toward the right location, but the final argument needs source-level precision.

This hybrid context gives the model both map and terrain. The summary tells it where to look. The raw excerpt tells it what can actually be said.

For business readers, this is one of the most portable ideas in the paper. Many enterprise AI failures are not caused by the model being unable to write. They are caused by the system giving the model either too little context or the wrong kind of context. A compliance response, investor Q&A answer, audit reply, or technical support escalation often requires the same pattern: broad compressed context plus narrow high-fidelity evidence.

The system should not ask, “Can the model answer?” It should ask, “What evidence packet should the model be allowed to answer from?”

External evidence is not decoration; it is a gate

Some reviewer concerns cannot be answered from the manuscript alone. A reviewer may question novelty, request comparison with another method, or cite an external paper. RebuttalAgent therefore includes on-demand external search, using scholarly retrieval and screening to construct citation-ready evidence briefs.

The important word is “on-demand.” External evidence is not sprayed across the response to make it look scholarly. The system first decides whether search is needed. If the manuscript already contains the answer, search is unnecessary. If the concern asks for comparison, related work, or outside positioning, the system generates targeted queries and filters candidate papers for actual relevance.

This is not just a retrieval module. It is a policy for when retrieval is allowed to enter the workflow.

That distinction is easy to miss. In many RAG-style business systems, retrieval becomes a decorative reflex: retrieve something, cite something, sound grounded. RebuttalAgent’s design is more disciplined. Evidence construction is concern-conditioned. External sources are useful only when they help answer a specific concern.

The ablation results make this point stronger. Removing evidence construction causes some of the largest drops in the paper’s component-level metrics. In the Gemini-3-Flash ablation setting, coverage falls from 4.51 to 4.26, specificity from 4.49 to 4.19, and suggestion constructiveness from 4.09 to 3.82. Those are not merely style losses. They indicate that evidence artifacts help the system produce more concrete and actionable responses.

The lesson is blunt: the evidence layer is not a footnote generator. It is load-bearing infrastructure.

The response plan is where hallucination is supposed to stop

The most consequential mechanism appears after evidence construction and before drafting: the response plan.

RebuttalAgent separates interpretative defense from necessary intervention. If the manuscript already supports a response, the plan can propose a clarification. If the reviewer asks for a missing experiment, the system should not fabricate a result. It should produce an action item.

This is the part of the paper that matters beyond academia.

In a direct-to-text workflow, a model under pressure will often generate the answer the user wishes existed. In a verify-then-write workflow, the system must mark the difference between:

The plan is a control surface. It lets the author inspect what the system intends to argue before the prose makes everything sound more settled than it is.

The appendix case study illustrates this logic. In one example, the system responds to concerns about theoretical clarity by proposing concrete revisions: rewriting the proposition, adding proof structure, introducing empirical sanity checks, expanding appendix evidence, and adding didactic aids. The case study is not main quantitative evidence; it is a qualitative illustration of how the intermediate plan makes commitments visible.

That visibility is the point. A human author can reject, modify, or verify an action item. They cannot easily audit a polished paragraph that has already hidden the reasoning.

RebuttalBench measures the right discomforts

The paper introduces RebuttalBench to evaluate rebuttal quality using OpenReview-derived peer-review interactions. The dataset construction focuses on real review-response contexts, with a larger corpus of roughly 9.3K review-rebuttal pairs and a more focused challenge set based on representative ICLR 2023 cases.

The evaluation is not based on BLEU, ROUGE, or other text similarity metrics. Good. A rebuttal can be lexically different from the original author response and still be better. It can also sound similar and be strategically useless.

Instead, the paper uses an LLM-as-judge rubric across three dimensions:

Each dimension has three subcomponents, producing nine component scores on a 0–5 scale. The judge model used in the experiments is Gemini-3-Flash. That choice matters for interpretation: the results are scalable and structured, but they are still automated evaluations rather than human area-chair judgments.

The purpose of RebuttalBench is therefore not to prove that RebuttalAgent will win every real peer-review exchange. It tests whether a structured pipeline produces responses that score better on the qualities a rebuttal should have: coverage, specificity, evidence support, coherence, tone, clarity, and constructiveness.

That is a narrower claim. It is also a more useful one.

The main results say structure beats raw fluency

The experimental design compares direct-to-text baselines against RebuttalAgent versions using the same underlying model backbone. This is important. If RebuttalAgent-GPT-5-mini beats direct GPT-5-mini, the improvement cannot be waved away as “they used a stronger model.” The model is held constant; the workflow changes.

The reported average scores show consistent gains:

The largest component gains appear in relevance and specificity. For GPT-5-mini, specificity rises by +1.33. For DeepSeek-V3.2, coverage rises by +0.78. For Gemini-3-Flash, coverage improves from 4.00 to 4.51 and specificity from 3.77 to 4.49.

This pattern is more interesting than the average score. Communication quality improves, but less dramatically. That suggests the system is not mainly making the prose prettier. It is making the response better aimed.

The weaker backbone also benefits more. GPT-5-mini gains +0.57 on average, while Gemini-3-Flash gains +0.38. The paper interprets this as evidence that explicit structuring can partially compensate for weaker model capability. I would phrase it slightly differently: structure reduces the amount of intelligence the model has to improvise.

That is not a small point. Enterprise AI often assumes that the answer to reliability is a better model. This paper suggests a cheaper and more controllable alternative: reduce the hidden reasoning burden through workflow design.

The ablation study identifies the load-bearing modules

The ablation study removes one module at a time: input structuring, evidence construction, and checkers. Its likely purpose is not to prove the whole system again. The main results already do that. The ablation asks which intermediate artifacts matter.

The answer is: evidence construction matters most, but the modules are complementary.

The evidence-construction ablation produces the clearest degradation. Removing it reduces coverage, specificity, engagement, and constructiveness. Input structuring also matters: without it, semantic alignment drops from 4.88 to 4.71 and evidence support from 3.39 to 3.23 in the reported ablation setting. Removing checkers has smaller effects, with some scores barely moving or even slightly improving, but the authors still observe degradations in quality dimensions such as evidence support and engagement.

That last detail deserves careful interpretation. Checkers may matter more in messy real-world use than in benchmark scoring, especially when humans interact with the plan. But the paper’s own numbers do not show checkers as the dominant source of gain. Evidence construction is the thickest wall in the house.

The business lesson is not “automate replies”; it is “separate evidence from wording”

The obvious business translation is to use systems like this for RFP responses, investor Q&A, regulatory letters, audit replies, incident reports, and high-stakes customer escalations.

That translation is valid, but only if the mechanism is preserved.

The shallow version would be: “Let AI draft professional responses.” That is exactly the failure mode the paper is trying to move beyond. The deeper version is: build systems where every outward-facing answer is generated from a tracked concern list, a verified evidence packet, and a reviewable response plan.

For business use, the equivalent workflow would look like this:

The ROI is not merely faster writing. Faster writing is nice, like a clean desk or a functioning printer. The larger value is reducing the cost of diagnosis: What exactly was asked? What evidence do we have? What do we need to verify? What are we promising? Which statement creates risk?

That is where AI agents become useful in business workflows. They should not simply produce documents. They should expose the decision objects that make documents safe to produce.

A compliance team should care more than a copywriting team

The most natural enterprise buyer for this type of architecture is not the marketing department. It is any team that must answer under scrutiny.

Consider a regulatory response. A regulator asks why a model made a certain decision, whether controls were in place, and what remediation will occur. A direct-to-text LLM can generate something respectful and plausible. That is precisely the danger. The response may sound complete while silently mixing evidence, policy, aspiration, and fantasy.

A RebuttalAgent-style workflow would force separation:

1. What concern is being answered?
2. Which internal records support the answer?
3. Which claims require external standards or policy references?
4. Which issues require action rather than explanation?
5. Which commitments must be approved before they appear in the final letter?

The same applies to investor diligence. A founder responding to a data-room question should not let an LLM improvise churn numbers, roadmap dates, or customer commitments. The system should produce an evidence-linked plan: available metric, source file, interpretation, missing item, proposed answer, required approval.

This is not glamorous. It is the dull machinery of trust. Conveniently, dull machinery is where many businesses make or lose money.

The boundaries are real, and they affect deployment

The paper’s evidence supports the workflow design, not universal deployment readiness.

First, the benchmark is based on academic peer-review contexts. Rebuttal writing is high-stakes, but it is not identical to legal, medical, financial, or regulatory communication. Those domains may require stricter evidence provenance, role-based approvals, audit logs, and liability controls.

Second, the evaluation relies heavily on LLM-as-judge scoring. The rubric is well aligned with rebuttal quality, and the component-level breakdown is useful. Still, automated judges are not the same as human reviewers, area chairs, regulators, or clients. A system can score well on coverage and still make a strategically poor commitment in a real negotiation.

Third, the main experiments run RebuttalAgent fully automatically. The paper argues this is a conservative lower bound because human checkpoints could improve performance. That may be true, but it also means the paper does not directly measure the value, cost, or friction of real human-in-the-loop use.

Fourth, the appendix prompt for rebuttal letter writing includes a placeholder convention for speculative experimental results: invented values must be marked with an asterisk for human verification. This is better than unmarked hallucination, but it is still a dangerous design pattern if users miss or remove the marker. In high-stakes business settings, the safer default is not “invent and mark.” It is “block and request verified input.”

These boundaries do not weaken the paper’s core contribution. They define where the contribution should be used carefully. The paper is strongest as an architecture argument: expose the evidence chain before generating text.

The future is not better paragraphs; it is accountable intermediate artifacts

The most useful idea in Paper2Rebuttal is not that LLM agents can help write author responses. Of course they can. A toaster can also warm bread; this is not the intellectual event.

The useful idea is that high-stakes communication should be decomposed into artifacts that humans can inspect before the final prose appears. Concern lists. Evidence bundles. Search decisions. Action items. Commitment checks. Drafts generated only after the system knows what it is allowed to say.

That is the difference between an AI writing assistant and an AI decision-support system.

RebuttalAgent happens to operate in academic peer review, but the design pattern travels. Any organization that uses AI to answer difficult questions will eventually face the same choice: generate fluent text directly, or build the workflow that makes fluent text accountable.

The first option is cheaper until it becomes expensive.

The second option is slower until it becomes scalable.

Cognaptus: Automate the Present, Incubate the Future.