Same Proposition, Different Stance: Grammar as a Model-Risk Variable

TL;DR for operators

Huang, Padó, and Weeber show that changing the grammatical realization of a political statement can change an LLM’s expressed stance even when the rewrite is intended to preserve the proposition.¹ Across four Gemma-3 and Qwen3 models, active/passive conversions and cleft constructions produced measurable stance movement, and variance decomposition indicates that the effect is systematic rather than ordinary run-to-run sampling noise.

For evaluation and prompt governance, this means two templates should not be assumed interchangeable merely because a human reader regards them as semantically equivalent. Robustness testing should include controlled grammatical reformulations, negation, semantic opposites, and combinations of these changes.

The mechanistic result adds a second layer. When a rewrite had changed the stance distribution, replacing internal states from the original wording could move the model back toward its original judgment. The strongest and most stable restorations came from complete decoder block outputs in mid-to-late layers at the final prompt position. That identifies states sufficient to restore behavior; it does not identify where sensitivity to the rewrite first arose.

Model scale also needs careful interpretation. Larger models in both tested families flipped polarity less often, but they also produced more neutral or unclear judgments. A lower flip rate can therefore reflect a different response distribution rather than unequivocally stronger judgment stability.

Semantically equivalent wording did not produce equivalent judgments

Consider two evaluation prompts that differ only in grammatical realization: an active sentence becomes passive, or an ordinary declarative becomes an it-cleft or wh-cleft. If their intended proposition is unchanged, an operator might reasonably expect the model’s substantive judgment to remain approximately fixed.

The experiment shows that this expectation is unsafe in the tested setting.

The authors start from 239 English political policy statements and evaluate six controlled rewrite types. Negation and semantic-opposite rewrites systematically reverse meaning. Four additional constructions—active/passive conversion, it-clefts, wh-clefts, and support verb constructions—are designed to preserve it. The newly generated variants were constrained and manually validated.

Four models—Gemma-3-4B-IT, Gemma-3-12B-IT, Qwen3-4B, and Qwen3-14B—answered each condition on a one-token seven-point agreement scale. Each prompt was sampled 30 times at temperature 0.8.

The strongest disruptions came from negation and semantic opposites, which is unsurprising given that they change polarity. More consequential for robustness testing is that meaning-preserving rewrites also moved judgments. Active/passive and cleft constructions could shift average stance by roughly half a Likert point or more in several tested conditions. Support verb constructions were comparatively stable.

The model therefore was not merely fluctuating around one fixed judgment. Linguistic realization itself contributed measurable variation.

The variation is larger than ordinary sampling noise

Repeated generation makes it possible to separate two explanations that would otherwise look similar: the model may be intrinsically stochastic under a fixed prompt, or it may systematically react to how the proposition is worded.

The paper decomposes response variation into three components. Purpose Sensitivity captures differences attributable to which policy issue is being judged. Articulation Sensitivity captures movement between an original statement and its rewrite. Model Uncertainty captures repeated-sampling variation under a fixed prompt.

Issue content remains the dominant driver overall, especially for the meaning-preserving constructions. But articulation effects consistently exceed sampling noise. For example, with passive rewrites, Articulation Sensitivity is 0.62 for Gemma-3-4B versus Model Uncertainty of 0.05; for Qwen3-4B the corresponding values are 0.23 and 0.11. It-clefts and wh-clefts show the same qualitative pattern.

This distinction matters operationally because repeated calls cannot diagnose the whole problem. If wording systematically moves the center of the response distribution, averaging more samples may estimate that shifted distribution more accurately without making the judgment invariant.

Larger models flip less, but they also abstain more

Within each tested family, the larger model produced fewer polarity flips. Across rewrite types, Gemma-3-4B-IT showed flip rates from 3.5% to 27.0%, compared with 0.0% to 19.0% for Gemma-3-12B-IT. Qwen3-4B ranged from 0.0% to 23.0%, while Qwen3-14B ranged from 0.0% to 14.3%.

Taken alone, that looks like straightforward robustness improvement.

The response distribution complicates the interpretation. Qwen3-4B produced clear-leaning responses in roughly 54.2%–68.2% of conditions, while Qwen3-14B was clear-leaning in only about 40.2%–56.2%. The larger model therefore moved toward neutral or unclear outputs more frequently.

For model selection, flip rate should consequently be paired with measures of decisiveness and distributional movement. An evaluation that reports only polarity reversals can rate a more conservative model as more robust even when part of the apparent gain comes from avoiding clear commitments.

Restoring an internal state can restore the stance

After establishing the behavioral instability, the paper asks a mechanistic question: when the rewrite changes the model’s output distribution, which internal states can reverse that change?

The authors use activation patching. They run the original statement and the rewritten statement separately, then replace one internal activation in the rewritten run with the corresponding activation from the original run. If the output distribution moves back toward the original stance, that patched state carries information sufficient to restore the behavior.

The intervention is evaluated with a normalized restoration score:

where $p_c$ is the original-statement distribution, $p_k$ the rewritten distribution, and $p_{int}$ the post-intervention distribution. A score near one means that the intervention removes most of the rewrite-induced distributional deviation.

Across both model families, patching complete decoder block outputs produced stronger and more stable restoration than patching isolated attention or MLP outputs. The most effective sites were concentrated in mid-to-late decoder layers at the final prompt position.

The likely interpretation is not that grammar is “stored” in those layers. Decoder block outputs integrate contributions propagated through the residual stream, so replacing the complete block state can restore a downstream representation on which the eventual stance decision depends.

That is also the limit of the causal claim. The experiment identifies sufficient restoration sites. It does not establish where the rewrite-sensitive computation originated, which earlier token positions contributed to it, or the complete causal circuit.

Controlled rewrites belong in robustness testing

Cognaptus inference: for policy, compliance, survey, eligibility, and other polarity-sensitive workflows, wording structure belongs in the evaluation specification.

Generic paraphrase tests remain useful, but they mix many linguistic changes at once. Controlled rewrites diagnose narrower failure modes: active versus passive voice, information-structural clefts, explicit negation, or semantic reversal. They can reveal whether a system is stable under a specific transformation that production templates may actually undergo.

The appendix reinforces this as a robustness question rather than introducing a separate thesis. Pairwise combinations generally produced higher flip rates than individual rewrites. For example, negation combined with a wh-cleft reached 35.2% flips for Gemma-3-4B and 48.3% for Qwen3-4B. Multiple linguistic perturbations can therefore interact rather than behaving as independent cosmetic edits.

A practical evaluation can accordingly track at least three quantities: polarity flips, distributional movement across the full response scale, and movement toward neutral or unclear outputs. Those measures answer different questions about stability.

The evidence is narrow enough to use carefully

The study tests four relatively small open-weight models from only two families, one English political-statements dataset, and a closed-ended seven-point stance task. It does not establish that the same sensitivity profile holds for larger frontier models, other languages, open-ended reasoning, or different application domains.

The meaning-reversing variants also come from the existing ProbVAA dataset, while the meaning-preserving constructions were newly generated and validated. Differences between those two groups therefore cannot be attributed entirely to linguistic properties.

Most importantly, the mechanistic evidence is useful for localization but not root-cause identification. Patching is performed only at the final prompt position, and some model-rewrite combinations lack enough stance changes for stable intervention analysis.

Within those boundaries, the result changes a concrete evaluation assumption. Semantic equivalence at the human level does not guarantee behavioral equivalence at the model level. For systems whose decisions depend on polarity, agreement, eligibility, or rule interpretation, robustness must be tested against the structure of language itself—not only against repeated sampling of one preferred prompt.

Reference

Cognaptus: Automate the Present, Incubate the Future.