TL;DR for operators

A post-trained classifier can disagree with its own-base pretrained model without that example becoming useless for calibration. The paper behind SupportCal finds that adding some disagreement examples back into label-free calibration improves Expected Calibration Error, but giving the entire disagreement set full weight eventually makes calibration worse. The useful decision is therefore not simply whether to retain disagreement, but how much influence each disagreement example receives.

SupportCal keeps agreement examples at full weight and grades disagreement examples using two signals: how much the own-base model still supports the post-trained model’s selected answer, and whether compatible pretrained references corroborate that support. Across the evaluated fixed-label tasks, this generally improves calibration over an agreement-only baseline without retraining the target model. For deployments that use confidence thresholds for abstention, escalation, triage, or routing, the result suggests a more informative use of unlabeled calibration data—provided the base model and reference models are available.

Disagreement is not one kind of evidence

Consider a classifier whose reported probability determines whether a prediction is accepted automatically or sent to a human. The organization has production examples but no correctness labels for a new calibration set. One available signal is the model family itself: compare the post-trained model with the pretrained model from which it originated.

A conservative rule is to calibrate only on examples where the two models choose the same answer. Disagreement appears contaminated: if the base model prefers another option, aligning the post-trained model toward it could soften confidence for the wrong reason.

The controlled diagnostic in this paper complicates that rule. When disagreement examples are reintroduced with ordinary unit weight, calibration initially improves. The strongest aggregate region occurs when roughly 40–60% of the disagreement set is included. As inclusion approaches the full set, however, the benefit deteriorates.

That pattern matters because it rejects two simple treatments of disagreement. Blanket exclusion discards information that can improve calibration. Full inclusion assumes examples with very different levels of evidential support deserve the same influence.

The resulting problem is one of weighting.

SupportCal grades disagreement instead of deleting it

Luo and colleagues propose SupportCal, a label-free calibration procedure for post-trained language models.1 The method separates calibration examples into those where the post-trained model and its own-base pretrained model agree and those where they disagree.

Agreement examples receive weight 1. For a disagreement example, the method first asks how strongly a pretrained reference supports the answer selected by the post-trained model. Its relative support is

$$ u_n^{(r)} = \frac{p_{n,\hat{y}_n}^{(r)}} {\max_k p_{n,k}^{(r)}}. $$

The score equals 1 when reference $r$ itself prefers the post-trained model’s selected option. It falls as that reference increasingly favors another option. This turns disagreement from a binary event into a graded one: a near tie and a sharp contradiction no longer look identical.

But the own-base model’s score is not used alone. A relatively high value could reflect genuine support for the selected answer, or merely a diffuse probability distribution. SupportCal therefore adds cross-reference corroboration from selected, size-compatible pretrained models.

The disagreement weight becomes

$$ w_n = \begin{cases} 1, & n \in A, \\ u_n^{(o)} c_n^{\star}, & n \in D, \end{cases} $$

where $u_n^{(o)}$ is own-base relative support and $c_n^{\star}$ summarizes support from the selected corroborating references.

This distinction is easy to miss: the extra references are not additional teachers. They only affect the sample weights. The target’s own-base pretrained model remains the sole distribution used in the KL alignment objective.

SupportCal then fits one scalar temperature:

$$ \hat{\tau} = \arg\min_{\tau\gt 0} \frac{ \sum_n w_n D_{\mathrm{KL}} \left( p_n^{(o)} \,\middle\|\, \operatorname{softmax}(z_n/\tau) \right) }{ \sum_n w_n }. $$

Because temperature scaling preserves logit ordering, this changes confidence without changing which class the target model predicts.

Corroboration is doing real work

The paper includes a relative-support-only ablation on representative models. Its purpose is narrower than the main benchmark comparison: it tests whether grading disagreement by the own-base model alone is enough.

On the displayed MedMCQA and MathQA targets, it is not. Relative-support-only weighting can perform worse than the agreement-only DACA baseline. Adding cross-reference corroboration restores or improves calibration on those examples.

That result supports the mechanism behind the method rather than proving that every additional reference model is valuable. The distinction is important. SupportCal is not simply averaging more model opinions. The references are used to decide how much trust to place in an ambiguous calibration example.

The reported results are also fairly insensitive to the two tested reference counts. The largest absolute per-target difference between SupportCal configurations is 0.19 percentage points on MedMCQA and 0.10 on MathQA. Within those tested settings, using more corroborating references did not produce a large systematic shift.

The benchmark gains favor weighting over blanket exclusion

The main comparison is against DACA, the closest agreement-only PLM-referenced calibration baseline.

Dataset DACA macro ECE SupportCal macro ECE Observed pattern
MedMCQA 6.654 pp 5.066 pp ($M=2$); 5.109 pp ($M=4$) Lower ECE on all 12 targets for both tested settings
MathQA 7.790 pp 5.696 pp ($M=1$); 5.714 pp ($M=2$) $M=1$ improves all 7 targets; $M=2$ improves 6 of 7
TweetEval Sentiment 16.59 pp 15.06 pp ($M=2$); 15.16 pp ($M=4$) Lower ECE on all 6 supplementary targets

The strongest reading is therefore comparative and bounded. Across these fixed-label evaluations, selectively weighted disagreement performs more consistently than discarding all disagreement.

TweetEval is useful because it extends the observed pattern beyond multiple-choice question answering to three-class sentiment classification. It does not establish performance for arbitrary output spaces.

The boundary margin explains when calibration softens too far

Support-aware weighting could otherwise be read as only an empirical heuristic. The paper adds a formal characterization of the fitted temperature.

Expressing the objective in inverse-temperature coordinates, each example contributes a signed boundary margin $m_n$. The weighted average is

$$ M_w = \frac{1}{W}\sum_n w_n m_n. $$

Under the paper’s nondegeneracy conditions, the weighted objective is strictly convex, and a unique finite optimal temperature exists if and only if

$$ M_w\gt 0. $$

If the weighted margin is non-positive, the optimum moves to the boundary corresponding to $\tau \rightarrow \infty$: extreme softening.

The empirical weight analysis is consistent with that interpretation. Across MathQA, MedMCQA, and TweetEval Sentiment, disagreement examples with negative boundary margins receive lower mean and median SupportCal weights than those with positive margins.

For implementation, this supplies more than a proof artifact. Cognaptus interprets $M_w$ as a useful diagnostic for whether the selected weighting scheme is pushing calibration toward a finite adjustment or toward effectively washing out confidence.

Where the method fits operationally

The immediate deployment case is a finite-label model whose probabilities drive downstream decisions and for which correctness-labeled calibration data are unavailable. Examples include confidence-based abstention, human escalation, risk triage, and routing among systems.

SupportCal is attractive in that setting because it is post-hoc. Once the scalar temperature has been fitted, production inference has the same form as ordinary temperature scaling. There is no target-model retraining.

The cost arrives earlier. Calibration requires forward inference from the corresponding own-base model and a compatible pool of pretrained references. That produces one-time compute overhead and makes model access part of the method’s applicability conditions.

Those conditions also define the evidence boundary. The experiments concern fixed-label tasks with a shared finite output space. The method assumes access to the target’s own-base pretrained model and compatible references. The paper does not establish the same weighting construction for free-form generation, where “support for the selected answer” would require a different formulation.

Confidence calibration benefits from degrees of disagreement

The paper’s more consequential contribution is not that disagreement should be trusted. It is that disagreement is heterogeneous.

Some examples contain enough support from the model family to help estimate an appropriate confidence adjustment. Others exert pressure toward excessive softening. Treating both groups identically loses that distinction, whether by assigning every disagreement example weight zero or weight one.

Within the evaluated setting, SupportCal shows that a graded treatment can recover useful calibration information without retraining or collecting correctness labels. For operators already using confidence as a control signal, that changes the design choice from “exclude disagreement” to a more precise question: what evidence justifies each disagreement example’s influence?

Cognaptus: Automate the Present, Incubate the Future.

  1. Linhan Luo and Lequan Lin and Dai Shi and Feng Chen and José Miguel Hernández-Lobato and Junbin Gao (2026). SupportCal: Label-Free Calibration of Post-Trained LLMs via Reference Support and Corroboration. arXiv:2609.24303. https://arxiv.org/abs/2609.24303 ↩︎