Disagreement Has a Weight: Calibrating LLM Confidence Without Labels
TL;DR for operators A post-trained classifier can disagree with its own-base pretrained model without that example becoming useless for calibration. The paper behind SupportCal finds that adding some disagreement examples back into label-free calibration improves Expected Calibration Error, but giving the entire disagreement set full weight eventually makes calibration worse. The useful decision is therefore not simply whether to retain disagreement, but how much influence each disagreement example receives. ...