Coverage Is Not a Range: RPS for Ordinal Conformal Prediction

TL;DR for operators

A risk system for ordered outcomes should return a coherent low-to-high range, not merely a collection of plausible labels. Coverage matters, but it does not distinguish an adjacent miss from one several severity levels away. A narrow range can therefore look efficient while hiding a rare, costly outcome.

The method examined here first attaches a calibrated set of plausible labels to each prediction, then evaluates ordered errors through cumulative probability across the label scale. Its ranked probability score produces candidate-label scores that fall toward a predictive median and rise afterward. Thresholding this V-shaped sequence yields nested, median-centered contiguous ranges, even when the classifier’s class probabilities are not unimodal.

The operational trade-off is deliberate: these ranges are not always the shortest, but they can better preserve distant outcomes and reduce the severity of misses outside the interval. The practical guarantee is marginal coverage under exchangeable calibration and test data; the stronger ordinal-risk result assumes the true conditional distribution is known and should not be interpreted as case-by-case coverage.

A covered outcome can still be badly represented

Suppose a clinical model predicts one of five disease-severity grades. A set containing grades 2 and 4 may count as uncertainty, but it omits the intermediate grade that makes the range interpretable. A set containing only grades 2 and 3 may look efficient, yet expose the operator to a grade-5 outcome whose distance is operationally larger than an adjacent miss.

These are different failures. Coverage asks how often the true label appears inside the prediction set. Contiguity asks whether the set forms a coherent interval on the ordered scale. Miss severity asks how far an uncovered outcome lies from the nearest interval boundary. Two methods can achieve similar marginal coverage and differ materially on the other two criteria.

The paper by Stefan Haas, Luca Killmaier, Alireza Javanmardi, and Eyke Hüllermeier targets this gap by using the ranked probability score as the nonconformity measure inside split conformal prediction.¹ The contribution is not a new base classifier. It is a post-hoc rule for turning an existing model’s class probabilities into calibrated ordinal prediction sets.

Cumulative probability determines the interval geometry

For a new case, split conformal prediction evaluates every candidate label and includes those whose nonconformity score falls below a threshold estimated from calibration data. With RPS, that score compares the model’s cumulative probability distribution with the cumulative step function implied by the candidate label:

This cumulative construction makes ordinal distance visible. Moving a candidate label across several class boundaries changes several cumulative comparisons, while moving it by one rank changes fewer. The method therefore uses more of the predictive distribution than a rule that starts at the most probable class and expands locally.

The decisive result is the recurrence between neighboring candidate-label scores:

Because $F_X(\ell)$ is a cumulative distribution, the increment is nondecreasing as the label index rises. It is negative before cumulative probability reaches one half and positive afterward. Candidate-label scores therefore descend toward a predictive median and rise after it. Thresholding a V-shaped sequence produces a contiguous interval containing that median.

This is a structural property of the score, not a shape restriction imposed on the classifier. The predicted class probabilities may be dispersed or multimodal. RPS can still return a median-centered contiguous set, and the sets remain nested as the miscoverage level changes.

Minimum length and minimum ordinal harm are different objectives

A mode-grown method asks which neighboring labels can be added while keeping the interval short. RPS asks which expansion best follows the cumulative probability mass around the median. Those rules coincide in some distributions and diverge when substantial probability sits away from the mode.

The paper’s illustrative multimodal example makes the divergence concrete. It uses class probabilities $(0.09,0.12,0.40,0.04,0.35)$ over five ordered labels. The median and mode are both class 3, so the comparison is not driven by different starting classes. The purpose of the example is theoretical mechanism demonstration, not benchmark evidence.

Risk here is the expected distance from the true label to the nearest label inside the set. The mode-grown interval remains narrow on the wrong side of a large probability mass at class 5. RPS expands toward that mass and reduces expected ordinal distance without changing cardinality.

The authors formalize this intuition under a stronger setting: if the true conditional class distribution is known and conditional coverage is available, the RPS-grown interval minimizes expected set-based $\ell_1$ ordinal distance among minimal-cardinality contiguous intervals. This theorem explains what the score is optimizing. It does not upgrade the practical split-conformal procedure to distribution-free conditional coverage.

The benchmarks support a trade-off, not universal dominance

The main empirical evidence compares RPS with nominal conformal methods and ordinal alternatives across three image datasets and ten tabular datasets. The image studies use ResNet-18 models on BACH, RetinaMNIST, and FGNet. The principal tabular studies use a one-hidden-layer, 64-unit MLP, with roughly 60% of each dataset used for training and 20% each for calibration and testing. Reported neural-network results average 50 random calibration-test splits.

The comparison includes LAC, APS, COPOC variants, min-CPS, and OCDF. Evaluation is broader than coverage and set size: it includes contiguity violations, interval width, mean and worst absolute miss magnitude, mean absolute interval error, and average interval score loss.

The recurring pattern is bounded but operationally relevant:

An appendix repeats selected comparisons with LightGBM on six tabular datasets. Its likely purpose is robustness to a different base-model class, not a second empirical thesis. The qualitative pattern persists, supporting the claim that the conformal construction is model-agnostic. The tabular MLP experiments intentionally omit hyperparameter tuning, so the study isolates the uncertainty method more than it demonstrates best-possible predictive performance.

Linear construction removes a practical objection

A naive implementation would recompute the full RPS for every candidate class, costing $O(K^2)$ per prediction. The adjacent-label recurrence reuses the previous score, reducing prediction-set construction to $O(K)$ once cumulative probabilities are available. Calibration costs $O(nK)$ for $n$ calibration examples.

For systems with many ordered categories, this matters because uncertainty calibration should not introduce a separate threshold search or repeated full-score calculation for every label. The computational result is an implementation contribution: it makes the score geometry usable at inference time. It does not address the larger operational costs of maintaining representative calibration data.

Where the method changes an operational decision

For a medical triage product, the affected user is the clinician or review queue manager deciding whether a case should be escalated. A contiguous severity interval preserves intermediate diagnoses and makes a distant uncovered severe class visible in evaluation. The boundary is substantial: the benchmarks do not establish prospective clinical utility, subgroup safety, or outcome improvement.

For credit or operational risk, the affected user is the analyst assigning a rating band or review priority. RPS is relevant when a four-notch miss is materially worse than a one-notch miss and when the existing model already outputs class probabilities. Risk governance should then monitor coverage, interval width, and ordinal miss distance together. Optimizing only set size would encode the wrong loss function.

For quality control and prioritization, the method offers a post-hoc layer rather than an architecture replacement. That lowers integration friction, but only under three conditions: the labels have a meaningful order, calibration cases resemble future cases, and the organization can defend how distance along the rank scale maps to operational harm.

The deployment boundary is narrower than the theorem headline

Three assumptions require explicit validation before production use.

First, split conformal coverage depends on exchangeability between calibration and test examples. A changing patient mix, lending regime, sensor process, or customer population can invalidate the calibration relationship. Monitoring drift is therefore part of the method, not an optional governance add-on.

Second, the set-based $\ell_1$ risk encodes classes as consecutive integers. It treats every one-rank movement as equally costly. Many ordinal systems do not have equal practical spacing: the jump from “moderate” to “severe” may matter more than the jump from “low” to “moderate.” A production team should validate a domain-specific loss function before treating the paper’s ordinal-risk result as its business objective.

Third, the practical guarantee remains marginal. A method can achieve its overall target while under-covering a subgroup or a difficult region of the feature space. The paper also does not separate aleatoric uncertainty from epistemic uncertainty. Operators therefore receive a calibrated range, not a diagnosis of why the model is uncertain.

A better uncertainty set is defined by the decision it supports

The paper’s strongest contribution is to separate three properties that are often bundled together: statistical coverage, coherent interval geometry, and the severity of an uncovered outcome. RPS links them through cumulative probability and a median-centered score sequence, then makes the construction computationally linear.

Its benchmark case is persuasive within scope. RPS is not the automatic choice when the only objective is minimum set size. It becomes more attractive when the operator needs an ordered range and distant misses carry greater cost than adjacent ones. The production decision should therefore begin with the loss attached to the rank scale, followed by calibration monitoring—not with a leaderboard of average set cardinality.

Cognaptus: Automate the Present, Incubate the Future.