Counterfactuals That Survive the Next Model

TL;DR for operators

A counterfactual recommendation can be valid for the model running today and fail after routine retraining. The paper behind AVCG addresses that problem by generating counterfactuals against a distribution of plausible predictors, rather than optimizing each recommendation for one fitted model.¹

The framework has two tested versions. AVCG-B represents predictive uncertainty through a Bayesian-style distribution approximated with Monte Carlo dropout. AVCG-R narrows attention to a set of models whose validation losses remain within a chosen tolerance of the best model. Across four tabular benchmarks, both variants generally report near-perfect or perfect validity when counterfactuals are checked against separately trained models and against the constructed set of near-optimal models.

The operational attraction is not robustness alone. AVCG moves much of the computation into training, then generates counterfactuals with a single forward pass at roughly $10^{-3}$ seconds per instance in the main experiments. Latent sampling also produces multiple candidate counterfactuals rather than one deterministic answer.

For teams exposing recourse or decision-support recommendations, the practical implication is to treat model-change robustness as a property to design and test, not something implied by validity against the current model. The boundary matters: the experiments use benchmark datasets and finite dropout-derived model sets. They do not establish protection against arbitrary retraining, architectural changes, or distribution shift.

A recommendation can expire when the model changes

Suppose a decision-support system tells a user which input changes would reverse an unfavorable prediction. The recommendation works against today’s model. A month later, the organization retrains the system on updated data.

The new model may have similar aggregate performance yet disagree on that particular case. The user’s earlier recommendation can therefore stop producing the promised outcome even though neither model is obviously defective.

This is the problem AVCG treats as part of counterfactual generation itself. Instead of asking only, “What small change flips this predictor?”, it asks for changes that remain effective across multiple plausible predictors.

That distinction becomes operationally relevant whenever model replacement is routine or when several models perform nearly equally well but differ locally. In such settings, single-model validity says less than it first appears to say.

AVCG optimizes over plausible predictors, not one decision boundary

The central abstraction is a hypothesis distribution: a distribution over predictors considered plausible for the task.

AVCG trains a conditional latent-variable generator $g_\phi(x,y’,z)$ and an encoder $q_\psi(z\mid y’,x)$. Its objective balances three elements:

The first term rewards counterfactuals that obtain high target-class probability on average across the chosen model distribution. The remaining terms regularize the latent representation and penalize counterfactuals that move too far from the original input.

That formulation is the paper’s main contribution. The uncertainty representation can change without requiring an entirely different counterfactual objective.

AVCG-B uses a Bayesian-style posterior approximated through stochastic dropout masks. AVCG-R instead focuses on a Rashomon set: models whose validation loss remains sufficiently close to the best observed value,

The distinction is meaningful. Bayesian uncertainty asks which parameter settings receive posterior support. The Rashomon construction asks which models remain acceptably good under an explicit performance tolerance. Both become distributions over predictors that the generator can train against.

The robustness metrics test the failure mode directly

Ordinary counterfactual validity checks whether the generated input flips the model used to create it. The paper adds measurements that expose what happens when that model is no longer the only reference point.

Cross Model Validity (CMV) measures the fraction of separately trained surrogate models that still assign the desired class to a counterfactual. Rashomon Validity Ratio (RVR) measures the fraction of models inside the empirical Rashomon set that accept it.

These are main evaluation measures, not secondary diagnostics. They correspond directly to the motivating failure mode: a recommendation that works against one predictor but not against another plausible one.

Across Adult Income, Breast Cancer, Heart Disease, and Spambase, AVCG-B and AVCG-R are generally at or near 1.0 on CMV and RVR in the reported main settings. Several post-hoc baselines fall substantially below that level.

Adult Income illustrates the contrast. Under the strict $\epsilon=0$ setting, AVCG-B and AVCG-R both report CMV and RVR of $1.000$. Batten et al. report CMV of $0.749$ and RVR of $0.675$; QUCE reports $0.883$ and $0.923$ respectively. On Spambase, AVCG again reports $1.000$ for both robustness measures, while the compared methods vary considerably more.

The repeated pattern supports a specific claim: optimizing the counterfactual across a modeled distribution of predictors can improve its survival across model variation represented by that distribution.

Amortization trades training work for low-latency use

Robustness across many models could be impractical if every user request required a new iterative optimization.

AVCG avoids that runtime pattern through amortized inference. Rather than solving a fresh optimization problem for each new case, training teaches a generator to produce counterfactuals directly. At use time, generation becomes a forward pass through the trained model.

The main tabular experiments report inference times around $0.001$ seconds per instance for the AVCG variants. The compared iterative methods range from roughly hundredths of a second to substantially longer in several settings.

The latent representation also permits sampling several counterfactuals. The main deterministic baselines report zero diversity in the tabular tables, while AVCG reports positive diversity across datasets.

For an interactive product, these two properties reinforce each other. Distribution-aware generation does not necessarily have to impose distribution-aware optimization latency on every user interaction. The cost is shifted toward training, including the work required to sample or construct the hypothesis distribution.

The image results expose the boundary more clearly

The supplementary MNIST and CIFAR-10 experiments are better interpreted as an extension test than as a repetition of the tabular headline.

AVCG retains high ordinary validity and RVR, positive diversity, and millisecond-scale inference. Cross Model Validity, however, weakens in several image settings. On CIFAR-10 under the strict setting, AVCG-B reports CMV of $0.774$, while AVCG-R reports $0.324$. On MNIST, the corresponding values are $0.891$ and $0.482$.

This matters because RVR and CMV are testing different populations of models. High RVR means the counterfactual survives the empirical Rashomon set constructed by the method. Lower CMV means transfer to separately trained models can still be weaker.

The supplementary evidence therefore sharpens rather than invalidates the main result: robustness is tied to how the plausible-model space is defined and approximated.

For deployed recourse, define the model space before promising robustness

Cognaptus infers a governance use from the framework: organizations that expose counterfactual recommendations can define an approved space of plausible model variants and test recommendations against that space before deployment.

That changes the relevant acceptance criterion. A counterfactual feature would no longer pass solely because recommendations flip the production model. It could also be required to meet CMV- or RVR-like thresholds across retrained or near-equivalent models considered operationally plausible.

This is especially relevant when retraining is scheduled, multiple candidate models regularly clear the same validation bar, or a user may act on advice over a longer period than the current model remains in production.

The paper does not establish that such a policy will work unchanged in a deployed recourse system. Its empirical evidence comes from standard benchmark datasets, with 100 evaluated instances per dataset over five seeds, and the Rashomon sets are finite approximations built from retained dropout masks.

Robust within a modeled hypothesis space is not robust to everything

The theoretical results carry the same boundary.

For fixed encoder and generator parameters, the paper shows that changes in the AVCG objective can be bounded by the square root of the KL divergence between two hypothesis distributions, provided the relevant target-class log probabilities are bounded. It also derives a probability bound connecting high expected target-class confidence to the probability that a sampled model exceeds a selected confidence threshold.

These results formalize stability relative to specified model distributions and assumptions. They do not imply that a generated counterfactual will survive arbitrary retraining, an unmodeled architecture, severe data drift, or a changed decision policy.

The empirical construction has another dependency: AVCG-R assumes that validation loss is a suitable criterion for deciding which models belong in the near-optimal set, and that retained dropout masks adequately approximate that space.

Those are design choices, not universal facts.

Counterfactual robustness becomes a model-risk question

The paper’s broader contribution is to move counterfactual explanations away from treating the current predictor as the only model that matters.

Once multiple plausible predictors are admitted, robustness becomes something that can be represented, optimized, and measured. AVCG provides one generalized objective, two concrete constructions, and metrics that expose whether recommendations persist across model changes.

For operators, the practical lesson is bounded but consequential: if users are expected to act on model-generated counterfactuals, validate those actions against the set of model changes the organization actually considers plausible.

A counterfactual that survives that test is stronger evidence than one that merely defeats today’s decision boundary. It is still evidence about a defined hypothesis space, not a promise about every model that could come next.

Cognaptus: Automate the Present, Incubate the Future.