Important, but Not Direct: When Time-Series Attribution Misstates Model Dependencies

TL;DR for operators

A forecasting dashboard can correctly report that an earlier observation influenced a prediction and still give the wrong impression about how that influence enters the model.

Amadeo Tunyi’s paper, The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations, argues that familiar scalar attribution methods cannot in general recover the model’s direct temporal dependency structure.¹ Marginal methods can assign importance to an earlier variable whose influence is entirely mediated through a later, autocorrelated observation. Gradient methods can report sensitivity that exists only outside the support of the data the model actually sees.

For model-risk and forecasting teams, the operational rule is narrower than “stop using SHAP” or “stop using gradients.” A large attribution score is evidence of predictive influence under a particular attribution computation; it is not automatically evidence that the variable is a direct driver of the forecast.

The paper’s proposed alternative is to evaluate explanations structurally: preserve source variable, target variable, and lag; keep model queries on the data manifold; and test candidate dependencies conditionally rather than marginally. That is a stronger audit target, but not yet a turnkey replacement. The paper proves failure modes and demonstrates them on controlled synthetic cases; it does not provide a universally DAG-faithful method validated across real forecasting deployments.

A high score does not tell you whether the dependency is direct

Suppose a forecasting system highlights an observation from two periods ago. The score is large. An operator now has two different questions available:

1. Would changing that historical observation tend to change the prediction?
2. Does the model depend on that observation directly, after accounting for the other relevant inputs?

Standard attribution tools are usually much closer to answering the first question.

The paper focuses on the second. Its central formal result is that flat marginal or gradient-based attribution computations cannot, in general, satisfy both requirements needed for structurally faithful temporal explanations: they must evaluate influence on the support of the reference data distribution, and they must separate direct dependence from effects transmitted through other variables.

This matters most when attribution is consumed downstream as an explanation of model structure. A predictive contribution can be genuine while the inferred direct relationship is wrong.

The paper formalizes the distinction using a model dependency DAG. An edge from a historical input to a forecast target exists when the two remain conditionally dependent after accounting for the remaining inputs and forecast outputs. The target is therefore not the true causal graph of the world. It is the conditional-dependency structure expressed by the trained forecasting model under a reference distribution.

An explanation is DAG-faithful when the graph it induces represents the same conditional independencies as that model dependency graph, generally up to Markov equivalence.

That is a stricter standard than asking whether masking an important feature moves the prediction.

Autocorrelation can turn mediated influence into a false direct edge

The first controlled construction isolates a familiar time-series problem: an earlier observation is correlated with a later observation, and the later observation is what the forecast actually uses.

A marginal attribution score that conditions on the earlier observation alone can still move the expected forecast. The paper writes the general form as

That score can be positive even when there is no direct edge from the earlier input to the forecast. The earlier observation changes what one expects about the mediator; the mediator changes the forecast. Marginal influence therefore survives even though direct conditional dependence does not.

The experiment is best read as a controlled demonstration of the formal failure mechanism, not as a general benchmark ranking attribution products.

The required structural pattern is zero under $f_1$, positive under $f_2$, and zero under $f_3$.

KernelSHAP and TimeSHAP distinguish the mediated case from the true direct edge in the first construction, but they later assign nonzero influence under $f_3$. ShapTime and TS-MULE already produce spurious attribution under the mediated case.

Section 5 generalizes the issue theoretically: SHAP is not DAG-faithful in general because coalition contributions can remain nonzero for an autocorrelated ancestor even when the corresponding direct edge is absent.

The relevant distinction is therefore predictive contribution versus direct conditional dependency. SHAP can remain informative for the former while failing as evidence for the latter.

Gradients solve a different problem and introduce another failure mode

One might expect gradient-based attribution to avoid the mediation problem because gradients inspect the model rather than marginal associations in the data.

The second construction shows why that is insufficient.

The paper creates models that behave identically on the support of the reference distribution but differ away from it. From the perspective of observed, on-manifold data, the models express the same relevant behavior. Yet their ambient-space derivatives can differ.

The reported gradient scores make the consequence visible:

Every tested gradient method gives the desired positive score under the genuine direct-dependency model $f_2$. Every one also reports a positive score under $f_3$, even though $f_3$ is identical to $f_1$ on the support of the reference distribution.

This is an on-manifold robustness test of the explanation computation. It does not show that gradients are generally useless. It shows that ordinary partial derivatives can encode sensitivity in directions the observed data never occupy. If the audit question is “what dependency does this model express on the data distribution?”, such sensitivity can be structurally misleading.

For high-stakes forecasting systems, this creates a second validation requirement: explanation methods should not gain explanatory authority merely because the model has arbitrary derivatives in unsupported regions of input space.

Temporally aware perturbations reduce false edges but can miss true ones

The paper also evaluates WinIT, Dynamask, and FIT because their perturbation strategies are closer to the proposed requirements. This comparison is an assessment of promising existing starting points, not evidence that any of them already solves DAG-faithful explanation.

Their scores are:

All three avoid the false positive under the mediated $f_1$ case and the off-manifold $f_3$ case. But WinIT and FIT miss the genuine direct edge under $f_2$, while Dynamask detects it only weakly.

That changes the engineering objective. Eliminating false edges is not enough; a production explanation layer also needs enough sensitivity to recover dependencies that are genuinely present.

The paper therefore does not crown a winning attribution method. It uses the comparison to identify the computational properties a stronger method would need.

The proposed replacement is structural before it is algorithmic

Once the explanation target becomes dependency structure, a single scalar per feature-time coordinate loses information that matters: which source variable affects which target variable, and at what lag.

The paper proposes a transition tensor indexed by source variable, target variable, and lag. Instance-level tensors are aggregated over the reference distribution:

The tensor is the proposed structural container; it is not itself the missing solution. The influence functional used to populate its entries still has to satisfy the paper’s admissibility requirements.

Evaluation also becomes edge-local. Rather than first reconstructing the entire model dependency graph, candidate edges can be checked through conditional-independence tests. On synthetic benchmarks, explanation graphs can be compared against known structure using measures such as structural Hamming distance. Where real-data ground truth is unavailable, the paper proposes interventional consistency as another validation route.

For model governance, this suggests a useful architectural split: maintain a richer internal representation for structural audit, then project it into a simpler interface for operators. Human readability does not require the audit representation itself to collapse into one score.

What changes for forecasting and model-risk teams

Paper evidence: scalar marginal and gradient attribution can generate structurally incorrect explanations under explicit constructions involving autocorrelation and off-manifold behavior. Temporally aware perturbation methods avoid some false positives but can lose sensitivity to a true edge.

Cognaptus inference: when an explanation will influence an intervention—changing a monitored variable, escalating a risk factor, adjusting a control, or documenting a model dependency—teams need an additional validation layer between attribution and action. That layer should test whether the alleged driver survives conditioning on relevant temporal context and whether the explanation depends on unsupported perturbations.

This is especially relevant when attribution output is being interpreted as a statement of mechanism rather than merely a diagnostic of prediction sensitivity.

The business decision affected is therefore not primarily which visualization to show. It is what evidentiary status to assign to an attribution score before someone acts on it.

The framework is stronger as an audit target than as a finished product

The paper’s formal negative results are strong: constructive counterexamples and proofs establish that the identified method families cannot guarantee DAG-faithfulness in general.

The positive side remains incomplete. No influence functional is provided that is proven universally admissible and DAG-faithful. The empirical demonstrations use small controlled synthetic constructions rather than broad real-world forecasting benchmarks. Multi-step forecast dependencies, distribution-shift robustness, exact identifiability, and practical visualization of high-dimensional transition tensors remain open.

Conditional separation can also require high-dimensional conditioning sets, creating computational and sample-complexity costs. The paper suggests reductions involving Markov blankets, discretization, or lower-order surrogates, but these are directions rather than established solutions.

So the immediate operational contribution is not a replacement library. It is a harder test for explanation claims.

A forecasting explanation can say that a past observation matters. Before converting that score into a claim about what the model directly depends on, the next question should be structural: does the dependency remain once the temporal pathway and the data manifold are respected?

Cognaptus: Automate the Present, Incubate the Future.