TL;DR for operators

Removing features from a production recommendation model is a release decision, not simply a feature-ranking exercise. Pinterest’s FORTE framework1 makes feature retirement repeatable through daily importance monitoring, joint feature testing, retraining, online validation, and dependency tracking.

The evidence explains why these stages matter. In Search Relevance, removing 59 features passed offline evaluation but failed a secondary online guardrail; removing 48 succeeded. In a separate interaction experiment, individually favorable feature changes summed to a +6.554% Closeup AUC effect, but their jointly measured effect was only +3.084%.

FORTE also addresses the cost of running these checks. Its pooled evaluation system was 54 times faster than sequential evaluation in the largest reported benchmark. Yet infrastructure returns depended heavily on architecture: removing 48 Search Relevance features reduced trainable parameters by 2.3%, while removing eight Pin Ranker features reduced them by 29.9% and lowered inference cost by approximately 10%.

For operators, the decision is which feature sets can be retired without detected production regressions, at what validation cost, and with which measurable infrastructure savings. Offline importance provides candidates, not final authorization.

The largest feature cut passed offline—and still failed

Production recommendation systems tend to accumulate inputs. A new feature can remain in service after its original value has diminished, while removing it requires retraining, testing, and coordination across data pipelines. Teams facing infrastructure pressure must therefore decide which inputs can disappear without damaging the recommendations users receive.

Pinterest’s Search Relevance system provides a revealing example. Researchers evaluated three progressively larger removal sets: 32, 48, and 59 features, drawn from a model using 111 inputs. All three passed offline evaluation and retraining gates. The 48-feature treatment was launched, while the 59-feature treatment was rejected after failing a secondary online guardrail (Section 7.1, Table 1).

That distinction changes what feature evaluation must accomplish. A candidate may look acceptable against offline predictions yet produce an unacceptable result under production traffic. The evaluation process must narrow a large feature inventory into removal candidates, identify dangerous combinations, and reserve costly online experiments for candidates that have survived earlier checks.

FORTE organizes these decisions into a staged production workflow.

Why individual feature importance cannot authorize removal

The first stage asks whether the current model actually relies on an input.

A common test is to disturb one feature’s values while holding the trained model and evaluation examples fixed, then measure how predictions change. This is permutation feature importance. FORTE also supports ablation, which replaces feature values using configured removal semantics.

Its signed, normalized metric-change score is:

$$ \Delta_M(G)=100\,s_M\frac{M_G-M_0}{M_0} $$

Here, $M_0$ is baseline performance, $M_G$ is performance after disturbing feature group $G$, and $s_M$ adjusts for whether higher or lower metric values are preferable. Under this convention, a negative score indicates model reliance; a near-zero or positive score makes the feature a possible removal candidate.

One measurement may still be misleading. FORTE tracks reliance across daily evaluations, filtering candidates by low average reliance, consistently low ranking, and non-increasing reliance trends across required prediction heads. Its scheduling system resolves the deployed checkpoint and accounts for delayed evaluation partitions.

The checkpoint distinction matters: changing importance across different models need not represent a change in the underlying data.

More fundamentally, two inputs can substitute for one another. Disturbing either separately may leave predictions largely intact, while disturbing both removes information the model needs.

FORTE therefore constructs nested candidate sets and tests them jointly.

A grouped-perturbation experiment involving 189 feature groups and 1.5 million matched examples exposed substantial non-additivity (Section 6.2):

  • Summed individual Closeup AUC changes: +6.554%
  • Measured joint Closeup AUC change: +3.084%
  • Negative incremental contributions along a fixed candidate order: 68 of 176 nonzero contributions

The negative increments arose even though those groups had passed individual screening. Adding a group to an already disturbed coalition could harm performance despite its apparently favorable individual result.

The diagnostic also showed disagreement across prediction objectives. In an order-averaged analysis, the full coalition produced +3.079% Closeup AUC but -3.769% Clickthrough PR-AUC. A removal decision based only on Closeup AUC would overlook the deterioration in a different metric.

These are frozen-model perturbation findings, not measured effects of deployed removals. They establish why candidate combinations and prediction heads need separate checks before retraining. The analysis of 20 randomized coalition orders further demonstrated context dependence, but did not provide precise Shapley-value estimates for all 189 groups.

Pooled evaluation makes recurring screening feasible

Joint testing creates a computational problem. If hundreds of features require separate model-loading, data-loading, and inference passes, even preliminary screening can consume substantial resources.

FORTE reduces duplicated work through pooled perturbation. Rather than repeatedly loading the same checkpoint and materializing the same examples, its execution system shares these operations across experiments, constructs perturbed feature dictionaries, and uses common distributed inference infrastructure.

The main scalability benchmark in Section 6.1, Figure 5, compared pooled execution with the legacy sequential implementation under the same compute allocation.

For 300 experiments on three million evaluation rows, pooled execution reduced wall-clock runtime 54-fold. At that scale, the marginal cost of another pooled experiment was approximately 41 seconds, compared with about 45 minutes for another separate evaluation pass.

This improvement makes daily screening more plausible without claiming that every model or workload will achieve the same acceleration.

The researchers also examined whether every feature requires repeated random permutations. This was a stability and evaluation-budget test, separate from the execution-speed benchmark.

Across 381 feature groups, ten permutations, and two evaluated sample sizes, the larger 150,000-example setting produced median leave-one-out feature-rank correlations of 0.907–0.959 and median top-20 overlap of 92–100%. At equal row-evaluation budgets, using more distinct examples rather than more repetitions improved ranking correlation against the larger-sample reference.

A supporting mathematical result establishes an $O(1/n)$ upper bound on permutation-induced ROC-AUC variance under fixed model and data conditions, non-vanishing class proportions, and row-wise inference. It does not establish the same rate for PR-AUC or architectures in which one example changes another example’s inference output.

The engineering conclusion is appropriately narrow: for feature ranking at the evaluated scale, one permutation can be an economical screening choice. It does not guarantee precise importance values or safe feature removal.

Feature count is a poor predictor of infrastructure savings

The two production deployments show how differently successful retirement can affect model economics.

Production system Launched removal Trainable parameters Serving throughput Inference cost
Search Relevance 48 of 111 features -2.3% +2.0% Not separately quantified
Pin Ranker 8 features -29.9% +19.8–25.5% Approximately -10%

Sources: Section 7.1, Table 1; Section 7.2, Table 2. Pin Ranker throughput was measured using a CUDA Graph benchmark.

Search Relevance eliminated approximately 43.2% of its inputs, but the remaining architecture still dominated parameter count. The launched treatment passed reported online guardrails, with neutral primary relevance results across US and non-US traffic segments. Its reported page wait time decreased by 0.1696% ($p=0.0012$).

Pin Ranker produced a substantially larger efficiency gain from far fewer removals. Both its two-feature and eight-feature candidates passed retraining and three-week randomized online validation, without statistically significant regressions in the reported clickthrough, closeup, or repin metrics. The eight-feature treatment launched.

The paper reports annual infrastructure savings of hundreds of thousands of dollars for Pin Ranker. That figure is a deployment outcome, not a general estimate of the return from removing eight features.

One additional step determines whether savings reach beyond model inference: feature lineage.

Removing a feature from one model does not necessarily stop its collection, storage, or materialization. Other models may still consume it. FORTE tracks these dependencies before retiring shared infrastructure.

For the eight Pin Ranker features, lineage checks found no remaining model consumers. The team could therefore remove corresponding dataset columns and decommission feature-materialization jobs.

This is the difference between reducing a model’s input list and retiring the production infrastructure associated with those inputs.

What production teams can adopt—and what remains uncertain

Cognaptus interpretation: For organizations operating large, frequently updated recommendation systems, feature retirement is best managed as recurring cost governance with explicit quality gates.

An operator can use temporal screening to shortlist candidates, pooled individual and grouped perturbation to prioritize retraining, randomized online experiments to enforce release guardrails, and lineage checks to determine which upstream costs can actually disappear. Potential savings should be estimated from feature-specific computation and architectural dependencies rather than input counts.

The evidence also sets boundaries on that recommendation.

Unconditional permutation can create feature combinations absent from real traffic. Manually constructed groups may miss dependencies. Changing model checkpoints complicates temporal comparisons. These problems limit what the offline measurements reveal, even when the execution infrastructure is efficient.

Online non-significance is not proof of exact equivalence. Search Relevance serves more than 45 languages, while its reported segmented results distinguish US from non-US traffic. Those comparisons cannot establish neutrality within every language or user subgroup.

Finally, two proprietary production systems cannot establish architecture-independent returns. Complete infrastructure cost accounting is not publicly reproducible, and energy consumption or emissions reductions were not directly measured.

These limitations do not invalidate the deployments. They determine which claims an operator can responsibly carry into another production environment.

The retirement decision belongs at the end of the workflow

FORTE’s contribution is to make repeated feature evaluation economically feasible while keeping the final removal decision tied to deployed behavior and actual infrastructure dependencies.

The rejected 59-feature Search Relevance treatment illustrates why the strongest offline result is not necessarily the right launch. The Pin Ranker deployment shows how a smaller feature cut can yield much greater computational savings when it removes expensive architectural and data-pipeline dependencies.

For production ML teams, the measure of successful feature retirement is therefore not how many inputs disappear. It is whether the remaining model passes the required quality gates, and whether the supporting infrastructure becomes measurably cheaper to operate.

Cognaptus: Automate the Present, Incubate the Future.


  1. Vishnu Arun and Adam Obeng and Tianyou Wang and Zilin Ren and Mukuntha Narayanan Sundararaman and Han Wang and Shivin Thukral and Zhixiang Wang and Saurabh Vishwas Joshi and Darren Reger (2026). FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems. arXiv:2610.13806. https://arxiv.org/abs/2610.13806 ↩︎