Forty-Eight Features Gone, Fifty-Nine Too Far
TL;DR for operators Removing features from a production recommendation model is a release decision, not simply a feature-ranking exercise. Pinterest’s FORTE framework1 makes feature retirement repeatable through daily importance monitoring, joint feature testing, retraining, online validation, and dependency tracking. The evidence explains why these stages matter. In Search Relevance, removing 59 features passed offline evaluation but failed a secondary online guardrail; removing 48 succeeded. In a separate interaction experiment, individually favorable feature changes summed to a +6.554% Closeup AUC effect, but their jointly measured effect was only +3.084%. ...