Cover image

Explain Before You Explore: What YouTube Music’s LLM Experiment Changed

TL;DR for operators YouTube Music tested an LLM-enhanced discovery experience and reported a 22.43% lift in shelf-level engagement and an 8.07% lift in its discovery-retention metric. The tempting explanation is that the LLM simply found better unfamiliar artists. The component tests point elsewhere: withholding the new nomination source alone had a neutral engagement impact, while removing personalized natural-language rationales produced regressions resembling removal of the complete LLM system. ...

October 8, 2026 · 7 min · Zelina
Cover image

Same Causal Effect, Different Bill: Derivation Graphs and the Estimand Trap

The formula is not a clerical detail A business asks a causal question: What happens if we change X? The analytics team returns a formula. Everyone relaxes. The effect is identifiable, the notation looks official, and a graph somewhere has probably been blessed by someone with a PhD. Excellent. Time to move to dashboards. ...

June 13, 2026 · 14 min · Zelina
Cover image

Personas, Panels, and the Illusion of Free A/B Tests

A/B tests are expensive in the least glamorous way. Not because the math is hard. Not because a conversion metric is philosophically mysterious. The real cost is coordination: product approval, legal review, user-risk arguments, instrumentation, waiting for enough traffic, and then explaining to someone why the “obvious winner” was not statistically obvious at all. ...

December 25, 2025 · 16 min · Zelina