Opening Night vs. Dress Rehearsal

Podcast: Clown Cast · Joey Musselman

Opublikowano: 2026-10-15

A team from EPFL discovered something every recommender builder should fear: the model that performed best in offline testing was the worst performer live. This episode breaks down why offline metrics deceive, and covers three evaluation approaches—offline metrics, live A/B tests, and off-policy evaluation—that reveal whether your system actually works. 00:00:00 - The Paradox: Best Offline, Worst Live 00:01:47 - Starting with Offline Metrics 00:07:10 - How Offline Evaluation Fails 00:15:35 - Three Evaluation Approaches 00:17:20 - Why Procedure Matters --- Sources & further reading: • Garcin, Faltings, Donatsch, Alazzawi, Bruttin, Huber, Offline and Online Evaluation of News Recommender Systems at swissinfo.ch, RecSys 2014 — — offline/on...