Predictive Models on Random Data

This week is an insightful discussion with Claudia Perlich about some situations in machine learning where models can be built, perhaps by well-intentioned practitioners, to appear to be highly predictive despite being trained on random data. Our discussion covers some novel observations about ROC and AUC, as well as an informative discussion of leakage. Much of our discussion is inspired by two excellent papers Claudia authored: Leakage in Data Mining: Formulation, Detection, and Avoidance and On Cross Validation and Stacking: Building Seemingly Predictive Models on Random Data.

Guest

Claudia Perlich: I started my career in Data Science at the IBM T.J. Watson Research Center, concentrating on research in data analytics and machine learning for complex real-world domains and applications. I tend to be domain agnostic having worked on almost anything from Twitter, DNA, server logs, CRM data, web usage, breast cancer, movie rating and many more. I am still actively publishing and have so far over 50 scientific publications to my name as well as a few patents in the area of machine learning. My claim to fame was winning data mining competitions, as well as several awards from industry and academia. I received my Ph.D. in Information Systems from Stern School of Business, New York University in 2005 and hold a Master of Computer Science from Colorado University. Besides my day job I like teaching and am available for panels, presentations, meetups, and debates around big (and small) data.