<p>This episode is a discussion of data visualization and a proposed New Year's resolution for Data Skeptic listeners. Let's kill the word cloud.</p>

This episode is a discussion of data visualization and a proposed New Year's resolution for Data Skeptic listeners. Let's kill the word cloud.

Let's Kill the Word Cloud

<p>Today's mini episode discusses the widely known optimization algorithm gradient descent in the context of hiking in a foggy hillside.</p>

Today's mini episode discusses the widely known optimization algorithm gradient descent in the context of hiking in a foggy hillside.

[MINI] Gradient Descent

<p>A recent paper in the journal of Judgment and Decision Making titled <a href="http://journal.sjdm.org/15/15923a/jdm15923a.pdf">On the reception and detection of pseudo-profound bullshit</a> explores empirical questions around a reader's ability to detect statements which may sound profound but are actually a collection of buzzwords that fail to contain adequate meaning or truth. These statements are definitively different from lies and nonesense, as we discuss in the episode.</p> <p>This paper proposes the Bullshit Receptivity scale (BSR) and empirically demonstrates that it correlates with existing metrics like the Cognitive Reflection Test, building confidence that this can be a useful, repeatable, empirical measure of a person's ability to detect pseudo-profound statements as being different from genuinely profound statements. Additionally, the correlative results provide some insight into possible root causes for why individuals might find great profundity in these statements based on other beliefs or cognitive measures.</p> <p>The paper's lead author <a href= "https://uwaterloo.ca/psychology/people-profiles/gordon-pennycook">Gordon Pennycook</a> joins me to discuss this study's results.</p> <p>If you'd like some examples of pseudo-profound bullshit, you can randomly generate some based on <a href= "http://www.wisdomofchopra.com/">Deepak Chopra</a>'s twitter feed.</p> <p>To read other work from Gordon, check out his <a href= "https://scholar.google.ca/citations?user=AIbJenwAAAAJ&hl=en&oi=ao"> Google Scholar</a> page and find him on twitter via <a href= "https://twitter.com/gordpennycook">@GordonPennycook</a>.</p> <p>And just for fun, if you think you've dreamed up a Data Skeptic related pseudo-profound bullshit statement, tweet it with hashtag <a href= "https://twitter.com/search?q=%23pseudoprofound&src=typd">#pseudoprofound</a>. If I see an especially clever or humorous one, I might want to send you a free <a href="http://dataskeptic.com/store.php">Data Skeptic sticker</a>.</p>

<div id="bbody"> <p>A recent paper in the journal of Judgment and Decision Making titled <a href="http://journal.sjdm.org/15/15923a/jdm15923a.pdf">On the reception and detection of pseudo-profound bullshit</a> explores empirical questions around a reader's ability to detect statements which may sound profound but are actually a collection of buzzwords that fail to contain adequate meaning or truth. These statements are definitively different from lies and nonesense, as we discuss in the episode.</p> <p>This paper proposes the Bullshit Receptivity scale (BSR) and empirically demonstrates that it correlates with existing metrics like the Cognitive Reflection Test, building confidence that this can be a useful, repeatable, empirical measure of a person's ability to detect pseudo-profound statements as being different from genuinely profound statements. Additionally, the correlative results provide some insight into possible root causes for why individuals might find great profundity in these statements based on other beliefs or cognitive measures.</p> <p>The paper's lead author <a href= "https://uwaterloo.ca/psychology/people-profiles/gordon-pennycook">Gordon Pennycook</a> joins me to discuss this study's results.</p> <p>If you'd like some examples of pseudo-profound bullshit, you can randomly generate some based on <a href= "http://www.wisdomofchopra.com/">Deepak Chopra</a>'s twitter feed.</p> <p>To read other work from Gordon, check out his <a href= "https://scholar.google.ca/citations?user=AIbJenwAAAAJ&hl=en&oi=ao"> Google Scholar</a> page and find him on twitter via <a href= "https://twitter.com/gordpennycook">@GordonPennycook</a>.</p> <p>And just for fun, if you think you've dreamed up a Data Skeptic related pseudo-profound bullshit statement, tweet it with hashtag <a href= "https://twitter.com/search?q=%23pseudoprofound&src=typd">#pseudoprofound</a>. If I see an especially clever or humorous one, I might want to send you a free <a href="http://dataskeptic.com/store.php">Data Skeptic sticker</a>.</p> </div> <div id="audiojs_wrapper0" class="audiojs error"> </div>

A recent paper in the journal of Judgment and Decision Making titled  explores empirical questions around a reader's ability to detect statements which may sound profound but are actually a collection of buzzwords that fail to contain adequate meaning...

Detecting Pseudo-profound BS

<p>Today's episode begins by asking how many left handed employees we should expect to be at a company before anyone should claim left handedness discrimination. If not lefties, let's consider eye color, hair color, favorite ska band, most recent grocery store used, and any number of characteristics could be studied to look for deviations from the norm in a company.</p> <p>When multiple comparisons are to be made simultaneous, one must account for this, and a common method for doing so is with the Bonferroni Correction. It is not, however, a sure fire procedure, and this episode wraps up with a bit of skepticism about it.</p>

Today's episode begins by asking how many left handed employees we should expect to be at a company before anyone should claim left handedness discrimination. If not lefties, let's consider eye color, hair color, favorite ska band, most recent grocery...

[MINI] The Bonferroni Correction

<p>Algorithms are pervasive in our society and make thousands of automated decisions on our behalf every day. The possibility of digital discrimination is a very real threat, and it is very plausible for discrimination to occur accidentally (i.e. outside the intent of the system designers and programmers). Christian Sandvig joins us in this episode to talk about his work and the concept of auditing algorithms.</p> <p><a href="http://niftyc.org/">Christian Sandvig</a> (<a href= "https://twitter.com/niftyc">@niftyc</a>) has a PhD in communications from Stanford and is currently an Associate Professor of Communication Studies and Information at the University of Michigan. His research studies the predictable and unpredictable effects that algorithms have on culture. His work exploring the topic of auditing algorithms has framed the conversation of how and why we might want to have oversight on the way algorithms effect our lives. His writing appears in numerous publications including <a href= "http://socialmediacollective.org/author/niftyc/">The Social Media Collective</a>, <a href= "http://www.huffingtonpost.com/christian-sandvig/">The Huffington Post</a>, and <a href= "http://www.wired.com/2015/05/facebook-not-fault-study/">Wired</a>.</p> <p>One of his papers we discussed in depth on this episode was <a href= "http://www-personal.umich.edu/~csandvig/research/Auditing%20Algorithms%20--%20Sandvig%20--%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf"> Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms</a>, which is well worth a read.</p>

Algorithms are pervasive in our society and make thousands of automated decisions on our behalf every day. The possibility of digital discrimination is a very real threat, and it is very plausible for discrimination to occur accidentally (i.e. outside...

Auditing Algorithms

<p>This episode reviews the concept of k-d trees: an efficient data structure for holding multidimensional objects. Kyle gives Linhda a dictionary and asks her to look up words as a way of introducing the concept of binary search. We actually spend most of the episode talking about binary search before getting into k-d trees, but this is a necessary prerequisite.</p>

This episode reviews the concept of k-d trees: an efficient data structure for holding multidimensional objects. Kyle gives Linhda a dictionary and asks her to look up words as a way of introducing the concept of binary search. We actually spend most...

[MINI] k-d trees

<p><a href="http://s.mehr.cz/">Samuel Mehr</a> joins us this week to share his perspective on why people are musical, where music comes from, and why it works the way it does. We discuss a number of empirical studies related to music and musical cognition, and dispense a few myths about music along the way.</p> <p>Some of Sam's work discussed in this episode include <a href= "https://drive.google.com/a/datascience.com/file/d/0B37emGJjrWcfR2dYZDVtRVlXOUk/view"> Music in the Home: New Evidence for an Intergenerational Link</a>,<a href= "https://drive.google.com/a/datascience.com/file/d/0B37emGJjrWcfS3F5NTZBNGhUTFU/view">Two randomized trials provide no consistent evidence for nonmusical cognitive benefits of brief preschool music enrichment</a>, and <a href= "http://journal.frontiersin.org/article/10.3389/fpsyg.2015.00988/full"> Miscommunication of science: music cognition research in the popular press</a>. Additional topics we discussed are also covered in a Harvard Gazette article featuring Sam titled <a href= "http://news.harvard.edu/gazette/story/2013/12/muting-the-mozart-effect/"> Muting the Mozart effect</a>.</p> <p>You can follow Sam on twitter via <a href= "https://twitter.com/samuelmehr">@samuelmehr</a>.</p>

joins us this week to share his perspective on why people are musical, where music comes from, and why it works the way it does. We discuss a number of empirical studies related to music and musical cognition, and dispense a few myths about music...

Scientific Studies of People's Relationship to Music

<p>This episode is a discussion of multiple regression: the use of observations that are a vector of values to predict a response variable. For this episode, we consider how features of a home such as the number of bedrooms, number of bathrooms, and square footage can predict the sale price.</p> <p>Unlike a typical episode of Data Skeptic, these show notes are not just supporting material, but are actually featured in the episode.</p>       <p>The site Redfin gratiously allows users to download a CSV of results they are viewing. Unfortunately, they limit this extract to 500 listings, but you can still use it to try the same approach on your own using the download link shown in the figure below.</p>

<div class="cell border-box-sizing text_cell rendered"> <div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>This episode is a discussion of multiple regression: the use of observations that are a vector of values to predict a response variable. For this episode, we consider how features of a home such as the number of bedrooms, number of bathrooms, and square footage can predict the sale price.</p> <p>Unlike a typical episode of Data Skeptic, these show notes are not just supporting material, but are actually featured in the episode.</p> </div> </div> </div> <div class="cell border-box-sizing text_cell rendered"> <div class="inner_cell"> <div class="text_cell_render border-box-sizing rendered_html"> <p>The site Redfin gratiously allows users to download a CSV of results they are viewing. Unfortunately, they limit this extract to 500 listings, but you can still use it to try the same approach on your own using the download link shown in the figure below.</p> </div> </div> </div>

This episode is a discussion of multiple regression: the use of observations that are a vector of values to predict a response variable. For this episode, we consider how features of a home such as the number of bedrooms, number of bathrooms, and...

[MINI] Multiple Regression

<p><a href="http://www.jesshamrick.com/">Jessica Hamrick</a> joins us this week to discuss her work studying mental simulation. Her research combines machine learning approaches iwth behavioral method from cognitive science to help explain how people reason and predict outcomes. Her recent paper <a href= "http://www.jesshamrick.com/publications/pdf/Hamrick2015-Think_again_the_amount_of_mental_simulation.pdf"> Think again? The amount of mental simulation tracks uncertainty in the outcome</a> is the focus of our conversation in this episode.</p> <p>Lastly, Kyle invited Samuel Hansen from the Relative Prime podcast to mention the <a href= "http://relprime.com/kickstarter">Relatively Prime Season 3</a> kickstarter, which needs your support now through Friday, March 11th, 2016.</p>

<div id="left"> </div> <div id="right"> <div id="header"> </div> <div id="main"> <div id="bbody"> <p><a href="http://www.jesshamrick.com/">Jessica Hamrick</a> joins us this week to discuss her work studying mental simulation. Her research combines machine learning approaches iwth behavioral method from cognitive science to help explain how people reason and predict outcomes. Her recent paper <a href= "http://www.jesshamrick.com/publications/pdf/Hamrick2015-Think_again_the_amount_of_mental_simulation.pdf"> Think again? The amount of mental simulation tracks uncertainty in the outcome</a> is the focus of our conversation in this episode.</p> <p>Lastly, Kyle invited Samuel Hansen from the Relative Prime podcast to mention the <a href= "http://relprime.com/kickstarter">Relatively Prime Season 3</a> kickstarter, which needs your support now through Friday, March 11th, 2016.</p> </div> </div> </div>

joins us this week to discuss her work studying mental simulation. Her research combines machine learning approaches iwth behavioral method from cognitive science to help explain how people reason and predict outcomes. Her recent...

Models of Mental Simulation

<p>How well does your model explain your data? R-squared is a useful statistic for answering this question. In this episode we explore how it applies to the problem of valuing a house. Aspects like the number of bedrooms go a long way in explaining why different houses have different prices. There's some amount of variance that can be explained by a model, and some amount that cannot be directly measured. R-squared is the ratio of the explained variance to the total variance. It's not a measure of accuracy, it's a measure of the power of one's model.</p>

How well does your model explain your data? R-squared is a useful statistic for answering this question. In this episode we explore how it applies to the problem of valuing a house. Aspects like the number of bedrooms go a long way in explaining why...

[MINI] R-squared

<p>Today on Data Skeptic, <a href= "https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/Lachlan_J._Gunn"> Lachlan Gunn</a> joins us to discuss his recent paper <a href= "http://arxiv.org/abs/1601.00900">Too Good to be True</a>. This paper highlights a somewhat paradoxical / counterintuitive fact about how unanimity is unexpected in cases where perfect measurements cannot be taken. With large enough data, some amount of error is expected.</p> <p>The "Too Good to be True" paper highlights three interesting examples which we discuss in the podcast. You can also watch a lecture from Lachlan on this topic via youtube <a href= "https://www.youtube.com/watch?v=Uz6xUjJHTII">here</a>.</p>

Today on Data Skeptic,  joins us to discuss his recent paper . This paper highlights a somewhat paradoxical / counterintuitive fact about how unanimity is unexpected in cases where perfect measurements cannot be taken. With large enough data, some...

Too Good to be True

<p>Certain data mining algorithms (including k-means clustering and k-nearest neighbors) require a user defined parameter k. A user of these algorithms is required to select this value, which raises the questions: what is the "best" value of k that one should select to solve their problem?</p> <p>This mini-episode explores the appropriate value of k to use when trying to estimate the cost of a house in Los Angeles based on the closests sales in it's area.</p>

Certain data mining algorithms (including k-means clustering and k-nearest neighbors) require a user defined parameter k. A user of these algorithms is required to select this value, which raises the questions: what is the "best" value of k that one...

[MINI] The Elbow Method

<p>Co-host Linh Da was in a biking accident after hitting a pothole. She sustained an injury that required stitches. This is the story of our quest to file a 311 complaint and track it through the City of Los Angeles's open data portal.</p> <p>My guests this episode are <a href= "https://twitter.com/pwnerchelsea">Chelsea Ursaner</a> (LA City Open Data Team), <a href="https://twitter.com/benberkowitz">Ben Berkowitz</a> (CEO and founder of SeeClickFix), and <a href= "https://twitter.com/russklettke">Russ Klettke</a> (Editor of pothole.info)</p>

Co-host Linh Da was in a biking accident after hitting a pothole. She sustained an injury that required stitches. This is the story of our quest to file a 311 complaint and track it through the City of Los Angeles's open data portal. My guests this...

Potholes

<p>Cheng-tao Chu (<a href= "https://twitter.com/chengtao_chu">@chengtao_chu</a>) joins us this week to discuss his perspective on common mistakes and pitfalls that are made when doing machine learning. This episode is filled with sage advice for beginners and intermediate users of machine learning, and possibly some good reminders for experts as well. Our discussion parallels his recent blog post<a href= "http://ml.posthaven.com/machine-learning-done-wrong">Machine Learning Done Wrong</a>.</p> <p>Cheng-tao Chu is an entrepreneur who has worked at many well known silicon valley companies. His paper <a href= "https://papers.nips.cc/paper/3150-map-reduce-for-machine-learning-on-multicore.pdf"> Map-Reduce for Machine Learning on Multicore</a> is the basis for <a href="http://mahout.apache.org/">Apache Mahout</a>. His most recent endeavor has just emerged from steath, so please check out <a href= "http://dataskeptic.com/epnotes/%3Ehttps://www.oneinterview.io/">OneInterview.io</a>.</p>

Cheng-tao Chu () joins us this week to discuss his perspective on common mistakes and pitfalls that are made when doing machine learning. This episode is filled with sage advice for beginners and intermediate users of machine learning, and possibly...

Machine Learning Done Wrong

<p>A dinner party at Data Skeptic HQ helps teach the uses of fractional factorial design for studying 2-way interactions.</p>

A dinner party at Data Skeptic HQ helps teach the uses of fractional factorial design for studying 2-way interactions.

[MINI] Fractional Factorial Design

<p>This week I spoke with <a href= "https://www.linkedin.com/in/elham">Elham Shaabani</a> and <a href= "https://www.linkedin.com/in/pauloshakarian">Paulo Shakarian</a> (<a href="https://twitter.com/pauloshakasu">@PauloShakASU</a>) about their recent paper <a href= "http://dl.acm.org/citation.cfm?id=2788618">Early Identification of Violent Criminal Gang Members</a> (also available on<a href= "http://arxiv.org/abs/1508.03965">arXiv</a>). In this paper, they use social network analysis techniques and machine learning to provide early detection of known criminal offenders who are in a high risk group for committing violent crimes in the future. Their techniques outperform existing techniques used by the police. Elham and Paulo are part of the <a href= "https://lab.engineering.asu.edu/cysis/">Cyber-Socio Intelligent Systems (CySIS) Lab</a>.</p>

This week I spoke with  and  () about their recent paper  (also available on). In this paper, they use social network analysis techniques and machine learning to provide early detection of known criminal offenders who are in a high risk group for...

Early Identification of Violent Criminal Gang Members

When working with time series data, there are a number of important diagnostics one should consider to help understand more about the data. The auto-correlative function, plotted as a correlogram, helps explain how a given observations relates to recent preceding observations. A very random process (like lottery numbers) would show very low values, while temperature (our topic in this episode) does correlate highly with recent days.      See the show notes with details about Chapel Hill, NC weather data by visiting:   <a href= "https://dataskeptic.com/blog/episodes/2016/acf-correlograms">https://dataskeptic.com/blog/episodes/2016/acf-correlograms</a>

<div id="bbody"> <div id="notebook-container" class="container"> <div class="cell border-box-sizing text_cell rendered"> <div class="prompt input_prompt">When working with time series data, there are a number of important diagnostics one should consider to help understand more about the data. The auto-correlative function, plotted as a correlogram, helps explain how a given observations relates to recent preceding observations. A very random process (like lottery numbers) would show very low values, while temperature (our topic in this episode) does correlate highly with recent days.</div> </div> <div class="cell border-box-sizing code_cell rendered"> <div class="input"> <div class="prompt input_prompt"> </div> <div class="prompt input_prompt">See the show notes with details about Chapel Hill, NC weather data by visiting:</div> <div class="prompt input_prompt"> </div> <div class="prompt input_prompt"><a href= "https://dataskeptic.com/blog/episodes/2016/acf-correlograms">https://dataskeptic.com/blog/episodes/2016/acf-correlograms</a></div> <div class="prompt input_prompt"> </div> </div> </div> </div> </div>

When working with time series data, there are a number of important diagnostics one should consider to help understand more about the data. The auto-correlative function, plotted as a correlogram, helps explain how a given observations relates to...

[MINI] Auto-correlative functions and correlograms

<p><a href="http://deepjazz.io/">Deepjazz</a> is a project from <a href="http://jisungkim.com/">Ji-Sung Kim</a>, a computer science student at Princeton University. It is built using Theano, Keras, music21, and Evan Chow's project <a href= "https://github.com/evancchow/jazzml">jazzml</a>. Deepjazz is a computational music project that creates original jazz compositions using recurrent neural networks trained on Pat Metheny's "And Then I Knew". You can hear some of deepjazz's original compositions on <a href= "https://soundcloud.com/deepjazz-ai/sets/deepjazz-on-metheny">soundcloud</a>.</p>

is a project from , a computer science student at Princeton University. It is built using Theano, Keras, music21, and Evan Chow's project . Deepjazz is a computational music project that creates original jazz compositions using recurrent neural...

deepjazz

<p>Bargaining is the process of two (or more) parties attempting to agree on the price for a transaction.  Game theoretic approaches attempt to find two strategies from which neither party is motivated to deviate.  These strategies are said to be in equilibrium with one another.  The equilibriums available in bargaining depend on the the transaction mechanism and the information of the parties.  Discounting (how long parties are willing to wait) has a significant effect in this process.  This episode discusses some of the choices Kyle and Linh Da made in deciding what offer to make on a house.</p>

Bargaining is the process of two (or more) parties attempting to agree on the price for a transaction.  Game theoretic approaches attempt to find two strategies from which neither party is motivated to deviate.  These strategies are said to...

[MINI] Bargaining

<p>I'm joined by Wes McKinney (<a href= "https://twitter.com/wesmckinn">@wesmckinn</a>) and Hadley Wickham (<a href="https://twitter.com/hadleywickham">@hadleywickham</a>) on this episode to discuss their joint project <a href= "https://github.com/wesm/feather">Feather</a>. Feather is a file format for storing data frames along with some metadata, to help with interoperability between languages. At the time of recording, libraries are available for R and Python, making it easy for data scientists working in these languages to quickly and effectively share datasets and collaborate.</p>

I'm joined by Wes McKinney () and Hadley Wickham () on this episode to discuss their joint project . Feather is a file format for storing data frames along with some metadata, to help with interoperability between languages. At the time of recording,...

Feather

<p style="text-align: left;">Mystery shoppers and fruit cultivation help us discuss stationarity - a property of some time serieses that are invariant to time in several ways. Differencing is one approach that can often convert a non-stationary process into a stationary one. If you have a stationary process, you get the benefits of many known statistical properties that can enable you to do a significant amount of inferencing and prediction.</p>

Mystery shoppers and fruit cultivation help us discuss stationarity - a property of some time serieses that are invariant to time in several ways. Differencing is one approach that can often convert a non-stationary process into a stationary one....

[MINI] Stationarity and Differencing

<p>I'm joined this week by Jon Morra, director of data science at eHarmony to discuss a variety of ways in which machine learning and data science are being applied to help connect people for successful long term relationships.</p> <p>Interesting open source projects mentioned in the interview include <a href= "https://github.com/eHarmony/face-parts-service">Face-parts</a>, a web service for detecting faces and extracting a robust set of fiducial markers (features) from the image, and <a href= "http://eharmony.github.io/aloha">Aloha</a>, a Scala based machine learning library. You can learn more about these and other interesting projects at the <a href= "http://eharmony.github.io/">eHarmony github page</a>.</p> <p>In the wrap up, Jon mentioned the <a href= "http://www.meetup.com/Los-Angeles-Machine-Learning-Data-Science/">LA Machine Learning meetup</a> which he runs. This is a great resource for LA residents separate and complementary to <a href= "http://datascience.la/">datascience.la</a> groups, so consider signing up for all of the above and I hope to see you there in the future.</p>

I'm joined this week by Jon Morra, director of data science at eHarmony to discuss a variety of ways in which machine learning and data science are being applied to help connect people for successful long term relationships. Interesting open source...

Data Science at eHarmony

<p>Goodhart's law states that "When a measure becomes a target, it ceases to be a good measure". In this mini-episode we discuss how this affects SEO, call centers, and Scrum.</p>

Goodhart's law states that "When a measure becomes a target, it ceases to be a good measure". In this mini-episode we discuss how this affects SEO, call centers, and Scrum.

[MINI] Goodhart's Law

<p>A startup is claiming that they can detect terrorists purely through facial recognition. In this solo episode, Kyle explores the plausibility of these claims.</p>

A startup is claiming that they can detect terrorists purely through facial recognition. In this solo episode, Kyle explores the plausibility of these claims.

Detecting Terrorists with Facial Recognition?

<p>Distributed computing cannot guarantee consistency, accuracy, and partition tolerance. Most system architects need to think carefully about how they should appropriately balance the needs of their application across these competing objectives. Linh Da and Kyle discuss the CAP Theorem using the analogy of a phone tree for alerting people about a school snow day.</p>

Distributed computing cannot guarantee consistency, accuracy, and partition tolerance. Most system architects need to think carefully about how they should appropriately balance the needs of their application across these competing objectives. Linh Da...

[MINI] The CAP Theorem

<p>Kristian Lum (<a href= "https://twitter.com/KLdivergence">@KLdivergence</a>) joins me this week to discuss her work at <a href= "https://twitter.com/hrdag">@hrdag</a> on <a href= "https://hrdag.org/policing/">predictive policing</a>. We also discuss <a href= "https://hrdag.org/2013/03/11/mse-the-basics/">Multiple Systems Estimation</a>, a technique for inferring statistical information about a population from separate sources of observation.</p> <p>If you enjoy this discussion, check out the panel <a href= "https://www.youtube.com/watch?v=PvmKGu2TMbs">Tyranny of the Algorithm? Predictive Analytics & Human Rights</a> which was mentioned in the episode.</p>

Kristian Lum () joins me this week to discuss her work at  on . We also discuss , a technique for inferring statistical information about a population from separate sources of observation. If you enjoy this discussion, check out the panel  which...

Predictive Policing

<p>If you'd like to make a good prediction, your best bet is to invent a time machine, visit the future, observe the value, and return to the past. For those without access to time travel technology, we need to avoid including information about the future in our training data when building machine learning models. Similarly, if any other feature whose value would not actually be available in practice at the time you'd want to use the model to make a prediction, is a feature that can introduce leakage to your model.</p>

If you'd like to make a good prediction, your best bet is to invent a time machine, visit the future, observe the value, and return to the past. For those without access to time travel technology, we need to avoid including information about the...

[MINI] Leakage

<p>I'm joined by Chris Stucchio this week to discuss how deliberate or uninformed statistical practitioners can derive spurious and arbitrary results via multiple comparisons. We discuss p-hacking and a variety of other important lessons and tips for proper analysis.</p> <p>You can enjoy Chris's writing on his blog at <a href= "https://www.chrisstucchio.com/">chrisstucchio.com</a> and you may also like his recent talk <a href= "http://www.ustream.tv/recorded/76610817">Multiple Comparisons: Make Your Boss Happy with False Positives, Guarenteed</a>.</p>

I'm joined by Chris Stucchio this week to discuss how deliberate or uninformed statistical practitioners can derive spurious and arbitrary results via multiple comparisons. We discuss p-hacking and a variety of other important lessons and tips for...

Multiple Comparisons and Conversion Optimization

<p>An ROC curve is a plot that compares the trade off of true positives and false positives of a binary classifier under different thresholds. The area under the curve (AUC) is useful in determining how discriminating a model is. Together, ROC and AUC are very useful diagnostics for understanding the power of one's model and how to tune it.</p>

An ROC curve is a plot that compares the trade off of true positives and false positives of a binary classifier under different thresholds. The area under the curve (AUC) is useful in determining how discriminating a model is. Together, ROC and AUC...

[MINI] Receiver Operating Characteristic (ROC) Curve

<p class="p1">This week is an insightful discussion with <a href= "https://twitter.com/claudia_perlich">Claudia Perlich</a> about some situations in machine learning where models can be built, perhaps by well-intentioned practitioners, to appear to be highly predictive despite being trained on random data. Our discussion covers some novel observations about ROC and AUC, as well as an informative discussion of leakage.</p> <p class="p2">Much of our discussion is inspired by two excellent papers Claudia authored: <a href= "http://dstillery.com/wp-content/uploads/2014/05/Leakage-in-Data-Mining-Formulation-Detection-and-Avoidance.pdf"> Leakage in Data Mining: Formulation, Detection, and Avoidance</a> and <a href= "http://www.kdd.org/exploration_files/v12-02-4-UR-Perlich.pdf"> On Cross Validation and Stacking: Building Seemingly Predictive Models on Random Data</a>. Both are highly recommended reading!</p>

<p class="p1"><span class="s1">This week is an insightful discussion with <a href= "https://twitter.com/claudia_perlich"><span class="s2">Claudia Perlich</span></a> about some situations in machine learning where models can be built, perhaps by well-intentioned practitioners, to appear to be highly predictive despite being trained on random data. Our discussion covers some novel observations about ROC and AUC, as well as an informative discussion of leakage.</span></p> <p class="p2"><span class="s3">Much of our discussion is inspired by two excellent papers Claudia authored: <a href= "http://dstillery.com/wp-content/uploads/2014/05/Leakage-in-Data-Mining-Formulation-Detection-and-Avoidance.pdf"> <span class="s4">Leakage in Data Mining: Formulation, Detection, and Avoidance</span></a> and <a href= "http://www.kdd.org/exploration_files/v12-02-4-UR-Perlich.pdf"><span class="s4"> On Cross Validation and Stacking: Building Seemingly Predictive Models on Random Data</span></a>. Both are highly recommended reading!</span></p>

This week is an insightful discussion with  about some situations in machine learning where models can be built, perhaps by well-intentioned practitioners, to appear to be highly predictive despite being trained on random data. Our discussion covers...

Predictive Models on Random Data

<p>Survival analysis techniques are useful for studying the longevity of groups of elements or individuals, taking into account time considerations and right censorship. This episode explores how survival analysis can describe marriages, in particular, using the non-parametric Cox proportional hazard model.</p> <p>This episode discusses some good summaries of survey data on marriage and divorce which can be found <a href= "http://www.nicholaswolfinger.com/blog/2015/04/29/revisiting-the-relationship-between-age-at-marriage-and-divorce/"> here</a>.</p> <p>The python <a href= "http://lifelines.readthedocs.io/en/latest/Survival%20Regression.html#cox-s-proportional-hazard-model"> lifelines</a> library is a good place to get started for people that want to do some hands on work.</p>

Survival analysis techniques are useful for studying the longevity of groups of elements or individuals, taking into account time considerations and right censorship. This episode explores how survival analysis can describe marriages, in particular,...

[MINI] Survival Analysis

<p>When humans describe images, they have a reporting bias, in that the report only what they consider important. Thus, in addition to considering whether something is present in an image, one should consider whether it is also relevant to the image before labeling it.</p> <p><a href="http://www.cs.cmu.edu/~imisra/">Ishan Misra</a> joins us this week to discuss his recent paper <a href= "http://arxiv.org/abs/1512.06974">Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels</a> which explores a novel architecture for learning to distinguish presence and relevance. This work enables web-scale datasets to be useful for training, not just well groomed hand labeled corpora.</p>

When humans describe images, they have a reporting bias, in that the report only what they consider important. Thus, in addition to considering whether something is present in an image, one should consider whether it is also relevant to the image...

Machine Learning on Images with Noisy Human-centric Labels

<p>Analysis of variance is a method used to evaluate differences between the two or more groups.  It works by breaking down the total variance of the system into the between group variance and within group variance.  We discuss this method in the context of wait times getting coffee at Starbucks.</p>

Analysis of variance is a method used to evaluate differences between the two or more groups.  It works by breaking down the total variance of the system into the between group variance and within group variance.  We discuss this method in...

[MINI] ANOVA

<p>Machine learning models are often criticized for being black boxes. If a human cannot determine why the model arrives at the decision it made, there's good cause for skepticism. Classic inspection approaches to model interpretability are only useful for simple models, which are likely to only cover simple problems.</p> <p>The LIME project seeks to help us trust machine learning models. At a high level, it takes advantage of local fidelity. For a given example, a separate model trained on neighbors of the example are likely to reveal the relevant features in the local input space to reveal details about why the model arrives at it's conclusion.</p> <p>In this episode, <a href= "http://homes.cs.washington.edu/~marcotcr/">Marco Tulio Ribeiro</a> joins us to discuss how <a href= "https://github.com/marcotcr/lime">LIME (Locally Interpretable Model-Agnostic Explanations)</a> can help users trust machine learning models. The accompanying paper is titled <a href= "http://arxiv.org/abs/1602.04938">"Why Should I Trust You?": Explaining the Predictions of Any Classifier</a>.</p>

Machine learning models are often criticized for being black boxes. If a human cannot determine why the model arrives at the decision it made, there's good cause for skepticism. Classic inspection approaches to model interpretability are only useful...

Trusting Machine Learning Models with LIME

<p>Paxos is a protocol for arriving a consensus in a distributed computing system which accounts for unreliability of the nodes.  We discuss how this might be used in the real world in the event of a massive disaster.</p>

Paxos is a protocol for arriving a consensus in a distributed computing system which accounts for unreliability of the nodes.  We discuss how this might be used in the real world in the event of a massive disaster.

[MINI] Paxos

<p>Our guest today is Michael Cuthbert, an associate professor of music at MIT and principal investigator of the Music21 project, which we focus our discussion on today.</p> <p><a href="http://web.mit.edu/music21/">Music21</a> is a python library making analysis of music accessible and fun. It supports integration with popular formats such as MIDI, MusicXML, Lilypond, and others. It's also well integrated with <a href= "http://elvisproject.ca/about.html">The Elvis Project</a>, enabling users to import large volumes of music for easy analysis. Music21 is a great platform for musicologists and machine learning researchers alike to explore patterns and structure in music.</p>

Our guest today is Michael Cuthbert, an associate professor of music at MIT and principal investigator of the Music21 project, which we focus our discussion on today.  is a python library making analysis of music accessible and fun. It supports...

Music21

<p>Heteroskedasticity is a term used to describe a relationship between two variables which has unequal variance over the range.  For example, the variance in the length of a cat's tail almost certainly changes (grows) with age.  On the other hand, the average amount of chewing gum a person consume probably has a consistent variance over a wide range of human heights.</p> <p>We also discuss some issues with the visualization shown in the tweet embedded below.</p> <p></p>

<p>Heteroskedasticity is a term used to describe a relationship between two variables which has unequal variance over the range.  For example, the variance in the length of a cat's tail almost certainly changes (grows) with age.  On the other hand, the average amount of chewing gum a person consume probably has a consistent variance over a wide range of human heights.</p> <p>We also discuss some issues with the visualization shown in the tweet embedded below.</p> <p><img src="http:/dataskeptic.com/epnotes/heteroskedasticity.png" alt="Image claiming relationship between income and tickets issued" width="501" height="501" /></p>

Heteroskedasticity is a term used to describe a relationship between two variables which has unequal variance over the range.  For example, the variance in the length of a cat's tail almost certainly changes (grows) with age.  On the other...

[MINI] Heteroskedasticity

<p>Urban congestion effects every person living in a city of any reasonable size. <a href="http://lewislehe.com/">Lewis Lehe</a> joins us in this episode to share his work on downtown congestion pricing. We explore topics of how different pricing mechanisms effect congestion as well as how data visualization can inform choices.</p> <p>You can find examples of Lewis's work at <a href= "http://setosa.io/">setosa.io</a>. His paper which we discussed during the interview is<a href= "http://escholarship.org/uc/item/9vz1b9rs">Distance-dependent congestion pricing for downtown zones</a>.</p> <p>On this episode, we discuss State of California data which can be found at <a href= "http://pems.dot.ca.gov/">pems.dot.ca.gov</a>.</p>

Urban congestion effects every person living in a city of any reasonable size.  joins us in this episode to share his work on downtown congestion pricing. We explore topics of how different pricing mechanisms effect congestion as well as how data...

Urban Congestion

<p>The F1 score is a model diagnostic that combines precision and recall to provide a singular evaluation for model comparison.  In this episode we discuss how it applies to selecting an interior designer.</p>

The F1 score is a model diagnostic that combines precision and recall to provide a singular evaluation for model comparison.  In this episode we discuss how it applies to selecting an interior designer.

[MINI] F1 Score

<p><a href="http://research.pomona.edu/johardin/">Jo Hardin</a> joins us this week to discuss the <a href= "http://www.amstat.org/">ASA</a>'s Election Prediction Contest. This is a competition aimed at forecasting the results of the upcoming US presidential election competition. More details are available in Jo's blog post found <a href= "https://www.r-bloggers.com/presidential-election-predictions-2016-an-asa-competition/"> here</a>.</p> <p>You can find some useful R code for getting started automatically gathering data from 538 via <a href= "https://github.com/hardin47/prediction2016/blob/master/predblog.Rmd"> Jo's github</a> and official contest details are available <a href= "http://thisisstatistics.org/wp-content/uploads/2016/07/2016_ElectionPredictionContest_DetailedRules_2016-0718.pdf"> here</a>. During the interview we also mention <a href= "http://www.dailykos.com/">Daily Kos</a> and <a href= "http://projects.fivethirtyeight.com/2016-election-forecast/">538</a>.</p>

joins us this week to discuss the 's Election Prediction Contest. This is a competition aimed at forecasting the results of the upcoming US presidential election competition. More details are available in Jo's blog post found . You can find some...

Election Predictions

<p>Random forest is a popular ensemble learning algorithm which leverages bagging both for sampling and feature selection. In this episode we make an analogy to the process of running a bookstore.</p>

Random forest is a popular ensemble learning algorithm which leverages bagging both for sampling and feature selection. In this episode we make an analogy to the process of running a bookstore.

[MINI] Random Forest

<p>As cities provide bike sharing services, they must also plan for how to redistribute bicycles as they inevitably build up at more popular destination stations. In this episode, Hui Xiong talks about the solution he and his colleagues developed to <a href= "http://www.kdd.org/kdd2016/subtopic/view/rebalancing-bike-sharing-systems-a-multi-source-data-smart-optimization"> rebalance bike sharing systems</a>.</p>

As cities provide bike sharing services, they must also plan for how to redistribute bicycles as they inevitably build up at more popular destination stations. In this episode, Hui Xiong talks about the solution he and his colleagues developed to .

NYC Bike Share Rebalancing

<p>For machine learning models created with the random forest algorithm, there is no obvious diagnostic to inform you which features are more important in the output of the model. Some straightforward but useful techniques exist revolving around removing a feature and measuring the decrease in accuracy or Gini values in the leaves. We broadly discuss these techniques in this episode.</p>

For machine learning models created with the random forest algorithm, there is no obvious diagnostic to inform you which features are more important in the output of the model. Some straightforward but useful techniques exist revolving around removing...

[MINI] Calculating Feature Importance

<p>Platform as a service is a growing trend in data science where services like fraud analysis and face detection can be provided via APIs. Such services turn the actual model into a black box to the consumer. But can the model be reverse engineered?</p> <p><a href="http://floriantramer.com/">Florian Tramèr</a> shares his work in this episode showing that it can. The paper <a href= "https://arxiv.org/abs/1609.02943">Stealing Machine Learning Models via Prediction APIs</a> is definitely worth your time to read if you enjoy this episode. Related source code can be found in <a href="https://github.com/ftramer/Steal-ML">https://github.com/ftramer/Steal-ML</a>.</p>

Platform as a service is a growing trend in data science where services like fraud analysis and face detection can be provided via APIs. Such services turn the actual model into a black box to the consumer. But can the model be reverse engineered? ...

Stealing Models from the Cloud

<p>AdaBoost is a canonical example of the class of AnyBoost algorithms that create ensembles of weak learners. We discuss how a complex problem like predicting restaurant failure (which is surely caused by different problems in different situations) might benefit from this technique.</p>

AdaBoost is a canonical example of the class of AnyBoost algorithms that create ensembles of weak learners. We discuss how a complex problem like predicting restaurant failure (which is surely caused by different problems in different situations)...

[MINI] AdaBoost

<p>Financial analysis techniques for studying numeric, well structured data are very mature. While using unstructured data in finance is not necessarily a new idea, the area is still very greenfield. On this episode,<a href= "http://deliarusu.github.io/">Delia Rusu</a> shares her thoughts on the potential of unstructured data and discusses her work analyzing Wikipedia to help inform financial decisions.</p> <p>Delia's talk at PyData Berlin can be watched on Youtube (<a href="https://www.youtube.com/watch?v=B9sIS2-_sv4">Estimating stock price correlations using Wikipedia</a>). The slides can be found <a href= "https://github.com/deliarusu/wikipedia-correlation">here</a> and all related code is available on <a href= "https://speakerdeck.com/deliarusu/estimating-stock-price-correlations-using-wikipedia"> github</a>.</p>

Financial analysis techniques for studying numeric, well structured data are very mature. While using unstructured data in finance is not necessarily a new idea, the area is still very greenfield. On this episode, shares her thoughts on the potential...

Unstructured Data for Finance

<p>The Gini Coefficient (as it relates to decision trees) is one approach to determining the optimal decision to introduce which splits your dataset as part of a decision tree. To pick the right feature to split on, it considers the frequency of the values of that feature and how well the values correlate with specific outcomes that you are trying to predict.</p>

The Gini Coefficient (as it relates to decision trees) is one approach to determining the optimal decision to introduce which splits your dataset as part of a decision tree. To pick the right feature to split on, it considers the frequency of the...

[MINI] Gini Coefficients

<p>The Bootstrap is a method of resampling a dataset to possibly refine it's accuracy and produce useful metrics on the result. The bootstrap is a useful statistical technique and is leveraged in Bagging (bootstrap aggregation) algorithms such as Random Forest. We discuss this technique related to polling and surveys.</p>

The Bootstrap is a method of resampling a dataset to possibly refine it's accuracy and produce useful metrics on the result. The bootstrap is a useful statistical technique and is leveraged in Bagging (bootstrap aggregation) algorithms such as Random...

[MINI] The Bootstrap

<p>Today's episode is all about Causal Impact, a technique for estimating the impact of a particular event on a time series. We talk to <a href="http://www0.cs.ucl.ac.uk/staff/W.Martin/">William Martin</a> about his research into the impact releases have on app and we also chat with <a href= "https://twitter.com/kjblakemore">Karen Blakemore</a> about a project she helped us build to explore the impact of a Saturday Night Live appearance on a musician's career.</p> <p>Martin's work culminated in a paper <a href= "http://www0.cs.ucl.ac.uk/staff/W.Martin/pubs/Martin_FSE_Causal_PrePrint.pdf"> Causal Impact for App Store Analysis</a>. A shorter summary version can be found <a href= "http://www0.cs.ucl.ac.uk/staff/W.Martin/pubs/Martin_ACM_SRC_cameraReady.pdf"> here</a>. His company helping app developers do this sort of analysis can be found at <a href= "http://crestweb.cs.ucl.ac.uk/appredict/">crestweb.cs.ucl.ac.uk/appredict/</a>.</p>

Today's episode is all about Causal Impact, a technique for estimating the impact of a particular event on a time series. We talk to  about his research into the impact releases have on app and we also chat with  about a project she helped us build to...

Causal Impact

<p>Cloud services are now ubiquitous in data science and more broadly in technology as well. This week, I speak to <a href= "https://twitter.com/mark_azurecat">Mark Souza</a>, <a href= "https://twitter.com/tobiassql">Tobias Ternström</a>, and <a href= "https://twitter.com/coreysanderswa">Corey Sanders</a> about various aspects of data at scale. We discuss the embedding of R into SQLServer, SQLServer on linux, open source, and a few other cloud topics.</p>

Cloud services are now ubiquitous in data science and more broadly in technology as well. This week, I speak to , , and  about various aspects of data at scale. We discuss the embedding of R into SQLServer, SQLServer on linux, open source, and a few...

MS Connect Conference

<p>Classically, entropy is a measure of disorder in a system. From a statistical perspective, it is more useful to say it's a measure of the unpredictability of the system. In this episode we discuss how information reduces the entropy in deciding whether or not Yoshi the parrot will like a new chew toy. A few other everyday examples help us examine why entropy is a nice metric for constructing a decision tree.</p>

Classically, entropy is a measure of disorder in a system. From a statistical perspective, it is more useful to say it's a measure of the unpredictability of the system. In this episode we discuss how information reduces the entropy in deciding...

[MINI] Entropy

<p class="p1">Today's episode is a reading of Isaac Asimov's <a href= "http://www.amazon.com/Isaac-Asimov-Complete-Stories-Vol/dp/038541627X/ref=sr_1_5?ie=UTF8&qid=1449253483&sr=8-5&keywords=asimov"> Franchise</a>.  As mentioned on the show, this is just a work of fiction to be enjoyed and not in any way some obfuscated political statement.  Enjoy, and happy holidays!</p>

<p class="p1"><span class="s1">Today's episode is a reading of Isaac Asimov's <a href= "http://www.amazon.com/Isaac-Asimov-Complete-Stories-Vol/dp/038541627X/ref=sr_1_5?ie=UTF8&qid=1449253483&sr=8-5&keywords=asimov"> Franchise</a>.<span class="Apple-converted-space"> </span> As mentioned on the show, this is just a work of fiction to be enjoyed and not in any way some obfuscated political statement.<span class= "Apple-converted-space"> </span> Enjoy, and happy holidays!</span></p>

Today's episode is a reading of Isaac Asimov's .  As mentioned on the show, this is just a work of fiction to be enjoyed and not in any way some obfuscated political statement.  Enjoy, and happy holidays!

2016 Holiday Special

<p>We close out 2016 with a discussion of a basic interview question which might get asked when applying for a data science job. Specifically, how a library might build a model to predict if a book will be returned late or not.</p>

<p>We close out 2016 with a discussion of a basic interview question which might get asked when applying for a data science job. Specifically, how a library might build a model to predict if a book will be returned late or not.</p> <div class="blog-mailinglist-container" data-reactid= ".1ybakrz04j6.0.3.2.1"> </div>

We close out 2016 with a discussion of a basic interview question which might get asked when applying for a data science job. Specifically, how a library might build a model to predict if a book will be returned late or not.

episodes (2016)