The Experimental Design of Paranormal Claims

Thanks to brilliant.org for sponsoring this episode.

In this episode of Data Skeptic, Kyle chats with Jerry Schwarz from the Independent Investigations Group (IIG)'s SF Bay Area chapter about testing claims of the paranormal. The IIG is a volunteer-based organization dedicated to investigating paranormal or extraordinary claim from a scientific viewpoint. The group, headquartered at the the Center for Inquiry-Los Angeles in Hollywood, offers a $100,000 prize to anyone who can show, under proper observing conditions, evidence of any paranormal, supernatural, or occult power or event.

It’s not always obvious how one should look into such claims statistically. Nevertheless, the IIG does a good job at critically examining fringe science and extraordinary claims from a rational, scientific viewpoint. Jerry describes his experiences in experimental design to test the different types of claims he has heard over the years.

What qualifies as a paranormal power? “We know it when we see it," says Jerry. But to go by the IIG's definition, it is anything that the applicant can do that would be outside the normal processes of natural science. Jerry gives a few examples of the common type of claims he and his colleagues have heard over the years, as well as notable ones that have warranted further examination.

How does one measure a claim? If we were to take anyone seriously, we would have to compare what they could do to how often someone else can do it purely by chance. As a lead investigator, Jerry works with applicants to design testable protocols to perform demonstrations of the claimed ability. In most cases, an applicant would be asked to perform a preliminary demonstration. If successful, the IIG follows up with a formal test.

During this episode, we would also like our listeners to think about how this discussion relates to how machine learning systems work. In an era when algorithms are increasingly being described as black boxes, how do you know they do what their creators say they do? How do we convince people that machine learning systems work? There’s excellent work in model interpretability, but perhaps we have to give up on the idea of a perfectly interpretable model.

CHICAGO Tues, May 15, 6pm. Come to our Data Skeptic meetup.

CHICAGO Saturday, May 19, 10am. Kyle will be giving a talk at the Chicago AI, Data Science, and Blockchain Conference 2018.