Interview with Ernie Tedeschi, Economist, on Poll Re-Weighting

Our recent episode Opinion Polling for Presidential Elections featured segments from an interview with Ernie Tedeschi. This post contains the full transcript for that interview conducted by Christine Zhang. The player below also contains a link to the full recording from the Data Skeptic Bonus Feed.

I wanted to talk to you mainly about our your re-weighting of the USC/L.A. Times poll. What motivated you to undertake this exercise?

So my background is not in polling. I'm an economist in the private sector and I do several different things. One of the things that I do is labor economics, so I work a lot with census data; in particular, one survey called the Current Population Survey, which is the monthly survey that the government conducts that ultimately leads to the unemployment numbers, labor force participation, and other labor market metrics that we see come out. If you see the news reports on the first Friday of every month, you'll see the “jobs number.” Part of that report comes from the Current Population Survey. Weighting in that survey is extremely important, because what you're doing is you're trying to get a snapshot of the U.S. population -- in particular, the adult population -- every month, and in order to you that you have to accurately weight the responses to the survey so that they reflect the American population at large. In other words, you're trying to take a survey of about a hundred and seventy five thousand respondents every month and use that to say something about an adult population of more than two hundred fifty million. So obviously how you weight each individual and that survey is very important.

As a politically-engaged American I was following the election very closely and following the polling closely. As I said I'm not a pollster but I was familiar with the L.A. Times/USC poll on I I followed it four years ago when it was are managed by the RAND Corporation during the Obama/Romney contest. There are things I really liked about the poll and that's why I was highly anticipating following it in this election cycle and so then when you USC took it over and it was funded by the L.A. Times on I started looking at it again and and following it.

Like a lot of people, I noticed on that it was out of the mainstream of the whole swath of polls that were out there. A lot of people attributed that to political bias on the part of the L.A. Times/USC poll. I knew that the methodology was pretty sound and serious back when it was RAND. I knew the reputation of USC and that they were extremely serious and not partisan. So I figured that couldn't be the answer. The main reason I did this project was just to answer the question, “why is this poll so different from all the other polls that were generally showing Hillary Clinton leading by, depending on where in the political cycle, between two and ten points whereas the L.A. Times/USC poll pretty consistently had Donald Trump winning the popular vote.

Remind me: did your Google site that had the re-weights and all of your analysis come out before or after Nate Cohn’s piece about the nineteen-year old person skewing poll averages?

So that came out around the same time. Nate and I had been emailing one another on that topic. I'm trying to remember the exact sequence of events. My recollection is that it started out as a Twitter conversation. I had been talking about sort of beginning to get into the L.A. Times/USC poll. Somebody had notified me in a Twitter conversation, “oh, there's this one nineteen-year-old African American that's skewing the poll in Texas or in Illinois, and that's throwing everything off.” I got into a conversation with this individual about digging into the data and ultimately found that there were two over-weighted African American individuals in the L.A. Times/USC poll, one of whom was leaning toward Trump, one of whom was leaning toward Clinton. And so I responded to this person, “well you know, this is a valid issue, you have to worry about weighting. On the other hand, there are these two individuals in the poll, and they seem to be counteracting one another, so I'm not sure that that can really explain the systemic difference in the L.A. Times/USC poll. And I think that either I had forwarded that conversation to Nate or he had seen the conversation, and I think that that formed the basis of Nate’s piece. And I think very shortly thereafter, I posted my first sort of deep dive into the weighting issue with L.A. Times and how to correct for it.

So even from the beginning of the election cycle, you were interested in looking at this poll. It wasn’t like you read this Nate Cohn thing and thought, “I think I'm going to do a deep dive into the poll,” right?

Right. And I should say that I'm a fan of the L.A. Times/USC poll. I was before even I went into this analysis. In fact, being a fanboy of the poll was what really motivated me. I like having raw data, and I particularly like a type of data called longitudinal data. Longitudinal data is just a fancy word for when you're able to match the same people across months. Normally, when polls are conducted or surveys are conducted, they survey one population in one month or in one session, and then it's a completely different set of people in another month. But in certain polls, like the L.A. Times/USC poll, what they'll do is they will track the same people over time. What's really interesting about that is that you can see how people's opinions change over time -- what makes them go from Trump to Clinton or back or to someone else, their self-expressed likelihood of voting, etc. -- a lot of data that you don't get from traditional polls. And I should say too that the other thing to the L.A. Times/USC poll’s great credit is that they released their data to the public on a regular schedule -- the microdata as we call it -- and that's something that most polls won’t do. Most polls will just give you a series of summary tables called cross-tabs that will give you the underlying results of the poll. But they won't let you go into the actual underlying responses of each individual and you be able to download them in an accessible fashion in a way that's easily analyzed and manipulable so that you're able to run different statistical tests on it. L.A. Times/USC does that. The analysis that I did wouldn't have been possible if L.A. Times/USC had just acted like most other polls. And the third thing I'll say about L.A. Times/USC is what I liked about them is that they didn't just ask “who are you voting for? Are you voting for Trump, for Clinton, or for someone else?” They allowed people to express their opinions on a spectrum from 0 to 1, which I think is much more realistic. Most voters were undecided in the sense that they hadn’t totally made up their minds, but they were probably leaning one way or the other throughout the cycle. L.A. Times/USC allowed them to express that in the poll. They could say “right now I'm seventy percent for Clinton, but I’m thirty percent for Trump” or “I’m forty percent for Clinton, but I'm forty percent for Trump, and I'm twenty percent for somebody else.” It also asked them, on a spectrum, “what's the likelihood that you're going to vote on Election Day?” Now, there are all sorts of issues of bias around that question, because you always say you're going to vote, and oftentimes for whatever reason you don't end up voting up. We feel like we should be voting, so you're more likely to tell a survey questioner that you are going to vote. But at least that provided a little bit of honesty, so that people could say “I'm eighty percent likely to vote come November,” or “I'm a hundred percent likely to vote” or “ no, I actually already voted -- I voted early -- so I'm at a hundred percent right now.” That was nice because it really allowed for a fine-tuned detailed analysis of what was going on during the election.

So what was the most interesting thing that you found from your results?

What the L.A. Times/USC poll did was -- I mention that they surveyed the same people over the over the course of the entire survey. There a couple people that came in later, and there were a couple of respondents who seem to have dropped out -- they stopped responding to the questions -- but for the most part it was the same people over the over the entire cycle.

So you have this survey of roughly three thousand people that you continuously survey, and then you have to figure out how to weight them. What L.A. Times/USC decided to do was in many cases very straightforward. They look at census data and they say, okay we're gonna weight based on age, sex, race ... in other words we're going to come up with weights so that the proportions of an African American thirty-year-olds who's married in this survey matches the same proportion of all African American thirty-year-olds who are married in the entire American population as well, or the entire adult population. That's very straightforward, and you can quibble a little bit with certain things about how that's done, but that's a very non-controversial, straightforward way of doing it.

The one extra thing they did, though, was they weighted based on self-responded vote back in 2012. In other words, they asked everybody in their survey, “how did you vote into 2012? Did you vote for Obama, did you vote for Romney, or did you go for someone else, or did you not vote at all?” We know, obviously, the results of the 2012 election. We have actual data on that. So they constrained the way that they weighted their sample so that it matched the actual voting results in 2012. And this was the thing that raised a red flag for me as I was digging into the methodology because there is a well-known phenomenon in survey response called “winner bias” -- that’s the easiest way to describe it -- where either because you forget or because you want to make it seem like you were on the winning side you tell a respondent that you ended up going with the winner in the last election even if you didn't actually vote for in this case President Obama in 2012. So what I suspected that meant is that there were more people in the L.A. Times/USC poll who said that they voted for President Obama than actually really had in 2012. And then what you're doing is you're taking all these people who say that they voted for president Obama even though some of them probably voted for Governor Romney at the time and then you’re constraining them down -- you’re reweighting them so that you're compressing them to equal the actual vote proportion that President Obama got in 2012. That creates a bias against people who might in actuality have voted for President Obama, and it creates a bias in favor of key people who voted for Governor Romney at the time, because those actual governor Romney voters are represented both in the self-responded Obama voters in the survey, and they're represented in the self-responded Romney voters.

So the way that I dealt with that was I started from square one. I said, “alright let's get rid of the weight that the L.A. Times/USC poll is using in their official results. Let’s strip those out, and let's just make new weights. And we’ll use the Current Population Survey,” -- which you recall I said is the survey that the government puts out that, among other things produces the unemployment rate, but it also has a rich array of demographic information and some economic information in it that we can use to make weights. -- “We’ll see what's available in the L.A. Times/USC poll, and we’ll just build new weights sort of in the same vein that the L.A. Times/USC poll did itself but we won't include self-reported 2012 as one of the metrics.”

I included a lot of the same metrics I included race, age, gender --

The usual suspects.

The normal ones. The other thing I'll mention is that -- this is a little bit more esoteric -- but it's important. The L.A. Times/USC poll had asked people in their survey, “what's your household income?” And then they had used the Current Population Survey of May 2016 to weight people in part based on that that income question. Because there's a question in the Current Population Survey about your household income. That question in the Current Population Survey is notoriously unreliable. Far from everybody answers it. It's heavily seasonal. It depends on when during the year you ask it. When you look at the distribution of that question in the Current Population Survey, it doesn't match at all what we know to be much more reliable data that's conducted on an annual basis. So for the income question I used a different survey, the American Community Survey, which is a much larger sample size -- two and a half million. It's conducted on an annual basis, so it wasn't quite as timely as what L.A. Times/USC had used for their poll, but very rich, very reliable data. It's where we get the official poverty number from. It's where we get the official household income number from. Generally thought to be a much better source of this income data than the monthly Current Population Survey. I used that for that specific question, and I re-weighted based on that.

Why do you think they used the Current Population Survey instead of the American Community Survey?

For some of the simulations I did early on, I just used the American Community Survey for everything. When it comes to demography like age, sex, race, etc., the Current Population Survey is generally very timely and very good, because actually the weights that they use are based on the American Community Survey and on census estimates. That's because they can't afford to have a representative sample of the population where they're re-estimating all of these demographics every single month. So what they do in the Current Population Survey is they just use the estimates from the latest census data as their assumption when they create the official government CPS weights. So using basic demography like that from the Current Population Survey should be fine and and shouldn't bias the results one way or the other. But it was those two things -- it was voting recollection from 2012 and this income question that raised red flags for me.

Once I had re weighted based on everything that I talked about before, the results of the L.A. Times/USC survey basically almost exactly the matched polling averages that you saw on a site like say RealClearPolitics.com. And all Real Clear Politics does is it takes the most prominent arm presidential polls and it just did an average of them over time. USC/L.A. Times, once I had re-weighted it, almost exactly matched that average. So it went from being an outlier to being sort of right in the middle of the consensus distribution.

What that told me was not that there was anything bad or wrong about L.A. Times/USC, but it that seemed to point in the direction that one of these two things: either this income question, or more likely, this recollection bias from the 2012 election was skewing the results in such a way that it made Trump seem to be running away with the popular vote. And at the end of the day, when November came, obviously Trump won the Electoral College, but I think my final re-weighted estimate of L.A. Times/USC was a Hillary win by 1.9% in the popular vote and I think at the end of the day her win was a margin of around 2% ... maybe just over 2%.

So the raw data in L.A. Times/USC seem to be very good, very representative on a national level. It was just a matter of weighting it correctly.

That's interesting. I had heard a lot about the 2012 vote thing, but I didn't hear much about the income question. I think that really speaks to the different sources of data. I think this is something that a lot of people don't realize, but in addition to making sure that your surveying sample correctly and using the right polling methods, the other thing that all of these estimates always include is an implicit assumption of what the American electorate looks like.

Right.

And that's where demography really makes a difference.

I'll say, too, that one of the nice things about the L.A. Times/USC poll -- you remember I said that they ask you the question of how likely was that you would come November. What's nice about that is that what many polls do is they have to explicitly model who is likely to show after show up on Election Day. And they have to make a lot of exogenous assumptions in their model about how that will work. That will be based on historical experience, that will be based on pollster judgment. What the L.A. Times/USC poll did was they just took the response as is. So whatever the respondent to the survey said was their likelihood of voting on Election Day -- that was the additional weight that L.A. Times/USC used in projecting turnout on Election Day. When I re-weighted the L.A. Times/USC poll, I made that same assumption. Just whatever self-responded likelihood of showing up was was the number that I used. I like that because it's it's well motivated. It allows the individual survey respondents control over how they're being weighted, based on their actual response and their actual estimation. Obviously you always want to kick the tires on any of these assumptions over time, and that is no exception. You would want to compare the performance of self-response voter turnout versus model-based voter turnout. But I think that my exercise showed at least that if you got the weighting right, self-response voter turnout seemed to perform just fine, at least in terms of the national popular vote.

You said that you liked that the L.A. Times/USC survey had asked this question [about likelihood to vote] in a spectrum way. In your experience, do other surveys just not ask this question, or do they ask it a different way?

I'll emphasize that I'm not a polling expert, so I will not say that no poll does this, but in doing this I looked at an array of different polls, and my experience is that most other polls will just ask a binary question. "If the election were held today, who would you vote for?" There are advantages to that question, too, because it forces you to make a choice. There's an argument from some that by giving survey respondents the ability to answer a question with a probability or a percentage rather than just making a choice is not a realistic … when you go into the polling booth, they don't ask you, “assign some percentage this candidate.” You have to vote.

And this was one of the sub-questions I wanted to explore with the L.A. Times/USC poll. If it wasn't just the income question or the 2012 vote recollection question, could it have been one of these other unique facets of L.A. Times/USC that led to these out of consensus results?

Since I was able just with the weighting procedure bring L.A. Times/USC in line with consensus that led me to think that, “no, actually these other unique features of L.A. Times/USC are a wash in terms of bias. They don't seem to bias the poll one way or the other.” It came out, again, very much in line with consensus once you've adjusted for weighting. And so if you're not changing the top line result from this, then it's great for the researcher who is able to dig into the microdata and do things like -- I looked at on Twitter, for example, how did the probability of voting for Hillary Clinton change after the Comey letter came out in the last week or so of the campaign? How did the probability of voting for Donald Trump change after the allegations about Access Hollywood came out? The L.A. Times/USC poll allows you to very easily do experiments like that. A -- because it's longitudinal and tracks the same people over time. B -- because even if people you know still support one candidate much more over the other, it allows them to [inaudible] “maybe I'm not [inaudible] Secretary Clinton, maybe no I'm only 65%percent in favor of her.” You can see those minute changes over time in the data.

On the discussion of the weighting [in your methodology section], on you talk about doing interactions between different demographic categories. So you have age, sex, race, etc. -- the usual suspects, as I like to call them -- and those might be broad categories. But then what makes it statistically even more difficult is when you start doing interactions between them … those questions [e.g. what percentage of older white females were for Clinton?] become harder and harder the more categories you break them down into. Can you talk a little bit about what you call iterative re-weighting?

That's a great question. It's also a very wonky question. So here’s the issue. The issue is you want your survey to match the overall population, and you want it to match the overall population, as you said, along several different dimensions, whatever dimensions you might think are the most relevant for the question that you're looking at. With an election, we can think of a lot of different relevant dimensions that would come up. The problem, as you hinted at, is that the more dimensions you add, the likelier it is that you could end up with some combination of those dimensions that ends up with just nobody right in a single month. Let me give you an example. If I were looking at, say, Hispanic sixty-two-year-old men who are divorced and make an income of less than fifty thousand dollars, that's a hyper specific combination of dimensions. I could end up with just nobody. Particularly if I'm looking at the Current Population Survey, which as I said are has a monthly sample of about a hundred and seventy five thousand individuals. Now that sounds like a really big number, and for most for most purposes that would be a wonderful sample size, but for estimating the American population of more than three hundred million people -- more than two hundred fifty million adults -- that's actually kind of a small sample. You can very quickly come to combinations of dimensions that are relevant for your research that don't give you an accurate read on the actual proportion in the overall population. So there are a couple of different ways to deal with that. One is you base your weights off of a larger broader sample, and so in this case we could have used the American Community Survey, which is an annual dataset -- a 2.5 million sample instead of a 175,000 -- much less likely that we're gonna end up with what statisticians call “cells,” or interactions of all the different dimensions that you’re interested in, of zero. So that's one way of doing it. Now, the downside of that is that you lose the timeliness of the monthly data from the Current Population Survey.

When I started this, by the way, the latest American Community Survey was from calendar year 2015. It just covered all of that. So when I originally did this, I did an ACS-only version and then I did a version with the Current Population Survey, because I wanted to make sure that much more recent data wasn't going to dramatically changed results. Then like in September, the data on the 2016 ACS was released by the Census Bureau, and that relieved a lot of my heartache of doing it that way. So one way of solving this problem is just to get a larger, broader dataset.

The solution that I went with is something called an iterative re-weight. Rather than trying to find the weights associated with every single hyper-specific combination of dimensions imaginable, on what you do is --t I like to think of it as like a sculptor with clay -- you cycle through a lot of different re-weights that are just based on a single dimension. And if you do it enough times over every single dimension, just like you have a piece of pottery on a turntable, you'll eventually get the shape that accurately reflects the population as a whole. So in practical terms, what that means is, “okay you start out with with sex, and so you have let's say fifty percent men, fifty percent women. So you make sure that your sample is weighted to be fifty percent men and fifty percent women. Then you move on to race as a separate re-weight and you do all of the different racial proportions that you have. Then you do you know income. And then you do education, for example. And rather than doing everything all at once, you iterate. You do it one at a time over and over again, sort of fitting all the proportions in. And then over time, if you do it enough times, what you end up with is something that is reflective of the overall population.

We have algorithms that will do this automatically, and in fact, I think that the L.A. Times/USC poll used a particular algorithm that's available for statistical software to come up with their original weights. To be honest, I wanted to do it the old-fashioned iterative way, just so I could sort of say to myself, “I can code this, I understand how it's working,” just to go through the motions of learning the computational process behind it. This was just as much a learning experience for me as it was an analytical experience for me.

That's fascinating. So walk me through this. You start with, say, sex. You weight your data to reflect the population of men and women, which is fifty-fifty...

My initial weight for every single person was some constant number -- let's say ten thousand -- for each survey respondent. And when you're starting out, you have no idea how that sort of pans out in terms of the final weights. So you pick a number. And then you say “okay, step one, let's start with the easiest category -- gender -- and again let's assume that it's fifty-fifty, so then what you're doing is you're taking that weight of ten thousand that you initially assigned and you are incrementally changing it so that the sum total -- the weighted sum total -- of everyone in your survey ends up being fifty percent male, fifty percent female. And so if your survey started out as having sex more men than women, that would mean that after that first stage, the weight on the men in your survey would be less than ten thousand, because now it has to underweight men in order to get the final weighted total to be fifty percent of the overall total.

So then you do that. So let's say that you ended up with -- men are now weighted at nine thousand, women are weighted at eleven thousand each.

Just in super simple terms -- if there are a hundred people on your sample, and you're supposed to have fifty men and fifty women, but you have like eighty women and twenty men, then you obviously would have to weight the men a lot more than the eighty women, in order to get the fifty-fifty.

That’s exactly right. And then let’s say the next one is education. And so then you say, “okay, I need to have, let's say, thirty percent with a bachelor’s degree or greater; I need to have thirty percent with some college; I need to have forty percent with a high school degree or less. And so you're starting out with, let's say, a nine thousand weight on men, an eleven thousand weight women. Whatever the results of that first stage were. And then you're doing the same thing with education, and you're changing that first stage re-weight with a second stage that's based on education, and you're making sure that your weighted sample has the correct educational proportions. Then you move on to the next category. You move on to, let's say race and ethnicity, and you re-weight based on that. And then you go through all the dimensions.

When you've gone through once, there's no guarantee that one that going through one cycle of dimensions is enough to get all of the weights correct. You have to start over and start from square one right back with sex again, and then work your way down again. But what's happening is every single every single cycle that you do this the final weights are gradually whittling down to the correct weight -- the weight that accurately reflects the overall population for each individual person in that survey over every dimension. Sometimes it takes as few as five complete cycles of re-weighting through these dimensions. I think I did it in the code fifty times or so, just to be absolutely sure, but on the fiftieth time there was virtually no change at every single cycle. It had reached an equilibrium where every single person was weighted appropriately. After going through all the cycles, and looking at your weighted data, you will find that weighted proportions of sex will be fifty percent each, you'll find that the weighted proportions of education will be the same as the overall population. Every single dimension will be exactly at or extremely close to the target weights that you put into it. And that's just a function of the fact that you're repeating this exercise so many different times and it whittles down to the right weight.

By “right weight,” what you mean is the weights such when you take the aggregate data, they reflect the total U.S. population. And then that presupposes that you know what the population is in terms of those all of those exact dimensions -- gender, race, age, income, education, marital status, and state of residence ... You get get those numbers from the CPS? No, sorry, you get those numbers from the ACS?

I did it a couple different ways. I get those numbers from one of those two census surveys, either the ACS or the CPS depending on which one I'm doing. The differences were not dramatic.

Another question, as somebody who hasn't worked very much with Census data. Do both the CPS and the ACS ask people whether or not they voted? Is everybody in there a voter, or are they just an adult?

Good question. It's actually technically the entire population in both cases. The ACS measures what's called the “resident population.” You can actually extract that from the Current Population Survey as well. But the focus of the Current Population Survey is what's called the “non-institutional civilian population.” And that's another reason why I favor the ACS.

I usually deal with the Current Population Survey. This unemployment survey / this labor market survey, which again focuses on the civilian noninstitutional population. Now what does that mean? “Civilian noninstitutional population”? Well, it means number one -- it's the adult population, which the Bureau of Labor Statistics for this purpose defines as sixteen or over. “Civilian” obviously means non-military. That's a key thing that I’ll return to in a second. “Non-institutional” means that you’re you're in a household that is not considered an institutional arrangements. So an institutional arrangement could be a college dorm, it could be a prison, it could be a monastery, it could be old-age housing …. There are a lot of households that might be excluded from the civilian noninstitutional population that are not captured by that monthly CPS data.

At first I was like, “well, I just really want the timeliest data, so I want to focus on the Current Population Survey. And then I was thinking about it and of course the most obvious thing that that struck me is “wow, I'm excluding members of the active duty military by looking at the Current Population Survey.” That for one might create a bit of a bias. And to be clear, remember I'm not using this to determine the number of voters. I'm just using this to re-weight the L.A. Times/USC survey. Going to the American Community Survey, which gives you the entire U.S. population, including for example, active duty military and I should say has a better estimate of college students who are in dormitories ... it's actually interesting, so in principle they’re not captured in the Current Population Survey, but they actually generally are because the Current Population Survey will go to their parents. And the question will be, “name all the members of your household. If you have a child in college who is living in a dormitory, include that child as well. And so the parents will say, ‘okay well you know John and Deborah you know they're in college and so I’ll list them as members of my household.’” So the Current Population Survey actually does capture a lot of other college students. But again that's one of the metrics where you wanna be like, “well, let's try a sensitivity analysis here with different data and make sure that the results are not dramatically different and then think more critically about what the best sort of trade-off between getting an accurate count of the U.S. population and getting the timeliest data is. And these are the trade-offs we have to think about.

If you use the ACS to look at, for example, the proportion of the male population overall, I think it rises two percentage points if you use the American Community Survey rather than the Current Population Survey. Which makes sense, because in the American Community Survey now you're bringing in active duty military, you're bringing in prisoners as well. So it's not surprising that the gender distribution will end up being more skewed toward men in the American Community Survey. But then you have to kind of take a step back and say, “okay, well obviously we want to include active duty military in whatever sort of weighting sample that we use. Is it appropriate to use prisoners? Are prisoners likely to vote?” That's not a question that is asked in the L.A. Times/USC survey. That was sort of like the next hurdle. Are all of the additional people the ACS bring in really appropriate for the purposes of re-weighting the L.A. Times/USC poll?

I think ultimately I arrived at a compromise where I brought in the active duty military personnel and used them for calculating the adult population proportions for each of these different dimensions, but I did not bring in prisoners, because I felt like that would that would skew the population proportions in a way that would not necessarily be reflective of either the L.A. Times/USC poll since I'm just based on reading their methodology it didn't sound like they surveyed prisoners. Or just the American electorate as a whole, since prisoners are much less likely to vote, either they're legally prevented from voting, or just in terms of population behavior. So that was another hurdle that we had to go through.

To be clear, weighting is to bring the sample to the population. And in your case the population is the American electorate or the American population?

In this case with the L.A. Times/USC poll, what I was weighting was the American adult population, and then I let the L.A. Times/USC poll sort of whittle that down to the electorate based on that question I was telling you about about the likelihood of each respondent voting on election day.

Okay that's that's what I gathered from from your previous response, but I just want to make sure.

In other words, I'm letting the server respondents do the work telling telling me whether they're going to vote or not.

I know that a lot of people say that the quote “best way” is to weight according to something called a voter file. The voter file tells you who the voters actually are. Is that right? Have you heard that before as well?

That makes a lot of sense to me. Obviously, that wasn't available with the L.A. Times/USC poll.

After the election, there have been a lot of post-mortems on this poll and frankly a lot of misconceptions. So a lot of people are now like, “it’s the only one that got it right.” I'm not even going to say that’s untrue, because it is true if you've really want to think about the actual outcome. It's just untrue in terms of how it was able to arrive at that and what it was estimating. I read it article in the L.A. Times about you know what the poll had missed and why it was off on the popular vote. The answer was because, according to the article, they had over sampled on rural voters and they didn't correct for urban/rural. This is not necessarily a common weighting factor. It wasn’t one of the things that you mentioned, for example. Do you have any thoughts on that at all?

That's very much possible. One thing to say is that I could have arrived at the “consensus answer” and the “right answer” by accident. That's entirely possible here. By re-weighting the poll and introducing these different dimensions, I could have introduced some dimensions that were skewing the poll one way and then some dimensions that were skewing the poll the other way, and I just happened to end up at this happy medium that over time was pretty close to a consensus average of the other major polls out there and ended up being close to the actual result. That's very possible.

I haven't done a deep postmortem on that analysis answering that particular question, but that’s one thing to keep in mind here. The specific dimensions that I used are not necessarily the right dimensions. To be perfectly honest, I was thinking that this was going to be a one-time thing.

Like, “here's a cool question I would like to be able to answer based on the data.” After spending some time thinking about it and trying it out, I got this result that was close to the consensus of different polls out there and honestly I was like, “I wonder if this is going to maintain itself going forward over time.” And I just kept updating that original methodology.

There needs to be a much more robust postmortem on which metrics to use, how to weight them, what the best ex-ante (beforehand) way of thinking about weighting these variables is. Anybody can go back after the fact and find exactly right weights that what will get the outcome but the real challenge -- and this is why I don't give L.A. Times/USC a lot of Flack for this. They had a big challenge. They had to decide before they had even published a single result from their poll how they were going to weight it. And they had to stick with that weighting throughout the entire sample. And that's a challenge. That's tough to do. So I don't want to minimize that.

Did you ever talk to them?

I talked with a couple of journalists at the L.A. Times. I didn't have extensive conversations with [USC]. There were a couple of workup like technical questions about the micro data that I emailed them, and they were extremely responsive. I have no idea whether they read my work or were trashing it --

They’re familiar with it, for sure. I know for a fact are doing a postmortem. They're going to AAPOR where they’re going to do a presentation on their findings. Now is the time that they're really doing a deep-dive into their results. And they do think the overweight matters a lot.

In terms of urban versus rural, that could be the real salient metric here, right? That if they had just directly accounted for from the beginning would have solved all a lot of this out of consensus issue with the poll. And I may have just kind of gotten that right through a back door, implicitly rather than explicitly by changing the income variable and by taking out a 2012 vote weighting. That may have effectively accounted for that urban-rural divide without actually putting in a variable for urban versus rural. So I don't know. That's one of the many interesting questions coming out the poll. And again what's great about the poll is that everybody can answer those questions or analyze those questions because all of the data is public.

Just out of curiosity, when the poll was administered by RAND in 2012, which appeared to get it right, did they use the same weighting by stated vote in the previous election?

Yes. You’re right that they were a lot more accurate. I think they ended up under-predicting President Obama's margin in the end, which is sort of the reverse -- if you're speaking in terms of Democrats versus Republicans -- is the reverse of what happened here in this election. My impression was that they did the same thing back in 2012 for that poll, but I haven't got a straight answer as to why when they did it, it had either no bias or it had a different directional bias than it did this time around. It might say something about just the change in the electorate overall since 2012. There might be a story there about how people have changed since 2012. I'm not sure.

Yeah, there are so many questions that have yet to be answered with all this stuff. Thank you so much for talking to me. Is there anything else that I haven't asked that you're dying to talk about?

I was sort of wondering whether we would get to the distinctions between the ACS and the CPS and we did, so that's well beyond my expectations -- to this podcast’s credit, that's quite a lot of detail to get into.

One thing I wanted to emphasize to is I made a decision early on when I was doing this that whatever updates I did to the analysis I would publish no matter what they showed. Because I got I got accused by people following me on Twitter of being in the tank for Clinton or being in the tank for Trump because I kept coming back to this poll, or because I kept showing adjustments that had Clinton winning. I had made a conscious decision that whatever the results showed, I would publish and sort of engage with the polling community and just make public whatever I did. Because this really wasn't about “unskewing the polls” to me. Somebody tweeted to me, “this is actually the only honest unskewing I have ever seen.” And I’m like, “yeah I guess literally that's a good way of describing it.”

Look, I'm an American who followed the election closely and I always vote, so I had strong opinions about the election. But my intention here was not to reassure or make a statement about my political priors. My question was -- “this poll that I really like, it's out-of-consensus. I'd really like to know why that is. It would be interesting to know.” That was the very narrow question. I was not even trying to get an accurate read of the final election number. This was not about placing a bet in Vegas on how the election was going to turn out. This was almost like a purely data science question at the end -- figuring out what was going on with this particular poll that wasn’t affecting all the other polls that were being brought out there.

The only prior I brought into this was “I know the reputation of the L.A. Times and the USC institute that's running this. I’m highly doubtful that they’re partisan hacks trying to show support for Trump. It’s gotta be something else that could be really interesting, that could say something really interesting about the election.