Interview with Jill Darling, Survey Director for USC’s Understanding America Study

Our recent episode Opinion Polling for Presidential Elections featured segments from an interview with Jill Darling. This post contains the full transcript for that interview conducted by Christine Zhang. The player below also contains a link to the full recording from the Data Skeptic Bonus Feed.

Can you walk me through the main factors that made the [L.A. Times/USC] poll so different from other polls?

Sure. Our poll was very different in a lot of different ways. It was an experimental poll. It had been piloted briefly in 2008 and then again for a whole pre-election season in 2012. This was basically its second outing in a full election season. The reason that we call it “experimental” is because it really does use techniques that are different than the sort of “tried-and-true” polling that everyone has been doing forever, which is often telephone-based if it's done right. It's maybe RDD -- in other words, random samples of telephone numbers -- that interviewers call to and screen for whether or not they're registered voters and then also ask some screening questions about whether or not they are going to be likely to vote in the upcoming election and then ask a normal categorical question of “if the election were held today would you vote for candidate A, or candidate B?” Generally if people are unsure, they might be leaning. “Today, are you leaning more towards candidate A candidate or candidate B?”

So that's the traditional way of doing polling: you try and find out who's a likely voter, you ask them who they're voting for, and you publish the results among either all registered voters if it's too early to really do a good job of finding likely voters, or among likely voters when it gets closer to the elections. So that's kind of the gold standard and has been. There's a lot of other ways that people do polling, including using registered voter samples, which are prone to having some problems, and then there might be over- or under-reporting of people who are in them --

Is this the voter file?

Voter files, yeah. A lot of political operatives the campaigns use these voter files because it allows them to really categorize people into people who vote all the time, people who vote only in presidential elections, etc. There's a lot of different things you can do. Nationally, it's a little more problematic because not all states keep all of that. So anyway, these are the sorts of things that have been part of traditional polling. More recently, there are online polls, some with panels like GfK’s, which started out as a big probability panel. It is still huge -- it's a huge panel -- not so much as a probability anymore as it was. And then there are SurveyMonkey polls among people who take SurveyMonkey polls. And they're very large and a lot of the researchers who use these kind of scraping polls like that one is try to weight them to be representative of the voter electorate. So there's a lot of different ways to do it. Really the RDD -- the random digit dialing -- is still the gold standard for a traditional poll, or being done through a panel such as ours where you have the ability to say that you have a probability sample. We know the probability that any person was selected to be in our sample, which allows us to use the statistics that we use to do the estimates.

So for ours, where we depart from that -- we have the gold standard sample -- and then we have a different way of going about asking the questions. We use what's called a probability method, which means that we ask people to -- rather than the categorical “if the election were held today, who would you vote for?” We asked them to tell us the likelihood that they're going to vote in November, so that's a 0 to 100 percent likelihood that they're gonna vote, and then we ask them to tell us on a also 0 to 100 percent scale, the likelihood of voting for that candidate. So there's a 0 to 100 percent likelihood of voting for Trump, Clinton, someone else.

Rather than having a likely voter screen, we used the information that people give us about their own likelihood of voting -- their prediction of their likelihood of voting -- and we multiplied that by their percentage vote for the candidate. And that gives us some sort of a probability of voting and we use that to cast our forecast estimate of the percentage vote. The real difference that you get from this is that rather than having just -- for example, if someone is a Trump voter, in a categorical traditional question, they're either going to be a Trump or Trump leaner. In our way of doing this, they might be a 60 percent Trump voter, a 75 percent Trump voter, a 100 percent Trump voter, but they're going to be perhaps changing over time from a 60 percent Trump voter to a 100 percent Trump voter if something happens. You know, if they attended a rally and they're all fired up now and they’ve gone from sort of tepid or not really sure to really thinking this is their guy. And if enough of those accumulate, you'll see those changes in our estimates, like a fever chart or something over time.

So we have, in that way, kind of a way of quantifying the level of certainty … quantifying uncertainty. [This is] particularly important, I think, during early days of the campaign, when people are still getting to know the candidates and still making up their minds. So that's one really significant difference.

Actually, two, because of our panel as well.

Because you asked the same panel of people?

Right. We recruited from our six thousand member panel -- it wasn't six thousand when we started, but we were actually building the size of the panel over that summer --

This is the Understanding America Study?

It's the Understanding America Study at the Center for Economic and Social Research at USC. Our panel was not created to do this survey. It's a panel that we use to do academic, policy, government and nonprofit and also private surveys for all these various reasons. Generally soft money/grant money-funded surveys. So that's what generally goes on, and since we're the Center for Economic and Social Research, a lot of our research is on economics, financial understanding, and health. This was a sort of an opportunity for our panel members to not only participate in some of those -- forgive me, dry subjects -- but for the ones who are citizens to volunteer to be in our election panel. So we recruited as many of those folks as we could. We got more than four thousand people who said they would participate every week. And that group of people were assigned randomly to a day of the week. So they knew they were gonna participate, for example, on Tuesday every week of the run-up to the election from July through November.

And so we would send them the invitation and they would login anytime day or night at their convenience. They knew it was gonna happen every week, so we had really good participation. They would answer just those simple questions, plus one more, which was “what percent likelihood do you think that these candidates will win?” We were interested to see if that might be somewhat more predictive or give us more information if they were predicting the chances of the candidate to win.

Was that a question that was asked in other polls in your experience or knowledge?

Well, we had that as part of the sample in 2012 and when I say we -- I wasn't with that team. I joined the Center for Economic and Social Research last year.

But this team that conducted the 2012 poll together when they were at RAND. So the way that we did it this time, including those three questions, were exactly the way that they did it at RAND at 2012, when it was extremely close. I think it really was the very closest in the final estimate of all the poll there are. There are published accounts of it. FiveThirtyEight.com had up. I think the last one they took was three days out or something, but when you look at the very last one, they were very close. So we had a lot of confidence in this method. It performed very well under pressure. It was criticized at the time for being Obama- centric -- for being too-Obama. But it turned out that Obama did win and so they were vindicated at the time.

For our poll this time for the 2016 election, we did the same process. We asked people to participate every day, and then we posted on our chart the previous seven days. So if you think about how we distributed our sample over the seven days of the week, if you look back any seven days over the previous seven days, you're going to have a rolling sample basically. What that means is that we have a very slow unfolding of information over time. So if there's a fixed event, for example one of the conventions, or if there's something that happens that's newsworthy, that may change people's feelings about it, you're gonna see that reflected in our numbers slowly because obviously looking back seven days, the first day after the event you have six days before the event, and then so on. Three days in, when everybody else’s polls come out -- the faster telephone polls -- you'll see their reaction immediately and for ours will have three days of post-even and foir dates of pre-event.

So we often caught a lot of flack based on just kind of that issue. That people would be like, “oh, there’s a huge surge for Clinton.” People would be coming out with six or fourteen point Clinton leads. And we’re trailing along with Clinton coming up slowly, but maybe Trump still ahead. So we did catch a lot of flack partially that reason. And also, we did put all of our data and our methods online, so we had researchers and analysts following along in real time with us and the estimates we were creating, including Nate Silver's group, who decided they were going to just basically put a plus four or plus six -- I can’t remember which -- Clinton on top of our estimates because they disagreed with our weighting.

We had folks at the Washington Post and the New York Times who were also doing analysis and a couple of other folks who were as well.

Were you in contact with them when they were doing this at all?

Well, no. The reporters would contact us when they were about to run a story, and just sort of get our feedback. Or, you know, check to make sure that what they were saying was something that we would agree with. So we did hear from time to time, and we of course followed what they were doing in the news, because we just had a lot of input from a lot of people and people sending us whatever was happening. And once we really became cognizant of what really was happening on Twitter we followed it there. You know, we were aware, but we weren’t in touch.

I wanted to come back to a couple points that you mentioned. You said that you recruited this group of people from your larger Understanding America Study. Did you offer them any incentives to participate?

As part of the [Understanding America] panel, we pay them you know basically for their time. In doing the surveys, they're giving us a lot of information, and we pay them -- not a large amount -- it ends up being you know like for a ten-minute survey they get seven dollars. It's not a ton. But for this one they were paid two dollars a week, so that was their incentive. We also had some other surveys that we did while we were going along. We did a survey around the conventions, we did another survey sort of in the middle looking at some issues of immigration and trade, and for those they paid according to the length of time that that survey took. But they had just sort of a regular two dollars a week from participation --

In addition to the other surveys.

Yes.

Was that the same as what they had done at RAND in 2012, do you know?

I don't know what their incentive structure was there. I do believe they did pay incentives, but I'm not familiar with that.

It seems like statistically your methodology was the same as in 2012.

Very similar. Our statistician would insist that I say “similar.” There were a couple of differences. I think those differences had to do with some technicalities in the weighting and estimation. But yeah, in the main, it was identical.

One main argument that I've read from from the media about your poll was that it weighted for how people said they voted in 2012. There was a thought that people would lie about the way that they had voted, or would say that they had voted for the winning candidate.

Here's the thing. It’s either that they're lying or they don't remember. So that would be measurement error. Or, when we go back to sample people, people who don't participate in elections may be less likely to participate in election surveys. So there's two schools of thought, and they require two entirely different approaches. If it's a sampling problem -- in other words, we don't have those people in our sample (that means there's a coverage error) -- what you do with coverage errors like that is that you apply weights that compensate for that. So we make sure to weight all of our samples to account for any biases for education, age, all of the typical things that everyone does in surveys. And then for something like this, if you come down on the side of it being sample error, you would also need to apply that weight to account for the fact that we don't have those folks in our sample.

There's not a lot of evidence, but there is some literature to show that people do differentially respond based on 1 -- interest and 2 -- sensitivity or issues with the subject. In particular, around elections, if there's an event that happens, you're going to have differential response based on whether that event was positive or negative for your candidate. If it was negative for your candidate, people are less likely to respond to a survey about the event.

So with those things in mind, we sort of came down on the side of sample error, as opposed to the other side, which is where everyone else basically came down on, which is that it's measurement error. That people are either lying or they’re mis-remembering or there's just sort of a psychological tendency to remember … like, if you really like Obama now you're more likely to remember that you voted for Obama. We don't really have a ton of evidence about this. We do know that there is drift. We don't really know why that drift occurs. So we need more research on that. That's definitely going to be a huge focus for us in subsequent elections that we do.

But for the moment, our statistician, who is an economist and a professor at USC, looked at the available evidence and I realized that if we came down on the side of sampling error, it would not only deal with the potential for differential non-response to the survey, but also differential non-response to any particular week, when you might have this issue come up where there's an event that's negative for the candidate and so therefore we might have some differential non-response among our panelists who had agreed to participate, that it would also account for that as well.

So that's what we did. We had to stretch Republicans who voted for Romney in the past, because we had an over-representation of Obama voters and an underrepresentation of non-voters. We stretched the Republicans and we stretched the non-voters. Those two things combined, because more 2012 non-voters tended towards Trump, they stretched the Trump numbers. And if you didn't do that, you got numbers that looked a lot like the numbers that everybody else was getting at that time, which all had Clinton ahead the entire time.

So that was the main point of contention between us and other methodologists, including Nate Silver. And we just basically said, “we hear you. Here's our justification. This was the decision that we made, and we're not gonna change it midstream. This is what we're doing, and we'll see how it works out.”

In the end, we really wondered -- because we had this overstatement of the popular vote for Trump … was that [the 2012 weight] really the problem? We are finishing up our analysis of this now, but what we really have found is that no, that was not the problem.

The problem was that we had an overstatement of [rural] voters. We had a bias towards [rural] voters that we were not aware of. That ended up really being it, and what we’re looking at now is the variety of models that we’re applying. And it really looks like what we're going to be doing is adjusting based on zip code population, which should take care of the issue for us.

It's not a typical weighting variable. And there hadn't been anything previously to tip us off to this bias in our election sample. I mean it's really sort of interesting that we had that rural overstatement stains. We're we're looking into where it came from and what its sources are.

But the bottom line is once you make that adjustment, you end up with a one point Clinton lead or a two point Clinton lead depending on which model you use. One model adjusts based on Census designation of whether or not it's urban or rural and the other one is based on just based on population and zip code.

Once our statistician is satisfied with the models, we have the 2012 data that we can look at, and hopefully we can run a comparable model that we had for our estimates against that data and make sure it’s still working. We now have to election samples to work with, so that's what our statistician is working on. He’ll publish that data later this year.

...

As a team, we reviewed use descriptions and you know basically Eric has final call on the decisions that were made. The ones that we did that were controversy were the 2012 [weight] and the other was that he did not trim the weights. Not trimming the weights gives you the ability to actually get the estimates lined up along all of the population estimates that we weight against. So it allows you to make sure that you have no bias in any of those estimates. But what it does mean is that you can have sub-groups that have really large weights for sort of the unicorns in the sample who are standing in for a lot of people. And so that is a controversial weighting decision that we will also review and look at what happens if you don't do that and what into the impact on our final estimates are. And also on the error that we also take into account -- no one else takes into account -- we look at the error that we have to think about in terms of the confidence intervals for our forecasts and the impact it has on those is part of what his concern was.

These are all things that we are reviewing, he is reviewing. Like I say, he and Ari will publish our about later in the year.

That'll be interesting to see for sure. With the sub-groups being weighted a lot like, it's can be issue in general with sampling, right? Because if you have these intersections of all these demographics -- race, gender, age, etc. -- then probably the more intersections you add, the fewer people you have that can be yeah representative.

Yeah, and if you trim the ways you do keep any individual weight down, and there's a big argument for that. I mean, looking at it from the point of view of just non-statistical but just looking at population, do you want one guy to stand in for fifty people?

That's the question. And generally speaking, survey researchers say, “no, you don’t.” And they keep those weights trimmed. But then what you do end up with is that you is that your sample may be really short on, you know, young black people who are Republicans or something along those lines. You have some intersections that you are not necessarily weighting on that cell, but it's the result of the weights that are twisting these cells. We try and keep the categories as large as possible and the number of cells as small as possible.

The main thing that we have to think about for next year is that these statistical issues are important, but when it comes down to it, these are choices that you make based on the trade-offs.

But I think that where we really fell down was in not really thinking through how to communicate the results that we were getting. And if you make a decision, for example, to not trim your weights, so you may have these very large weights in sub-groups, then maybe what the trade-off for that is that you don't publish subgroup numbers. So that's something that we're gonna have to think about. If what we're aiming at is only really getting our overall estimate right, then putting out sub-group numbers may be problematic.

Oh, I see. Because the sub-group is made up of, say, like --

One guy who has a weight of fifty.

One nineteen-year-old Illinois man who votes for Trump, according to Nate Cohn, anyway?

Which I have to say that was --- that was a moment he seized. I can't blame him for seizing the moment. It was really distressing for us that he identified that person as much as he did. We have a data use agreement that precludes anybody using our data to identify someone. My biggest concern was that that might happen, or that he [the respondent] might recognize himself. We were very concerned about that. We talk to our IRB -- our research review board -- to think through the ethical implications and think about if we needed to do anything. But there hadn't really been any technical infringements. It just really was -- it was just unfortunate that he chose to do that even though we asked him not to.

Not to identify the panelist?

Not to give so many identifiers. He didn't need to say as many identifiers as he did at once. We also objected to his mis-representation of the impact of that, because the whole point of those weights is to make sure that our overall estimates were being fairly cast, unbiasedly cast. And they were not moving the aggregates that people were doing like Nate Silver's aggregates. It was not moving those.

If the USC/L.A. Times poll was a big polling outlier, it would be would be having an effect on the national polling average, wouldn't it?

Yes, it would. But that's not the fault of our one nineteen-year-old guy with the large weight.

The reason that we were as much of an outlier as we were -- there's two things. One is that we did detect many more movements for Trump among people that I really truly believe were not in traditional polls. And we saw those aggregate movements, like I was describing earlier, in the way that people consolidated support for Trump or moved away in relation to events that I think were not necessarily caught [by other pollsters]. Also remember, in traditional polls, research organizations do them every now and then. They do one now, or they one three weeks from now or a month from now. And then you look at those numbers in the media that move from a month ago to now. Whereas in ours, we were looking at daily changes over time. So we just detected a lot more movement up and down than you would detect doing it on larger increments. There was a lot of comparison of apples and oranges there.

What I'm in the middle of analyzing right now is, “how much can we say from what we know in our panel, where we had the same people over time, how much can we say about what we could see about differential non-response?” I mean those are the really interesting things here, because those would be affecting everybody’s polls. They may have been working on differential non-response that was suppressing Clinton voter participation. They may have been working in a situation where people were less than comfortable saying that they were voting for Trump. There could be many of these factors depending on the type of poll and the organization that was conducting it and their sampling methods and all of those things.

When you see that the USC poll captured a lot of people who weren't previously in polls, are you talking about rural voters that you mentioned earlier?

We had to two groups that fall into that category. One is that we did have the 2012 non-voters entirely represented in our poll. They told us what their likelihood of voting was in this election. Overall, that group’s average likelihood of voting was much lower than the people who voted in 2012, but it wasn't zero. It was more like fifty percent on aggregate. So we had that group of people who was participating.

We had rural voters and people who were able to participate in our surveys at their own convenience. We give people who don't have internet connectivity and people who do not have computers an internet-connected tablet that they use to take our survey. They can also use them for their own purposes, but the trade-off is that they agree to go ahead and take our survey. So that way we make sure that we have coverage of groups of people who are hard to reach and also never going to show up in anybody else's internet poll.

And that group really are lower socioeconomic status. They’re more rural. They were voters that are more likely to be disabled or have continued to be unemployed after the downturn and that kind of thing. These are folks who are in some of those categories that you see in those swing voters from Obama to Trump, who are looking for somebody to help.

So all of that analysis we're in the middle of finishing up right now. We will be presenting that information and publishing it as we go along this year as well.

I’m sure you saw that Ernie Tedeschi posted on his website a re-weighting of the L.A. Times poll. I think the one that he took out was the 2012 vote from the [weighting]. There were a lot of things, but that I think that was the main change.

Our statistician took what he did and replicated it for us to make sure that we saw what it was and that we believed it, too. So he did both Ernie Tedeschi’s and he did Nate Cohn’s weighting. He took their both of their approaches and re-weighted our data to take a look at it to see if he could replicate it in the same way. So we saw what they were doing, and we're looking at the impacts of these.

What’s really interesting about a lot of this is that each one of these analysts has a model based on their favorite way of doing this and their own sense of what the most accurate way of going about creating these models for estimation are. And in each case, you get a different result.

Their results, by taking out the 2012 voting, were much more Clinton. And what we saw is that when we were taking that out and also adjusting for urbanicity, you end up with a big overstatement for Clinton that just doesn’t compute.

We saw what they were doing, we were following along --

Do you still believe that the rural versus urban was the main issue, rather than the --

That’s what we're seeing. Yes. That's what we're seeing in our post-election analysis of this data. When you adjust for the rurality overstatement, leaving the 2012 vote intact as we have it, we have a one or two point Clinton win.

And that’s better than doing it either Ernie or Nate’s way?

Well, I don't know. Because if you did it either Ernie or Nate’s way and you adjusted our data for urbanicity, you're gonna end up with a large overstatement for Clinton.

But if you only adjusted for the the 2012 vote without adjusting for urbanicity --

If you do not adjust for urbanicity, and you just take the 2012 vote out, it looks good, but you still have that rural bias in the data. And that rural bias is gonna bias other things as well. You can’t have this bias just sitting there and say, “oh well, here superficially we’ve corrected for it by the 2012 non-voters.” You still have that underlying issue there, which is going to affect other things about issues and and decision-making policy and attitudes.

That’s never been a weighting factor, though. Is that what you’re saying?

It hasn’t been, for us or in the traditional polling that I did for the Los Angeles Times, we did not have that as a weighting factor.*

So do people just generally not adjust for urban-rural stuff?

I mean, it's hard to do it. You need to get people's zip codes. You need to know the underlying sample and its zip codes. You don't have that in a telephone poll. In other words, there is a lot of information that's missing in order to be able to do that in a telephone poll. But because we have a sample that we know pretty much everything about -- we know of the zip code of the home ... we don't publish that data but it's part of our underlying information because that's how we select our sample is by households in the United States it's a random sample of households from zip codes across the United States. So we know their zip codes and we can then you know use other data that is available by zip code and by other means of identification -- geo-coding and other things -- to identify areas.

So we have that information and it's sort of a luxury that a lot of pollsters don't have. So it's not easy to weight [by urbanicity]. You would have to do it in in an overall categorical manner where you would look at it and ask everybody their zip code, figure out what you had in terms of rural, urban, suburban, and then weight that on a national level and just those like those four categories to make that adjustment. I don't know which units do that and don’t. There may be some that do. They may not find it necessary if you compared their telephone samples. It may be an artifact of our internet sample.

That's a really interesting finding, and I’m definitely looking forward to the report. Do you know when it'll be out this year?

Maybe your podcast can put a little pressure! It takes a long time to do this. The analysis is quite a bit and then you need to write up the manuscript and then you need to find a journal and, you know, the process takes a while. So we're hoping it'll be this year.

Academia is definitely not journalism.

No, it’s definitely not.

It's hard, during and after an election. People want to create this narrative and the data that are necessary to create a better-informed opinion are generally not available when you want them to be.

There is some data that we're providing. I've got two presentations at the upcoming American Association for Public Opinion Research (AAPOR) conference, and Ari and I are going to be presenting at the European Survey Research Association.

So we're going to be presenting preliminary results from this analysis that don't have to wait for a formal publication for people to be able to cite it, but that we can communicate to our fellow survey researchers for their own thinking. They can come and work with us and collaborate and think about how to move forward. Because that’s really one of our main aims here -- to think about whether or not the the methods that we were using, which include all of these pieces that we've been talking about, if they are a useful contribution to creating more accurate forecasts of elections. Because I really don't think that the desire to forecast elections is ever going to go away. It's a human to human need. Thinking about how to how to more accurately model pre-election sentiment ...

I think that getting away from the idea that every estimate at every point in time is the forecast for what's actually going to happen in November is one of the main things that if we can change that narrative … because I mean it just doesn't even make sense you're looking at it you know two months out. We have no idea depending on you know what happens between now and and two months from now how voter sentiment is going to change.

And there are so many things that can affect it. So to say that you know you look at the data two months out to give a forecast what's gonna happen in November -- you can talk about where voters are at this point in time and focusing on that rather than these horse races … I don’t know.

I sort of doubt that there’s any way to take the drama down a notch because that's what everybody relies on. The media relies on it to sell commercials and to sell subscriptions and newspapers and all of the things that the media requires to be able to exist.

And so there really isn't any incentive to really look at this in a less dramatic way. But what we can do is we can think about on how to try and do a better job of modeling how people are feeling and and what they're doing, rather than the very simple kinds of things that we've been doing all along.

Our center actually has been looking into the idea of putting together a multidisciplinary way of looking at what democracy looks like in America now, what people's attitudes are about democracy and elections and civic life and what it means to be in America now. And I think that a lot of those things used to be taken for granted as sort of the underlying structure for how elections are run and how they take place and how people react to them. And I think that a lot of that underlying structure was really called into question this year, and that's something that I think is important for us all as researchers to try and take into account as we do these measurements.

I don't it's the same old world where we can just ask “right track/wrong track” and a categorical question and go from there. I think that we all need to dive deeper. And I think that people really are. Survey researchers are really thinking hard about this and other you know people who are working in related fields of political science and sociology and communications are all thinking about these issues.

I think you should get a lot of credit for making the microdata and everything available beforehand.

Making the data available to other researchers is very valuable. It is uncomfortable --

So next time, you would still do the same thing?

Oh, sure. We may we may want to do it under some kind of an agreement that makes it much clearer what is okay to do with the data and what is not. We do have everyone of course sign a data use agreement that says that they will not reveal people's identification, they will not try to re-identify people, etc. We have a general standard data use agreement. We’ll think about strengthening that. We may think about perhaps partnering with some analysts, where we would have maybe some representatives from a variety of other organizations -- media, policy, who knows … these are some ideas.

There’s also a concern about about misuse of data in this era of playing fast and loose with analysis and with facts. Our data was quite misrepresented by certain organizations, and we just assumed that people who consume information from those organizations we're doing so at their own risk.

We had our own releases of data, we had our own media partners. And in general, the media treated the data carefully and fairly, with some minor exceptions. But there were some real misuses of it. So all of these things are things that we will be thinking about.

On the other hand, there was a conflation between the actual election outcome and which candidate was ahead in your poll afterwards.

Before [the election], we were completely pilloried as being the outlier -- wrong. And that was not right, either. And then we were completely celebrated afterwards as being the only poll that got it right. And wasn’t true, either. It’s crazy.

So I'm just saying if you wanna complain one, you should complain about the other, too. The important thing is to get the story right.

Yes I absolutely agree with you. And throughout all of the interviews that Ari and I did after the election, we kept making the point that all the other polls were not wrong, number one, and number two, we did not get it right! We were measuring the popular vote, and we had Trump ahead. While we feel that a lot of the things that we measured were valid and strong, we did not nail the outcome, which, you know, anybody that’s trying to forecast an election, you have an actual outcome, and you want to nail it.

Okay, last burning question. This is something that I wondered as I was looking at the poll and at what you would call the “horse race graphics” throughout the election. The election forecasts -- basically, the “who would you vote for?” question had really different results -- the opposite as the predicted winner like question, which was “who do you believe will win?” Why was there such a big divergence? I.e., people said that they thought that Clinton would win, but then people said that they thought that they would vote for Trump. What's going on there, in your opinion?

Well, remember -- the whole narrative in the country was that Clinton was going to win. The only real burning question was, “by how much?” If the entire narrative in the media and in everything is that Clinton's going to win, the voters have that also. They're affected by that as much as the pundits and the pollsters. We had people over and over saying that there was no way that Trump could ever win [the Republican primary], and then when Trump became the Republican candidate, there was a lot of, “well, he can't win a general election.” There was just this ongoing narrative. So of course people absorbed that.

Because honestly that's what you're told every day in every way.

That's really interesting.

There had been some previous research indicating that people's idea about who was going to win might be at least sort of a glance into something that might be more accurate, but I really do think that particularly because now the modern world of elections is so saturated

with this information in a way that it didn’t really used to be -- we used to have periodic interested in poll-related information -- and now with tracking polls and forecasts that are going on and all of the things where there's information every day and you have Twitter and you have 24-hour news cycles and you have polarized media who are playing these up to their bases -- I think that that measure may be less useful. But it's an interesting measure still.

In hindsight, would you present the graphics the same way that you did?

Well, no, I really think that we're going to want to think about what's a good way to really present the data that we're collecting. Is the best way to have a seven day lookback rolling average? Is about the best way? Are we going to do sub-groups? I mean, all of these things we can now really think through. I think that this was once again, “let's just see if it works this time. It worked in 2012, let's see if it works this time.” We really were a little naive about the level of attention it would receive. It didn’t get that much attention in 2012.

So we were were a bit naive about that that. And then, this time we’ll really get some input from people who can help us think about the best way to communicate the information and be a little more professional about it, I think, rather than being kind of like a little academic backwater that we thought we were.