Why the Polls Keep Getting It Wrong

October 18, 2018

Opinion polls have been used across many industries from marketing to politics, but recently they are becoming more and more unreliable.


FEMALE_1: [MUSIC] This pile of beans and

FEMALE_1: this chart help explain this,

FEMALE_1: or why polls that are supposed to

FEMALE_1: predict election results keep missing the mark.

FEMALE_1: [MUSIC] To really understand

FEMALE_1: the problems with modern polling,

FEMALE_1: we've got to start at the beginning.

FEMALE_1: In 1936, the US presidential election

FEMALE_1: pitted incumbent Democrat,

FEMALE_1: Franklin Roosevelt, against Alf Landon,

FEMALE_1: a Republican governor from Kansas.

FEMALE_1: Now, up to this point polls have been informal.

FEMALE_1: Newspapers used to gauge

FEMALE_1: voter sentiment by surveying their readers,

FEMALE_1: but it wasn't based on any formal survey methodology.

FEMALE_1: In 1916, a magazine called

FEMALE_1: The Literary Digest started conducting national polls.

FEMALE_1: They sent out millions of

FEMALE_1: postcards to their mailing list,

FEMALE_1: mostly drawn from phone books

FEMALE_1: and vehicle registration records.

FEMALE_1: The 1936 survey was their biggest one yet. Meanwhile.

MALE_1: [MUSIC] From offices of Princeton,

MALE_1: New Jersey, a famous statistician,

MALE_1: Dr. George Gallup, tells Washington

MALE_1: from week to week what the nation is thinking.

FEMALE_1: George Gallup had a background in consumer research.

FEMALE_1: He accurately predicted that

FEMALE_1: Franklin Roosevelt would defeat Alf Landon,

FEMALE_1: the Digest had predicted a Landon victory.

FEMALE_1: Gallup called that too,

FEMALE_1: he knew the Digest would miss the answer

FEMALE_1: because he saw the flaw in their method.

FEMALE_1: Remember how they got their mailing list

FEMALE_1: from phone and auto records?

FEMALE_1: People with landlines and cars tended to be older,

FEMALE_1: more affluent, and more conservative,

FEMALE_1: therefore, the Digest results under

FEMALE_1: counted FDR supporters and

FEMALE_1: overestimated support for Landon.

FEMALE_1: Gallup got it right because he used a method

FEMALE_1: from statistics called quota sampling.

FEMALE_1: He said that, "Sampling public

FEMALE_1: opinion is like sampling soup.

FEMALE_1: One spoonful can reflect the taste of

FEMALE_1: the whole pot if the soup is well-stirred."

FEMALE_1: In other words, you don't have to

FEMALE_1: survey every single voter.

FEMALE_1: You just need to survey the right subgroups of voters.

FEMALE_1: Gallup used beans to explain his method.

FEMALE_1: You've got a barrel of beans,

FEMALE_1: millions of beans, half of them are

FEMALE_1: white beans and half of them are black beans.

FEMALE_1: Now, count out 3,000 beans,

FEMALE_1: those beans are your sample.

FEMALE_1: Write down what percentage of white beans you've

FEMALE_1: got and what percentage of black beans you've got.

FEMALE_1: And repeat the process 1,000 times,

FEMALE_1: what you'll find is that 997 out of those 1,000 times,

FEMALE_1: your sample will have the same make

FEMALE_1: up of beans as the whole barrel.

FEMALE_1: So, you'll have 50 percent white beans,

FEMALE_1: 50 percent black beans.

FEMALE_1: Well, roughly the same,

FEMALE_1: within three percent. So.

Gallup: In every Gallup poll,

Gallup: we include people from all walks of

Gallup: life and in the right proportion of farmers,

Gallup: skilled workers, uh, white collar workers. This is the-

FEMALE_2: If you're trying to figure out which

FEMALE_2: presidential candidate Americans prefer,

FEMALE_2: you want your sample to be

FEMALE_2: the American voting public in miniature.

FEMALE_2: The sample is the key to the whole thing.

FEMALE_2: So, here's where the trouble starts.

FEMALE_2: [MUSIC] Look at this chart,

FEMALE_2: we just don't use landlines as much as we used to.

FEMALE_2: More than 60 percent of us primarily use

FEMALE_2: our cell phones and this demographic tends to be younger,

FEMALE_2: more diverse, and more liberal leaning.

FEMALE_2: The people who tend to still use landlines

FEMALE_2: are older and more conservative.

FEMALE_2: But if you don't get in touch with the people

FEMALE_2: who mostly use their cell phones,

FEMALE_2: you're going to throw off your sample in

FEMALE_2: the same way that Digest did back in 1936.

FEMALE_2: So, can't we just call people on their cell phones?

FEMALE_2: It's not so easy.

FEMALE_2: There's a law against auto dialing cell phone numbers.

FEMALE_2: It's from 1991 and it was supposed to

FEMALE_2: protect us from invasive telemarketing practices.

FEMALE_2: But opinion surveys were designed to be conducted

FEMALE_2: over the phone using automated dialing.

MALE_2: Hello.

FEMALE_3: This is a nationwide radio survey.

FEMALE_3: Were you listening to your radio just now?

MALE_2: No, I am not.

FEMALE_4: Were you listening to your radio just now?

FEMALE_5: Well, yes I am.

FEMALE_2: It can be twice as expensive to

FEMALE_2: reach poll respondents on cell phones

FEMALE_2: because you have to hire and pay real

FEMALE_2: live humans to dial the numbers.

FEMALE_2: And whether we're talking landline or cell phone,

FEMALE_2: there's a societal shift

FEMALE_2: that's endangered Gallup's method.

FEMALE_2: We've been trained by telemarketing and

FEMALE_2: caller ID to screen our calls.

FEMALE_2: We just don't answer random numbers.

FEMALE_2: That means low response rates.

FEMALE_2: In the 1970's, a good response rate for

FEMALE_2: a national poll was in the 80 percent range.

FEMALE_2: Today, it's an abysmal eight percent.

FEMALE_2: That's bad news for polling

FEMALE_2: because the smaller the sample size,

FEMALE_2: the less accurate the poll.

FEMALE_2: Pollsters can try to compensate

FEMALE_2: for low response rates using waiting,

FEMALE_2: but that's another place where

FEMALE_2: things can go off the rails.

FEMALE_2: Let's say you can't get

FEMALE_2: any Hispanic millennial men

FEMALE_2: to answer the phone and respond to your survey.

FEMALE_2: You finally get this one guy,

FEMALE_2: but demographically, you should have

FEMALE_2: three times as many Hispanic millennial men.

FEMALE_2: So, you wait his answer three times.

FEMALE_2: One guy, standing in for

FEMALE_2: the whole Hispanic millennial male population.

FEMALE_2: If he's enough of an outlier that

FEMALE_2: he actually answered the phone,

FEMALE_2: he could be an outlier in other ways

FEMALE_2: including his answers to the survey.

FEMALE_2: If you're just trying to measure preferences,

FEMALE_2: opinion surveys offer valuable insight.

FEMALE_6: As an American,

FEMALE_6: which do you think it's more important to do,

FEMALE_6: keep out of war ourselves

FEMALE_6: or helping them win even if the risk of war?

FEMALE_2: It's when we try to predict

FEMALE_2: human behavior that we get into trouble.

FEMALE_2: While it's easy to get people to

FEMALE_2: tell you which candidate they prefer,

FEMALE_2: it's a lot harder to predict who's

FEMALE_2: actually going to get off the couch and go cast a ballot.

FEMALE_2: Gallup himself had some notable misses.


MALE_3: On election night 1948, Republican campaign leader,

MALE_3: Herbert Brownell and a legion of news men

MALE_3: continued to echo public opinion polls that said,

MALE_3: ''Thomas E. Dewey would be

MALE_3: the next president of the United States.''

MALE_4: We, poll takers, proved in 1948 that the poll [inaudible 00:05:54] fallible.

MALE_4: But at the same time,

MALE_4: we know that no better methods have yet been found for,

MALE_4: ah, measuring political trend.

FEMALE_2: Ironically, he didn't think predicting

FEMALE_2: election outcomes was all that important.

FEMALE_2: He only got into the election prediction game

FEMALE_2: because he wanted to

FEMALE_2: demonstrate the value of his main product,

FEMALE_2: his opinion research surveys.

FEMALE_2: What we're seeing today is

FEMALE_2: it's harder to get a good sample,

FEMALE_2: thanks to changes in how we use the phone and it's

FEMALE_2: just as hard as it ever was to predict human behavior.