One of the most annoying trends I’ve seen pick up in recent years is the public use of polls to craft public opinion. As it gets easier for polling companies to connect to people through their emails, phones, websites they visit, and any other place, anybody and their mother can run a “poll” and then tout that poll as evidence of the superiority of their position.
People on twitter and the media in the age of Orange Man Bad are particularly susceptible to this thinking. You’ll see people trot out polls saying, “Haha, well 71% of Americans disagree with you about [TOPIC]!” implying that that is supposed to be some sick burn. I’ve seen this countless times since the c*ronavirus pandemic started, with the “sick burns” being in respect to public support for shutdown measures.
Let’s forget the fact that appeal to popularity is a fallacy for a moment and just dig into one of the biggest reasons why wide-scale polling falls apart: people are terrible at thinking at scale.
There are two classic examples I come back to here: congress and schools.
If you poll a wide swathe of people about Congress, you find that it always has a ridiculously low approval rating as an institution. Yet it is almost unheard of for incumbents to lose reelection campaigns. In fact, when they do lose, that’s newsworthy.
Similarly, if you poll a wide-swathe of people about whether or not public education in the US needs to be reformed, you’ll find most people do think it does. Yet very few people move their children to different schools.
So something is mis-aligning between people’s stated and revealed preferences. People aren’t lying, they’re just really bad at thinking at scale.
If you poll the same people about their congresspeople, you’ll find that they generally, on average, like their congresspeople. If you poll the same people about their public schools, you’ll find that they generally like their public schools.
Take 2020 for example. If you poll the public at large about support for closing non-essential businesses, you tend to find anywhere from a slight majority to 70% of the public supports the idea. But if you ask people if they would be willing to take a 25% pay cut to support that agenda, the support drops to 10-20% (all depending on polls and times the polls were taken).
“Yeah, congress sucks, but MY congressperson is fine.”
“Yeah, the schools need to be reformed, but MY public school is okay.”
“Yeah, we need to keep people from going to work to keep them safe, but MY paycheck should keep coming.”
People are bad at thinking at scale. Public choice theory is useful for thinking about this. If you spend any time studying it, you’ll find that people can be duped if costs can be spread out over a large enough span of people. So if the cost of a terrible congress is spread out across a large enough group, few people will do anything. If a cost of a terrible government policy is spread out across a large enough group, few people will do anything.
This post has, up until this point, largely been a cathartic commentary on why you should be skeptical of any kind of large-scale polling, especially if that polling is used to influence popular opinion. But there are also implications for anybody trying to build a product or service.
You should dig into the real story behind any set of numbers. Top-down, large scale numbers are only so useful and really can tell a different story than what the end-user or end-consumer experiences. My colleagues Michael and Danielle wrote about that in our Anti-Pitch Playbook here. A top-down analysis doesn’t tell a story. A bottom-up analysis, which is much harder to pull off well, can put the numbers from a top-down analysis in context.
I can always tell during a pitch when the person I am talking to was once in political or management consulting. They’ll start with big polling numbers and move from these numbers to the conclusion that a pain point or problem is big enough to justify their product or solution. But just because somebody says it’s a problem IN GENERAL or AT SCALE does NOT mean that they think it’s a big enough problem FOR THEM to pay for a solution.
Never take polls too seriously. Always remember that just because something is factual, that doesn’t mean that the message it is conveying is truthful. Dig into the story behind the numbers before you use those numbers to try to convey a position.