Make polls great again

By Shachi Kurl and Ian Holliday

Pollsters and polling organizations are frequently the target of criticism, derision and disdain.

People don’t like the data, so folks get mad.

We sometimes get it wrong, so folks get mad. And that’s fair play.

Most major pollsters really do everything in their power to conduct public opinion research in a way that is accurate, and fair.

And while a lot of time is spent by observers dissecting the methodology pollsters use to collect data, an equally important but overlooked part of the survey process is the way the questions are written.

We can tell you, at the Angus Reid Institute, we take this part very seriously. We sweat over vocabulary.

We fact check our questions. How much explanation is too much?

Have we now created a leading question? Then it’s back to the drawing board.

How much explanation is not enough? Will our respondents fully understand the context around the issues we’re canvassing?

We’ve been on conference calls with colleagues that have lasted hours, trying to get the phrasing just right.

And we publish our survey questionnaires along with the data, so people can see for themselves what we asked and why we asked it.

And then there’s the Trump poll.

Or rather, the “Mainstream Media Accountability Survey,” published on the Donald Trump campaign website Thursday afternoon.

When it started making the rounds on social media, and then around our offices, reactions ranged from horror to humour.

This is, in our humble but professional opinions, a really, supremely, incredibly badly written poll.

But hey, we’re all about education and furthering polling discourse at the Angus Reid Institute, so let’s just turn this into a teachable moment, and walk through all the ways this survey could have been written better.

First of all, there’s nothing wrong with canvassing American trust in mainstream media organizations.

It’s a legitimate subject. One that Pew Research tackles ably every day.

But these questions in the Trump survey?

Let’s start by considering question 5, which asks: “On which issues does the mainstream media do the worst job of representing Republicans?” and then instructs respondents to “select as many that apply.”

How can they be the things the media is “worst” at if we can pick them all?

Question 6 suffers from almost the opposite problem.

It asks respondents from which television source they primarily get their news from, then lists a grand total of four options: Fox News, CNN, MSNBC, and “local news.”

There are other channels, Mr. President.

The rest of the 25-question survey, aside from a pair of open-ended prompts, follows the same pattern:

A yes or no question is asked, and respondents are given four possible response choices: “yes,” “no,” “no opinion,” and “other, please specify.”

Here’s why that’s a terrible response metric. First, it leads to nonsensical situations like this one:

Sometimes it’s good to have a “no opinion” option. Questions about personal experiences are not those times.

The data from these kinds of questions will be totally useless, moreso even than the average unscientific quick-poll on a news website.

Of course, we suspect the Trump administration wasn’t attempting to get academically rigorous data from this survey anyway.

Still, we thought it would be valuable to demonstrate some ways to Make this Questionnaire Great Again.

Shachi Kurl is executive director and Ian Holliday is research associate for the Angus Reid Institute. Read the full article here.

This article is written by or on behalf of an outsourced columnist and does not necessarily reflect the views of Castanet.