4 ways to misinterpret your measurement

It’s tempting to twist data into the results you want, but doing so won’t help your organization get ahead. Beware these measurement traps.


At Paine Publishing, we keep a supply of data geek jokes for when we need to get through tough number-crunching days.

Our favorite is: “Data are like political prisoners: Torture them long enough and they’ll say anything.”

Now, we’re all for deep data analyses, and there’s nothing we love more than cross tabs. But unfortunately, as we know, torture doesn’t always lead to truth. It’s no different with manipulating measurement data. There’s a fine line between data analysis and misinterpretation.

Download this free white paper to see how your organization can better measure its internal communication strategies.

Here are four ways you may be mishandling your data:

1. You’re asked for percentages of small subsets of all the data.

We once delivered a launch report to a client that included data from about 250 social and traditional media items. It showed that the desired messages appeared in about 7 percent of the coverage, which isn’t bad.

But it wasn’t what the client expected or wanted.

The client asked us to redo the analysis after eliminating all social media, arguing their outreach focused on traditional media. That reduced the data by half. It was still an acceptable amount of data to analyze under the circumstances and, as it happened, the data didn’t change that much. The percentage of items containing one or more key messages went up slightly to 10 percent.

But that still wasn’t acceptable. The client asked us to include only top tier media. That analysis produced a much higher percentage. The client was finally able to proudly declare that one-third of all coverage contained a key message.

The problem was for all the effort put into the campaign, the data set was now so reduced that only three media items contained a key message. When you put it that way, the campaign probably won’t get much praise from the corner office.

Lesson 1: The person you’re trying to convince is probably a numbers person, so your numbers better stand up to careful scrutiny.

2. You’re asked to use multipliers.

In organizations that still believe impressions count (no matter how they’re counted), inflating impressions is a common problem. For some reason, people want to “add weight” to media outlets or story types.

The right way to do this is to develop a custom index you can track.

The wrong way is to multiply impression counts (probably flawed to begin with) by multiplying those from top tier publications by two, three or whatever number someone dreams up. Don’t do it. The Institute for Public Relations published a great paper by Mark Weiner and Don Bartholomew on why you should never use multipliers.

Lesson 2: Your top-tier list should reflect the degree to which it reaches your target audience. That’s why it’s called “top tier.” Why do you need multipliers if you are already reaching a high percentage of your target audience?

3. You’re asked to do a benchmark comparison against the competition, even though the circumstances are vastly different.

Comparing your organization to the competition is a powerful and persuasive way to report results. However, to be truly comparable you must make sure you compare the same media outlets in a similar time frame and in the same geography.

If your competitor launched its program the day Princess Diana died, it was lucky to get any coverage, and it’s hardly a fair comparison. If you don’t have the competition’s full data set because its launch or activity exceeds your data’s time frame, it’s not a fair comparison. If you are launch into a mature market, but your competitor was the first brand to market there and had to educate the audience as well as promote its products, that’s not a fair comparison, either.

Lesson 3: Compare apples to apples, or else you’re just blundering about in the fruit aisle.

4. You are asked to include only “the good news.”

If someone gives you a bunch of data collected from agencies that may or may not have been clipping consistently with your methodology, you are treading in dangerous waters. Additionally, if you are told to show just “the good news,” you had better expect alligators.

Lesson 4: Beware unknown waters. Also, if someone wants you to ignore the bad news, there might be something hiding in there that could bite you.

Katie Paine is CEO, publisher, consultant and founder of Paine Publishing, where a version of this article originally appeared.

COMMENT

Ragan.com Daily Headlines

Sign up to receive the latest articles from Ragan.com directly in your inbox.