When an organization sets up a measurement program, the conversation will inevitably come around to: “Do you know any industry benchmarks we can use to set our goals?”
My answer is invariably, “None that are valid.”
I know, because I used to publish them.
You can’t compare tanning lotion to tanks
A decade or more ago, we published the Delahaye Benchmark study. We gathered all our clients’ results into a database to come up with an overall average for positive, negative and neutral, as well as message content.
I won’t keep you in suspense: It was invariably 3-5 percent negative and 15-20 percent positive, and 20 percent of all articles contained one or more key messages.
The problem with this approach is that it’s like blending vintage Bordeaux, tasty Chilean Merlot, perhaps a little spicy Gewürztraminer and some robust Napa Valley Zinfandel. Despite your fine ingredients, you get a concoction even the least discriminating wine drinker would call undrinkable.
You can’t combine a lot of good stuff into one bucket and expect it to still be good. That’s exactly what you get when you try to benchmark with an average of results across many different industries, cultures and geographies.
As it turns out, results in Asia were 10 times more positive than they were in parts of the world where the press is freer and much less polite. Needless to say, the percent of negatives was significantly higher for chemical companies than nonprofits. And it’s much easier to get messages across for beauty products than for fighter jets.
Why does everyone ask for these homogenized benchmarks? All it would take to destroy the credibility of the PR person using these numbers would be one good data analyst asking a few pointed questions.
People want to know where to start
Start with benchmarks your senior leaders will believe. Who do they worry about? What competing organizations keep your CEO up at night? Study these organizations for a month or two-preferably a quarter.
If you’re at a utility or a government agency that doesn’t have direct competitors, study your results for three months. That will establish where you are, how fast data shifts and what reasonable goals are.
Five clips doesn’t make a benchmark
In another case, someone asked my team for a benchmark against the competition around a product launch. The trouble was the client’s product was mentioned in only about five clips out of 50. Only one was positive. It would be accurate to say 20 percent of the coverage was positive during the competitive launch period, but it wouldn’t be meaningful.
As with any data set, you need enough content, survey responses or data to pass a common sense test.
Inconsistency, thy fruit is madness
Another major problem with most benchmarking efforts is inconsistency in methodology. When we were initiating a standards-compliant measurement program for a client, we thought we could use the prior year’s data, which the client’s PR agency had collected and coded.
Unfortunately, the agency had completely different definitions of positive and negative from ours. Pretty much anything the agency did was positive. Had we used the agency’s data, the benchmark would have been unrealistically high.
A rule of thumb: If you don’t know how the data were collected and coded or you can’t replicate that process, you can’t use it as a benchmark.
Three rules for better benchmarking:
- Make it relevant to leadership.
- Gather enough data to make the analysis valid and meaningful.
- Be consistent in the methodology.
A version of this article originally appeared on The Measurement Advisor.