Be Careful with Those Averages

When I began reporting PPC and other metrics to C-level corporate officers, I had the good fortune to be reporting to people who didn’t just want the numbers. They wanted to know what the numbers meant. If an average something-or-other went up from month to month, they wanted my best explanation — or my best guess — of the reasons. Was it seasonal? Was it random? Which components of the average went up, and which went down? Did things we were doing to bend the curve on other metrics affect this one?

These executives knew something important: You don’t know what a number means, when all you know is the number. You need to know where it came from, what parts comprise it, what it was last year and last month.

One of the simplest, most commonly used metrics is the average. Practically anyone who can read this post knows that to get the average, you add up all the values and divide the sum by the number of values.

There are also weighted averages, moving (or rolling or running) averages, and more. And sometimes we call the average the mean. And if you don’t mind me going totally geek for a moment, there are a bunch of those: arithmetic mean (what we usually call an average), geometric mean, harmonic mean, quadratic and cubic and generalized means, and truncated and interquartile and midrange and Winsorized means. Each has its uses, but I won’t be explaining them here.

I went all geek to begin to make a point: the humble average is not as simple as it seems. Or in other words, ahem, you keep using that number. I’m not sure it means what you think it means.

Example: Home Prices

To wit — and I’ll simplify a little for this discussion’s sake —

Let’s suppose that in 2012, 11 homes sold in our area. Ten sold for $100,000 each. One sold for $2,000,000. The average sale price was $272,727 and change. Let’s note in passing that none of the actual sale prices was even close to the average, so we certainly should not speak of the typical home in our town selling for that price. (In this case the median would be $100,000 and would better suit our sense of the typical sale price. But the median is a discussion for another day.)

Let’s further suppose that in 2013, another 11 homes sold in our area. Ten sold for $90,000 each. One sold for $4,000,000. (Clearly, the town has some prime lakefront or beachfront property.) The average sale price was $445,454 and change.

In 2014, 11 more homes sold. Ten sold for $80,000, and one sold for $6,000,000. The average sale price was $618,181 and change.

Using rounded numbers, from 2012 through 2014, the average sale price of a home increased more than 126% — more than doubled — from $273K to $618K. Yet for most of the market, home prices fell 20%. For the high end, sale prices tripled — that is, increased 200%. To the extent that the average represents anything in our reality, it is the performance of the high end only — which is very much the opposite of most buyers’ and sellers’ experience in our little market.

Credibility and the CPI

When the news story hits that the average sale price of a home more than doubled, for most people who know the local market it will have no credibility, because it doesn’t resemble their own and their neighbors’ experience. Then it will become just another discredited statistic, like the US inflation rate based on the Consumer Price Index.

We have been treated to story after story about how inflation is minimal over the past several years, yet virtually every household has seen its cost of living soar. The CPI is basically a weighted average, so in theory the weighting could be wrong, but in this case the problem is more fundamental: essential commodities with soaring prices, like food and gasoline, are not included in the average.

Using obviously flawed numbers tarnishes credibility.

A PPC Example: CTR

There’s a little average trap built into PPC metrics. People who know just enough to be dangerous like to fixate on the overall click-through rate (CTR) of an AdWords or Bing Ads account. It’s not a very useful number, if the account includes both display and search campaigns.

It’s like our real estate example: CTR only makes sense if you look at search and display separately. The reason for this is quite simple. In a typical account, a good display CTR is about one-tenth of a good search CTR. Put another way, a CTR of 0.5% is terrible in a search campaign, but good in a display campaign. Users simply act differently with respect to search and display ads.

Here’s another way to look at it. If you’re expanding your display campaigns rapidly, it’s entirely possible for the CTRs of both search and display campaigns to be higher than they were last week or last month, while the account’s overall CTR is lower.

If you’re running both search and display ads, the best thing you can do with the account’s overall CTR is ignore it.

Keys to Average Safety

The examples suggest these precautions for handling averages safely:

• Average only things that are alike, if you plan to use the average as an indicator of a typical case — but average them all. (Don’t be like the CPI.)
• Subdivide your data, to test the significance of your average. In both the real estate and CTR examples above, the averages are quite meaningful if you separate high-end and low-end home prices, or search and display CTRs. They’re not very useful if you don’t.
• Learn to calculate and evaluate a median. Often it gives a clearer picture of a typical case than an average does.
• Be as interested in understanding what a statistic doesn’t tell you as you are in understanding what it tells you.

Observe these precautions, and you’ll be — sorry, I can’t resist — well above average.