Is My Advertising Working?

The Solution to Help Answer this Timeless Question

If you’re responsible for analyzing advertising results and making strategic decisions from “the numbers,” a fantastic read to improve understanding of data translation is Donald Wheeler’s, “Understanding Variation: The Key to Managing Chaos.”

As Donald beautifully explains, the odd phenomenon of how data is reported and analyzed, whether it be in businesses, media outlets or governments, is fundamentally wrong. For whatever reason, the norm for data communication is in results that are “good” or “bad,” or a binary manner.

“Profit is up 5% this April versus last April.”

“Year to date sales are down 3%.”

“2Q is pacing 8% ahead of last year.”

This is hyperbole of course, but advertisers and the respective media partners tend to justify the numbers, good or bad, based on the tactics employed. They “attribute” results to the advertising mix, or any changes that might have been made with respect to the mix (ie, “we moved direct mail dollars to digital”). Raise your hand if you’ve been guilty of attributing a successful or unsuccessful sales period to the advertising. However, as I’ll explain further, therein lies the problem.

Turn on the TV, read the latest GDP numbers, or wail over your quarterly performance report. Good or Bad, binary results. The critical issue is that there is no accounting for inherent random variation.

For example, retail stores deal with variables in weather, community events, consumer confidence (stock market), competitive pricing, merchandising constraints, and of course political elections (my favorite). The list is virtually endless. Variation exists at every moment, in every business system and process.

The solution is to start analyzing your data using “control charts,” which inherently account for random variation, and are much better suited to detecting “noise” vs. “signals” and trends.

Below is an example control chart reflecting monthly new patient calls for a health care clinic. This particular clinic uses a variety of advertising to drive new patient calls, including TV, Pay-Per- Click, pre-roll video and digital display. So is it “working”? Let’s investigate…

The blue line shows the average monthly calls (190) and the red dotted lines, known as the “upper and lower control” limits, are calculated three sigma above and below the average. The control limits account for inherent variation.* Any data point within the limits is mere “noise” and your system/advertising is in control. Any point above or below the control limits, or call counts higher than 277 or lower than 105, are true signals, and need to be carefully considered by management as to what in the system changed. Did the marketing change? Did the pricing or product mix change? Did a new competitor enter the market (or exit)? Was it our promotional offer Did the sales staff receive new training? Was it seasonal?

If the results are not up to expectations (or “in control), change the system, which could be anything from your advertising mix, messaging, to internal on-boarding processes. Otherwise, as it’s crystal clear from the chart below, these results should continue as it has in the past, assuming the same plan is implemented moving forward.

New Patient Call Control Chart

Advertising Measurement


We once had a retail client who was concerned the current TV advertising plan attributed to slowing traffic. So when asked to see the historical traffic numbers, the response was “We just know and don’t ask us to get a door counter!” Aside from the obvious issues related to not measuring a basic retail metric such as traffic, using a control chart would have given a clearer picture of customer counts, trends, and if any “concern” was truly warranted.

According to Mr. Wheeler, data apart from context is meaningless. Understanding noise vs. signals and using control charts is critical for advertisers to improve evaluation of their marketing efforts and ask better questions. If you don’t like the limits (upper/lower), then you need to change the process. Otherwise, if the data is within the limits, expect your business and advertising results to behave as such in the future.


*This post will not get into the statistics of why three-sigma limits are the standard for control charts. For further study, please refer to Donald Wheeler’s, “Twenty Things You Need to Know” (SPC Press, 2009).

Schedule a Free Discovery Meeting