You probably know that testing your direct marketing campaigns is important, but do you know how to test or what to test? If not, don’t worry! In this issue, our industry insider offers up an easy-to-understand overview to help you get started.
Questions about segmentation? Read Peter’s previous post here.
No matter how successful we are as marketers, we may find ourselves wondering, “Could our results be even better?” Luckily for us, there’s a surefire method to evaluating the success of any campaign. That method is (drumroll, please…) testing!
No doubt, that’s why direct and data-driven marketing guru Denny Hatch, in his invaluable Ultimate 85-Point Marketeer’s Checklist, quotes Malcolm Decker’s unequivocal dictum:
“In direct marketing, two rules and two rules only exist.
Rule #1: Test everything.
Rule #2: See Rule #1.”
Denny also wisely recommends to, “Always have tests in the works,” and he backs this up by suggesting marketers allocate 20% of their marketing budget to tests.1 That may sound like a lot, but it’s the kind of advice you can’t afford to ignore.
The Big Picture
So you know that you should be testing, but do you know how to test or what to test? Before diving into the details, it is essential to understand the big picture.
These are the overarching terms to understand:
For most intents and purposes, constants can’t be tested because they can’t be changed. Your variables are the items that can be changed in order to test varying results. Under these broad umbrella rules, there are many different types of testing that can occur.
Here are some of the most common types of testing:
The Results Are In
When it comes to analyzing the results of all this testing, I’m apt to paraphrase one of the great lines in the brilliant British playwright Tom Stoppard’s Jumpers –Democracy is not in the voting. It’s in the counting. Test results should not be judged by raw numbers but rather by how these numbers are interpreted and how they stack up against anticipation. In other words, we need to make sure that we have a solid standard metric for measurement.
For example, if we are measuring whether one price or offer is superior to another, we need to measure not only which generates the MOST business responses but which also delivers the best FINAL economic result, profit or contribution. Sad to say, some great front-end responses can often seed disastrous back-end, bottom-line results. Better to catch them early.
(The biggest disaster in my long career was popping champagne corks too early after testing the rollout of a new negative option music club. Because negative option businesses send products to interested consumers before they pay, it appeared on the front end that we were wildly successful. However, we didn’t pay sufficient attention to whether or not the happy new members would actually pay for their music choices. They didn’t. And we got killed! Had we followed good testing practice and carefully examined front- and back-end results before rolling out, the disaster would probably have been averted.)
A Subscription By Any Other Name
Recently, I was at a seminar and asked the participants if they had any subscriptions. Very few raised their hands. Then I asked how many had streaming video services and nearly all the hands went up.
This clearly bore out in the results of an A/B email test which offered the A group a “subscription” and the B group, a “free trial” of a no-commitment newsletter. After a month or so, a fully paid and committed “subscription” was offered to responders from both groups.
As you no doubt guessed, the B group’s response was much higher than the A group’s. (The word “subscription” obviously carried the odor of an unwanted commitment even though in this case there was none.) And even though the subsequent conversion to paid subscriptions of the A group takers was somewhat higher in percentage terms than the B group takers, the revenue generation, return on marketing investment (ROMI) from the B group was higher.
It is also important to remember that the size of the expected response from each test group must be large enough to be statistically valid. The smaller the sample, the greater the likelihood of inaccurate data.
Following an accepted hierarchy in a testing program is almost always beneficial and should tell you if any essential element of the promotion is not pulling its weight so you can make necessary adjustments.
The Bottom Line
With the complexities of testing, it is important to not lose sight of the following key strategic objectives:
The bottom line is always how much response has to increase over the current results to justify the different variable cost elements. And if up-front that increase doesn’t seem to be realistically achievable, perhaps it is better not to include it at all.
Surprisingly, marketing managers often judge the result of tests by including the so-called “first costs” – the creative or other non-repeating costs. This can badly distort results. Tests, to realize their real value, must be judged not on what they actually cost but what a large distribution of the promotion with the first costs amortized over the whole initiative, would actually cost. Many economically valid programs have been abandoned as a result of this error.
We all know that testing is like peeling an onion (tears and all) but the final dish is almost always worth it. While there may be many more layers left to peel, this simplified overview should serve as an invitation to learn more and to make a regular testing program an integral part of your marketing effort.
Leave a Reply