Testing 101: Create marketing campaigns that convert with an effective testing strategy
Building a test strategy for your marketing initiatives is not an easy task, especially if you want to learn quickly. But according to testing expert Michael Krueger, being purposeful with testing and putting accuracy at the core of your strategies is the key to identifying program improvements and opportunities.
Mike provides thought leadership to the development of data science solutions that benefit our clients’ data enrichment needs and marketing opportunities. He oversees a talented team of data scientists that have a similar natural curiosity in the impact data and machine learning can have in achieving business goals. Mike brings over 20 years of extensive experience ranging from machine learning, segmentation, analytics, BI and market research.
You have decided to test because you think some changes to your marketing campaigns or program will positively affect performance, but you need the data to prove your hypothesis.
Before choosing the right type of test, you need to ensure that the results are measurable – meaning you would have the data to confirm the success (or failure) of your experiment.
For example, if you’re testing email subject lines or creative, you must have access to the open and click data to assess performance. If you are testing landing pages, you need to have access to the traffic data; if you’re testing video length, product mix, message frequency, offer type, or any other variable, you need to be able to see the resulting metrics to measure performance.
Once you are confident you’ll have the necessary data to measure results, you can start thinking about a test plan. The table below outlines the two primary testing options, as well as their key benefits and limitations.
Naturally, multivariate testing requires more discipline and patience, but the resulting insights are greater. If the size of your audience is not a concern, this is typically the best choice because what you learn through multivariate testing would require several A/B tests to conclude.
According to our ‘Ask the Expert’ on Learning Agendas, a learning agenda is the foundation of an effective test plan.
So, you’ve gone through the steps I outlined above, and you have both your hypothesis – a.k.a. what you want to test – and the outcome you expect to achieve – a.k.a. why you are testing.
Now, consider the following:
Answering each of these questions will allow you to determine the eligible population you can include in your experiment and provide a high-level timetable of when you can socialize your test results.
Before addressing this question, this section could easily turn into a Statistics class, so the content will remain digestible by touching on some key concepts used in testing.
Randomization
If your hypothesis is confirmed after the experiment is complete, you want to be able to apply the learnings to the entire eligible consumer audience in future marketing efforts. Avoid the common mistake of list bias when you select a randomized audience from your eligible consumer pool. Instead, you must randomize the selection so, if for example, your audience is 70% female and 30% male, those same proportions are represented in your test sample.
Another common error occurs when testing a new channel to understand its impact on conversion. You might want to split consumers into two segments: one that is served digital ads vs. one that is held out from digital ads. An important thing to remember is that the latter segment must still consist of consumers who qualify to be served digital ads but are intentionally held out (as opposed to those who are held out because they didn’t qualify.)
Sample Size
There are two factors that help you determine the appropriate size of your test: confidence level and power.
The higher each of these factors, the more accurate your test results will be. Generally, a 95% confidence level with 80% power is used to define the validity of your test. This loosely translates to your being 95% confident in the results for 80% of the time it was executed.
One important thing to note is that achieving high values for power and confidence generally means you need a larger test size and/or a larger difference in the results achieved in your test groups.
As part of the test, you also want to determine whether this is a one-tailed test or a two-tailed test. In short, with a one-tailed test, you are claiming that your hypothesis will perform better than the control or some other treatment and then setting up the experiment to confirm that.
With a two-tailed test, you are not making any judgement on which would perform better or worse, but simply want enough data to be able to measure that after the experiment concludes. The default option in most software tools is two-tailed test and is the option that typically provides more accuracy.
Short answer: NO.
You might pause testing or deprioritize it if a program is outperforming benchmarks. However, as the foundation of your marketing program changes (i.e. targeting strategy, channel mix, etc.) the performance achieved by your prior tests should be validated.
As the industry and consumer expectations change, you should continuously try to improve consumer experience by providing relevant messaging at the right time. The level of interaction with your marketing programs can tell you if you are delivering the right experience. As consumer needs and desires change, your experiments must restart. In marketing (as in life), change is the only constant.
Happy testing!