Description

The purpose of an A/B Test is to compare the performance between two different versions of a marketing tactic. The A/B Testing template allows you to clearly capture the critical elements of your test prior to execution. It also forces you to consider why a potential A/B test is valuable and what you will do if the result of your test supports your hypothesis. A/B Testing (also known as Split Testing) is an increasingly popular method to make incremental improvements to advertisements, websites, and other customer touchpoints. Marketers now have the opportunity to run several A/B tests, often at the same time. Using a common A/B Testing template can help you to clarify strategic and operational details with your team.

 
Question

How are we planning to execute an A/B Test?

 
Steps

1. To plan an A/B Test, you need to first have an understanding of what you want to improve and how you might improve it. Capture these background details and explain why the A/B test is worth investing in.
2. You likely have a Control (existing Version A) and a Challenger (modified Version B) that you are testing. Write a hypothesis for your test by explaining why you expect the Challenger (Version B) to ‘win’.
3. Document your test by describing your independent variable (what is being changed) and your dependent variable (what is being measured). How will your Version A be different than your Version B?
4. Capture what and how you will measure the difference in performance between Versions A and B. This metric should relate back to your original reason for investing in the A/B Test. What are you trying to improve?
5. Finally, capture how you will execute the test and the actions you will take based on the results. This is a useful litmus test for value - if there are no specific actions coming out of this test than it is likely not worth doing.

 
Considerations

• Make sure that you are not running so many A/B tests concurrently that you may impact your results.
• Continue to review trends in your marketing performance data to identify opportunities for new A/B tests.
• Only run A/B tests if you can achieve the sample size required for your target confidence level.

 
Reference

Bland, D., Osterwalder, A., “Testing Business Ideas: A Field Guide for Rapid Experimentation”, John Wiley & Sons, Inc, 2020.