A split test or AB test as it is sometimes known is a great way to test the quality and success of your campaigns, be in marketing, product development, and many others.
There are several steps that are required in order to successfully execute one. The test has to be planned out and several questions need to be answered such as who are the recipients and what is going to be tested. The product or campaign to be tested needs to be prepared and the split inserted.
After the campaign is executed, the results need to be compiled and the splits compared. Lastly, there is the feedback loop, where the results help determine the next course of action.
The planning phase
The first thing we start with is planning out all of the details. There are four aspects that have to be worked out before the test can be executed.
- The first is the split itself. What is going to be tested? Common test items are subject lines, sender name, content choices, and layout.
- Next is the success metric. The metric has to be appropriate to the split. Eg. if the split test is testing two different email layouts, then the open rate wouldn’t be an appropriate measurement since in most cases, they can’t see the content until after they open the email. If applicable, the lists of recipients for each split need to be pulled and stored somewhere.
- How and where this is executed and stored needs to be decided beforehand.
- The last consideration is how long the trial will run for and how many trials are planned. If these details are hashed out and documented before the start of the trail it makes building the knowledge base much easier.
Selecting the audience
If there is going to be a specific audience in each split, the selection should be random. A proper randomization selection should give you the proper distributions across major categories such as geographic location, income, or other attributes (or even action states like a prior relationship or prospect).
The size of the split is also something that needs to be addressed. The only hard rule is that you need to have a statistically significant sample size. After that, you can split the lists in whatever manner you want.
- Most split tests as the champion/challenger setup for this. In this setup, the champion is the message that either has won out in prior tests or if this is an initial test the message you have used in the past. The champion (the control group) receives a larger share of the population than does the challenger.
- Another split size that sometimes gets used is the 50/50 split. This is useful because it gives two large groups and helps to cut out the noise when comparing the two, however you are also increasing the risk (and by extension reward) by opening up more people to an experimental treatment. As long as your groupings are statistically valid, there is flexibility in how the groups are assigned.
It is essential that everything else about the different versions has to be exactly the same except for the aspect you want to test. For example, when you are testing the effectiveness of the subject line of an email, the content inside of the email needs to be the same.
If the content was different on the inside for each of the splits then you wouldn’t be able to isolate the effect of the subject line on the click-through rate. The same holds true for any test. The benefit of the split test is being able to isolate and measure the effects of a single aspect. Therefore, it is essential that only a single aspect changes.
Analysis is crucial
Since a split test is a test, there needs to be an analysis portion of it. There needs to be a predetermined period when the test officially ends and the results are tabulated. The reason the period needs to be pre-determined is to prevent bias. It is very likely that in the event if the hypothesis is being demonstrated, the examiner would cut the test short and declare victory even if the results were still coming in.
After the trial ended, the success metrics are compared, and if one side has outperformed the other in a statistically significant manner then it is declared the winner. If there is no winner the examiner needs to determine if they will extend the trial or if it will be abandoned without a clear winner.
Document everything
One aspect that can’t be stressed enough is to document every aspect of the trial. Everything from the initial thoughts, goals, and justification for the test should be written down in detail. Every so often, go through your notes and pull together the verified (or rejected) points and build documents out of the points. Building this knowledge base gives you a good initial starting point for setting up additional tests and making strategic or operational decisions.
If the tests are thoroughly set up and executed, they will build a bed of knowledge that you can decisions as well as quantifiable forecasts off of. Ideally, you should be running as many tests as you have the capacity to. The more tests you run, the faster and broader your base is built. If you find success running trials, share your success with your colleagues and encourage them to run trials of their own and share out the results. Every trial builds your business intelligence up and makes each future decision that much better.
…………
There you have it, a detailed view of how to conduct a split test and why every step is crucial. If you have any questions or would like to make a suggestion, drop us a message here.