A/B testing any sort of campaign message is a great way to ensure you are giving customers the information they want to see. When you first start out creating a message or campaign, you are probably basing the content off of what you expect your customers want to see, but they might not always react the same way you are expecting.
With A/B testing you can try out a couple different options on a smaller sample of customers before sending it out to your entire email base. This gives you some time to test which layout, design, colors, and content people interact with best.
Important things to keep in mind when A/B testing
- The sample size you are testing should be dependent on the total number of recipients you are sending the message to. If you have a smaller group of people you should test on 10-15% of the recipients, but in a bigger group (50,000) you only need to test on 5% or even less to get just as accurate results.
- Don't test too many things at once! A/B testing should only be comparing one or two components of the message. This could include the subject line of the message, images used, text, or different links, but if you try testing all of these things at once it won't be clear why one message is performing better than the next because there are so many different variables.
- Don't send too many variants. If you send a message with four different variants, for example, each message is only going to 25% of the test recipients. You need to make sure you are carefully picking each message and make sure you are adding enough recipients to your list so it is proportional to the number of messages being tested.
- Time of day is important! Make sure you keep in mind when you are sending the message out and when your list of contacts is most likely to open an email message.
Make sure you leave enough time for the test. Think about when you want the main message to be sent and make sure you start your A/B test with enough time for recipients to interact with the message so you get accurate results.
After you run your test you can pick the winning message. In Sitecore this can be selected automatically or you can choose the message manually. Once the winner is selected it will be sent out to the remaining recipients on your list.