The ABCs of A/B Testing in Email Marketing

Getting the Biggest ROI From Your Email Campaigns

Getty Images

Like the famous proverb goes, “A rolling stone gathers no moss”, it is important to keep moving forward, be it in life or in your email marketing campaigns. Stagnation in terms of catering the same type of emails to your subscribers, just because the emails were received well and got astounding open rates earlier, is no longer acceptable. In fact, the more alike all of our communication looks, the easier it is to tune out.

And the last thing you want to happen is to see your hard work deleted without even opening the email. 

It is important to modify your emails in accordance with the ongoing trends. The key is to find out what’s clicking with most of your subscribers. Makes sense, but it's not so simple. You need some data to "test" your theories and find out what's clicking.  By testing your options through what the email marketers preferentially call “A/B testing your email campaign”, you can start increasing your ROI without increasing your marketing dollars. 

A/B testing is basically pitching an email campaign with another version of the same email campaign by making changes to only one element of the campaign. One version is sent to a set of subscribers and the other to another set of subscribers. The vector is decided based on the metrics collected.

The effectiveness of A/B testing your email campaigns lies in the fact that even though there isn’t a drastic change happening in the email by changing a single element, there can be a tremendous change in the response rate of the campaigns.


Moreover, for accurate results, it is essential that you change only one element at a time. For testing more than 2 versions of the email simultaneously, Multivariate Testing is the best approach. 

What Elements Can Be A/B Tested?

An email is made of several little components and there is a possibility to test each and every one of them.

But marketers generally tend to test the following components:

Preparing to Conduct A/B Testing

Before you conduct A/B testing, it is important to chart out your strategy. Consider: 

  • What component shall be tested.
  • What is your test group size. (Keep in mind, your test segments need to be large enough to reach statistical relevance).
  • The time period between the send time and analysis time.
  • The metrics to analyze the winning campaign.

Once you have charted out your strategy, it’s time to put the strategy into action. Thankfully, most email marketing campaign companies have the provision of A/B testing your emails integrated right inside their email editors.

Let’s check out how it works in MailChimp (since this is the software I use for my email marketing.) 

You just need to select a new campaign and choose A/B test. (P.S: Using this option you can only choose to test between Subject line, From Name, Email Content, and Send Time)

In A/B campaign page, choose which element you will test, what percentage is your target group size, and metrics to be considered after the time period.

Based on your choice, the emails shall be equally sent to the selected percentage and the winning email, based on the selected metrics, shall be sent to rest of the list. I love that last part. Essentially you are testing the email before it goes to everyone. So you're chances of success are going way up! 

Based on what element you choose to A/B test, you need to input the alternatives in the setup section or in the email template (in case of email copy). For rest of the components such as CTA (Call to Action) copy, email layout, image placements, send times and Plain text version, you shall need to create two different campaigns with variations, segment your list as per your target group size and analyze manually.

Congratulations! Now you know how to A/B test your email campaigns. But this is not the end of the road, you can put the following tips into practice to improve the effectiveness of your A/B testing.

Tips to Improve the Effectiveness

  • Keep it Simple. Focus on those variables that shall yield better and more vital results. For example, you may find a big difference when you reduce the number of images in an email campaign. Some of your subscribers may have limits on the size of the email and many email providers automatically grab emails with lots of images and consider them SPAM. 
  • Ensure consistency in test campaigns. Ensure that the variation between the two elements is not to diametrically apart. For example, with subject lines you might try “Special offer! Buy Now” & “We have a special offer! Just for you” as your variation.
  • Learn from your past. Evaluate campaigns from the past to analyze what has worked best and what went ‘dud!’.
  • Determine your end goals. Decide whether you need to improve your open rates or increase click rates and conduct your test accordingly. In case you have more than one end goal then…
  • Frame your test according to your goal. Set up one test for each of the elements and pitch your winning email from each such test. 
  • Split the list wisely. Before you test your email campaign, check if the list is large enough for an A/B split or needs to be broken into several smaller test segments.
  • Trust the metrics. It is easy to be seduced by pretty emails with all the color and graphics, but the numbers do not lie. Sometimes the plain emails do the best. 

A/B testing can be taxing to those uninitiated, but veteran email marketers swear by it in order to serve nothing but the best to their subscribers and thereby happily reap a great ROI. Online marketing done wrong can be very costly. And most times you don't know what went wrong. But with this type of testing format you can. 

Author Bio:

Kevin George is the head of marketing at EmailMonks - one of the fastest growing email design and coding companies specialize in crafting beautiful email templates, PSD to email templates and much more. He loves gadgets, bikes, jazz, and breathes email marketing. He is a brand magician who loves to engage, share insights with fellow marketers. Feel free to connect with him on Twitter and LinkedIn.