Email Marketing Automation And The Pitfalls Of Multivariate Testing

by Melle Staelenberg
10 November 2014

If you work in digital marketing, there is a good chance you will be familiar with metrics such as Click Through Rates (CTR) and Open Rates.Both metrics are commonly used as performance indicators in email marketing, especially during variation testing. When running A/B or multi-variate testing, two or more versions of an email are sent to a small subset (e.g. 10%) of subscribers. Many of today’s email platforms use Opens or Clicks to highlight the best performing version.

With more businesses reaping the benefits of smarter email platforms and marketing automation in general, testing multiple variations of email campaigns has become increasingly simple. As a result, digital marketers tend to rely more on automation to pick the winning variation which is often sent to the remaining 90% of customers automatically.

This article will demonstrate that it pays to do a few simple checks after testing instead of letting your platform decide what works best.

Open rates

Open rates, although skewed as a metric because of differences in how these are measured, are predominantly driven by subject lines.If you have run two similar campaigns at the same time of day with one campaign significantly outperforming the other in terms of Opens, there is a good chance it is caused by your subject line.In most email clients, the subject line is the only part of your message that is seen before the recipient will decide to open, ignore or delete the email.This is great info to have but getting your email opened is only half of the story.

Click Through Rates

Unless you are only interested in brand exposure from the email itself, most companies run email marketing campaigns to drive an action or transaction on their website (click throughs, or simply ‘Clicks’).
Clicks can be increased by making sure the email uses responsive design so it renders nicely on smartphones, tablets and desktops. Some email clients block images by default, so make sure your email is not too image heavy and uses HTML where possible. Like Opens, Clicks is another popular email metric. A high CTR indicates your recipients engage with the content in your email (instead of just passively reading it).

Click to Open Rate

Let’s start with an example to introduce another, less popular metric. Company ABC is testing 2 versions of a new email template with subtle differences in subject lines and email content. The results of the first round of tests are as follows:

Which version to use? ‘Drop Version B and go for Version A’, the Marketing Manager says, ‘Version A outperforms in both Opens and Clicks – a no brainer!’. ‘Hold on’, the Email Specialist says, ‘The content of B seems more engaging’. Why is the Email Specialist keen on reusing elements of a template with lower Opens and Clicks? This is where click-to-open rate, or CTOR, comes in. CTOR simply divides the number of clicks by the number of opens, so in this case:

Template A returns a CTOR of 3.5/40 = 8.8% vs 3/30 = 10% for template B. The 10% shows that of the users who opened email version B, 1 out of 10 clicked on a link, a better score than template A. The low open rate of 30% in Template B could mean a poor performing subject line. Remember: clicks are only generated by recipients who opened an email. If we can get the opens of Version B up to 40%, we can expect the clicks to grow at the same rate (from 3% to 4%). Clearly, there is something in the content of Version B that drives more clicks per user who has opened the email compared to Version A, an insight only CTOR provides.

So what to use?

There is no single metric that measures or determines universal success. The advantage of reviewing CTOR, especially when applied to A/B testing, is that it helps optimising campaigns. Had Company ABC continued with Version A, it would have settled for a CTR of 3.5%. By listening to the Email Specialist, it learned the good and bad bits of both campaign versions, managing to create a hybrid (Version C). How? Simply by applying the subject line of Version A to the creative and content of Version B. As a result the company was able to drive the same 40% open rate as with Version A, keeping their original CTOR at 10%: Clicks went up to 4%.

If Company ABC were an online retailer, the difference between a CTR of 3.5% and 4% could equal a 14% increase in sales!

It must be said there are many factors driving Opens and Clicks – it’s not just subject lines and content. Having said that, the above simplified logic of picking the best of two versions works well for variation testing as such tests are isolated from many of the typical variables such as time/day of send, IP reputation, ISP acceptance and Inbox placement.

Use the smarts built into your email platform to test your multi-variate campaigns but make sure to analyse the results before selecting your final version. Ultimately, the success of your email campaign can't be measured without knowing your business' goals. As with any metric, Clicks, Opens and CTOR are simply tools you may find useful trying to optimise your email campaigns.

Why Salmat?

Salmat Digital is Australia’s leading email service provider, sending out emails on behalf of more than 100 clients. With an in-house Creative Agency, HTML Coders, Campaign Producers and Data Analysts, we offer services and solutions that vary from self-service to fully managed.

If you would like to become an expert in email marketing, contact us today on 1300 725 628.

If you would like a review of your current email program or simply check if your current HTML code is up to standard, please get in touch with us.

50 search marketing tips

Page 1 of 2
  < 1 - 2  > 
About the author
Melle Staelenberg
Business Manager

Melle has been with Salmat since 2010. In his current role, Melle is responsible to maximise business value from Salmat’s core products, including email and mobile. His product expertise combined with a strong focus on research and a healthy dose of creativity helps driving Salmat’s digital roadmap. Melle holds a Masters degree in Business Economics and a BA in New Media & Digital Culture and is passionate about all things digital. Being Dutch, he likes to cycle – he’s also a big Ajax fan (the football team, not the cleaning product)!

More articles by Melle Staelenberg