Best Practices and Common Fails of Email A/B Testing

Proficient digital marketing managers consider email testing necessary for boosting their email marketing. You may know that making good email campaigns is not an easy thing. It is important to take into account the subject line, color, font, content, images, videos, and more. Even a smiley face, or the color of a button, can affect the outcome.

However, relying on your taste only is a way to lose. So, you need to go the other way. This is when email marketing testing comes to the rescue.

In this article, you will learn how to run email A/B tests. We’ll look at the complexities of testing and the benefits of this email marketing method. 

Why use split tests in email marketing campaigns?

• Better contact with the audience

You need to learn what your clients want. So, you will be able to get statistically significant results. Even if you are experienced in doing business, your assumptions may not match the reality. Having your own opinion is good. But it’s better when the test results of the marketing campaign prove it's effective.

• Increasing conversion and other metrics

You may have doubts about the design, email subject lines, products, the tone of messages, and so on. A/B testing gives answers to all your questions on how to create a winning email. It allows you to make data-driven decisions about ideas you've brainstormed with a marketing team. Check which one works the best in practice.

• Reducing the risks associated with the changes

Repeat after me -- customers don't like sudden changes. That’s why part of the audience may unsubscribe from your emails. Consult with users through tests and avoid the risks.

How to analyze the results of a test?

Relevant and up-to-date data is crucial for conducting resulting marketing campaigns. If you already have some analytical tools but don’t know what metrics to check, start from these 3. They are open rate, CRT, and CTOR. Include them in your email marketing strategy and keep your fingers on the pulse of any changes.

Open Rate (or Open Rates, OR)

One of the key metrics of email testing is the open rates. If the metric decreases, that means something goes wrong. The Open Rates shows how many times your email has been opened by a user.

The criterion that influences most to this metric is the email subject line. It shouldn’t be too long and confusing. Write clear and keep the intrigue, so the user will be curious to click on the subject line and open your email.

Click-through rate (CTR)

CTR shows how many times subscribers clicked on links in emails. Let's suppose that your email was received by 100 people, and only 25 clicked on the links in it. The CTR is 25%. If the click-through rates are low, it means your email strategy has some mistakes. In this case, it is necessary to analyze and fix.

Click-through rate (CTOR)

The engagement of users is determined by CTOR. For example, your email was delivered to 100 people, 60 people opened it, and only ten clicked on the links. The rate is 16.6%.

A/B testing: Elements to check and improve in your email campaigns

1. Subject lines

The subject line is the first thing a customer sees in his/her inbox. The user can now open the email or delete it even without reading it. 

Pay close attention to the wording. The subject should show the main message, call to action, or at least have some intrigue. Check the examples of subject lines below and guess which is the most appealing of them. The subject line length seems to be not important, but it matters to get more clicks through email. You can try a few variations, compare the results, and find out the winning variation to check if that’s true. Sometimes even word order makes the difference.

2. Email text length

In the era of TikTok, Instagram stories, and YouTube shots, long texts aren’t popular. That’s especially related to new customers. So, don’t make email recipients scared with wordy newsletters. Your message should be short, clear, and engaging. If you want to write lots of useful information, take care of your readers. Make a good structure of the email. Add lists and pictures.

(Source: Email from Moosend)

Add lists with Stripo in 3 simple steps:

1. Highlight a necessary part of your text.

2. Click the “List” button on the toolbar.

3. Choose a type of list that you like.

3. Your email design

First of all, find a color scheme you’ll use for your email marketing campaign. Color palette affects people, so check more information on how it works. For example, Wikipedia says that red creates a sense of excitement. It is usually used for emotional messages and sales. At the same time, the blue creates a sense of reliability. That’s why B2B marketing managers use it to emphasize a reliable partnership. What color works better for your audience and goals? AB testing can help you answer this question.

The colors, shapes, and types may provoke more clicks from your email clients.

3.1. Visuals 

Visual elements in emails aren’t less significant than the subject line. Most subscribers perceive images better than text. So, you can add pictures, videos, and GIFs, to attract more people and receive a better ​​conversion rate. But be careful with the style and colors of the visual elements. They should be in harmony with your brand book.

(Source: Email from Stripo)

3.2. CTA buttons

Call-to-action headlines, links, and buttons are essential for any email click-throughs. Conversion rates directly depend on the CTA button. It should be obvious to email subscribers where to click to get more details or make the purchase.

CTA button may link to the product page, contact form, or a place for review. Change its size, color, and location to find the one option that is best for your email campaign.

(Source: Email from Nike)

Remember that the multivariate testing, the button’s name should be clear and concise.

Get more useful tips about CTA buttons in the previous post from the Stripo blog.

3.3. Fonts

The rules of using fonts in email campaigns are simple. Avoid italic and other decorative fonts. Make sure your text is easy to read. So, anyone can understand it without special effort. Also, don’t forget to put dots at the end of sentences and lists.

4. Tone of voice

Use the tone of voice for your brand to follow your communication consistency. If you don’t have it yet, then it's time to fix this mistake! Detect in what tone you’ll talk to your customers, starting from a subject line. That influences the associations with your brand and its effectiveness. The tone of speech could be friendly or formal and businesslike. Split testing helps to choose one variation for all future campaigns!

You can see below “version A” with an example of a formal speech.

(Source: Email from Adweek)

The “version B” shows an example of a friendly, cheerful speech.

(Source: Email from Chubbies)

5. Personalization

The simplest way of personalization is to address recipients by their names.

(Source: Email from ProductPlan)

You can also use any other information, like users’ marital status, hobbies, or occupation. Divide your customers into segments according to special criteria. Write personalized texts for each segment.

More information about email personalization is in the article.

6. Send time of emails

Entertainment/promo emails should be sent on Friday when people have some free time on them. But business emails are to be sent on Tuesday. It’s time when workers are mentally ready to perform new tasks. Is it relevant for your business? Test several days to determine the most effective send time for your conversion process.

Step-by-step testing of email campaigns

Let's figure out how to conduct testing and get meaningful results:

Step 1. Choose what element you will test.

Recklessly changing elements in an email hoping to get a million dollars isn’t the best idea. Decide what metrics you’re going to consider successful. When you have a particular goal in mind, the results will be better. 

There are situations when we need to find out how a new element in the email campaign will work instead of metrics. 

In February 2020, the Stripo team embedded an AMP form in an email. The form allowed recipients to share feedback directly in their inboxes. We wanted to check which option would collect more reviews:

• regular third-party form;

• embedded AMP form. 

The second turned out to be five times more effective than the others. This encouraged us to create the AMP block "Form." Now our users can also create a similar form for their newsletters.

Interactive forms let you know what users think about your product and services.

Step 2. Select a variable.

Decide what you will be testing in the email campaign. For the test result to be as effective as possible, test one variable in one email at a time. For example, change the subject line or the way the company name is written. Even small changes, like emoticons, can lead to high results in different versions.

(Source: Gmail)

Important to note:

Make changes to the variable in the copy of the email. Change the color, font, or CTA. But remember, just one variable at a time.

Step 3. Segment your contact base.

Do it yourself or automatically in your ESP.

Step 4. Process the results.

An essential step to see how resulting the split test was. Pay attention to comparing metrics of emails that were sent in the same period. Split testing has many nuances. That’s the most important of them.

Step 5. Act.

The analysis of metrics shows you the next steps. Also, that’s okay if you need to do a few more tests to get a picture of your ideal email.

Make changes, test, and analyze. This is the only way to improve email conversion, increase self-confidence, and earn trust.

TOP 5 tools for conducting A/B testing

Several email marketing tools can save your time and efforts. Let’s take a look at them.

1. For the subject line.

You already know the importance of the subject line. But how do you evaluate it? by using the Email Subject Line Grader by Net Atlantic.

2. For email design.

Stripo allows you to create stylish visuals for your emails. Use templates, customize them, and make changes just in one click.

Using Stripo, you can simply copy a version of your email campaign and duplicate it for each new test with the needed changes.

3. For detecting the tone of speech

Grammarly app will help you with unique communication styles. Use tests to find out what tone of voice is more engaging for your audience. Grammarly offers 10 of them.

4. For technical parameters testing.

Imagine that you tested all the visual parts of your email, and everything looks just perfect. Then, you need to be sure the email will be delivered properly. A fake SMTP server solves the problem. We suggest you use Mailtrap.

5. For the best send time.

Any modern ESP is supposed to save and analyze the time when the email is opened. So, you will be able to send emails at a convenient time for your target audience. Here’s our review of the TOP 10 ESPs useful.

A/B testing mistakes that will cost you efficiency 

Some inexperienced email marketing managers make dumb things that cause their tests to fail. That causes the opinion that A/B testing isn’t an effective marketing approach. But it should be used wisely to help you win.

So, let’s check the seven main mistakes of A/B testing. All of them stop you from getting higher open rates and better results. 

1. No care about the hypothesis 

Before the launch of any email with a changed element, think about hypotheses. What will this change affect? What will the result be? You shouldn’t predict it perfectly. But you should understand what you want to get as the result of your a/ b test. And why you have made exactly this change, not something else.

2. Copying case studies thoughtlessly

That’s essential for marketing managers to be curious about industry best practices. But just copying your competitors' marketing campaigns, you won’t get satisfying results. The better idea is to check and analyze case studies, find insights, and use them to make your hypothesis.

3. Unprofessional measuring results of the email campaign

Be careful with the metrics which you track and compare. The important thing is to have only one variable in each new test. For example, you want to check what subject line length works. So, change the number of words and don’t test any other elements. Thus, you’ll get clear results.

The data-driven approach is considered the best way of making performance marketing today. But if your Google Analytics and CRM aren't adjusted, you can’t make the right decisions.

4. Wrong timing of email testing

Pay attention that you don’t compare different periods while looking at the “a version” and “b version.” You should also think about the context and season. During Black Friday or winter holidays, the data varies in the two exactly same emails.

Also, don’t try to compare the results too early. Users need time to check their inboxes and react to your newsletter.

5. Testing of the wrong page or website

You should choose an appropriate web page for your testing to get accurate results. Having a modern and appealing landing page with all the necessary information is important. Don’t forget about call-to-action and the purchase button. Use only those web pages that are on the stage of your marketing funnel.

Maybe, the silliest mistake is to add a link to your administrator’s panel of the website to the email. Be careful and check that the traffic goes to the published page, which is available to all users.

6. Problems with finding your target audience

You may know that email marketing campaigns always start with audience research. A marketing manager prepares a database of relevant contacts, their names, and emails. These users may have interests related to your product. Another case is when they have already bought something in your shop. The problems start when these emails are wrong or these contacts aren’t relevant to your store.

And the most important thing in A/B testing is to have enough subscribers to get meaningful results. You need thousands of them, or even more. That’s why the 500-1000 contacts database doesn’t work for this marketing tool. 

7. Too many email tests at once

The last mistake also relies on the size of your audience. We don’t recommend launching more than four tests simultaneously. The best option is to have 1 or 2 at once. The thing is, the more campaigns you need to test, the more subscribers are required. Don’t divide users into small audiences for that goal.

To sum up

Following the best practices of email testing, you gain a proficient tool. Tests help to be on the same wave as the audience. After the analysis of open rate, and other metrics, everything becomes clear. Use the data to see which elements of the campaign require special attention. It could be the subject line, photos, texts inside, and the landing page. 

Stripo team work for you to make email marketing as easy as possible. Check the instructions from our blog to improve your email marketing metrics!