By Dolly Howard

The-5-Frequently-Asked-Questions-About-AB-Testing-in-Marketing

Wouldn’t it be great if we could always know how people think and act to the cues and stimuli we present?  Then we would be able to properly create websites and offers that would have 100% conversion rates.  How wonderful is that!  Alas, in the real world this isn’t true.  Which is why it’s very important that we use A/B testing to find the best way of getting our message across to our audience.

A/B testing, otherwise known as split testing, is a method in which we can compare two variables to determine which one produces the best response from our visitors. By doing this, we are able to learn more about the behavior of our target audience to improve our messaging and bring in more traffic, and thus more conversions.  According to HubSpot, testing landing pages can increase lead generation by 30-40% for B2B websites and 20-25% for eCommerce sites.

Knowing how crucial A/B testing is for increasing leads, it’s important that you begin experimenting as soon as possible.  However, many of you may be hesitant to start due to a lack of understanding about A/B testing.  There is no need for you to feel this way.  

Here are just a few of the most frequently asked questions about A/B testing:

1. Why do A/B testing?

If you’ve read any books about startups, you’ve probably heard the principle about failing your way to success or learning through failure.  This rings true for just about most things in life—and marketing is no exception.  Marketing is always changing and the best way to learn what works and doesn’t work is through trial and error.  But you don’t want to rely on just guesswork and this is where A/B testing comes in play.  Remember, your target audience is a collection of diverse and real individuals; they will not always act in the way we expect.

2. What should you test?

Start simple.  Figure out the page or email that you want to test, then determine what element (also called the variable) that you want to test.  There are 2 criteria for choosing the page you want to test:

  1. You want to choose pages that have a lot of traffic.
  2. You want the page to have a clear and specific goal for the visitor, such as a call to action.

Look at your analytics data to identify the high traffic pages.  Landing pages are great for A/B testing because they are somewhat secluded and wouldn’t affect the rest of the webpages.

When choosing which element to test, you want to consider that which will affect conversion the most.  Call to action buttons and copy is typically the first variable to be tested, but you can experiment with headers, email subject lines, images, etc.

3. How long do you run the experiment until you can call it a success?

First of all, you must make sure you have a clear goal for the test—when you have no clear winner, then your test is inconclusive and you should allow the test to continue.

Nonetheless, there are certain factors that you should consider when determining whether to end the test and declare a winner.  The first number to consider is the statistical significance. This number should be 95% or higher.   If you have a statistical significance of  99%, this means that there is a 1% probability that the data is wrong.  In essence, this is saying that it is very unlikely that the results are due to chance but are caused by the specific change you introduced.

Another number to consider is the standard deviation of the conversion rate.  The standard deviation measures the amount of variation from the average.  Therefore, since conversion rate is what you are using to measure the success of an A/B test, you want to make sure that the ranges of the conversion rate (aka conversion range) do not overlap.  There should be a clear distinction between the conversion ranges of the two pages tested.  For example, if you have a standard deviation of 1% and a conversion rate of 5.5% for page variation A, your conversion range is 4.5% to 6.5%.  Now if your page variation B has a standard deviation of 2% and a conversion rate of 7%, your conversion range of this page is 5% to 9%.  As you can see, there is overlap in the conversion ranges of the two pages.  When this happens, you want to allow your test to continue until you have a clear distinction between the two conversion ranges, and thus determine a clear winner.

The third number you must consider before pulling the plug is the sample size.   This represents the number of people who took part in your experiment and for statistical significance this number should be large.

4. How often should you be testing?

Now this is a question that doesn’t have a numerical answer because this depends on whether or not you have a good reason for testing.  Every test you do must have a clear goal in mind. 

5. What is multivariate testing and how is it different than A/B testing?

While A/B testing allows you to test two variables to choose the better of the two, multivariate testing allows you to test many variables simultaneously.  However, statistically significant multivariate testing requires an incredibly high volume of traffic (the type of traffic that Google, Yahoo, YouTube, Facebook get).

A/B testing isn’t difficult and there are many software platforms that have this built-in.  At SmartBug Media, we use HubSpot which is one of the platforms that offers this tool.  This article should help answer some of the many questions floating around in your mind about A/B testing, but please let us know by tweeting us at @smartbugmedia if you have any others we can answer.

New Call-to-action

 

The-Ultimate-Guide-to-Inbound-Marketing-Personas-cover

Refine your inbound marketing efforts with:

The Ultimate Guide to Inbound Marketing Personas

Check It Out
Dolly Howard

About the author

Dolly Howard is the Director of Marketing for SmartBug Media. Previously she worked as a senior marketing consultant leading SmartBug Media clients in strategy for lead conversion improvement and total marketing ROI. As a past HubSpot employee, Dolly is excited to share her knowledge and help enterprise companies grow their business.

Subscribe to the SmartBug Blog below to see more posts from Dolly and the SmartBug team.

Read more articles by Dolly Howard.

Subscribe to get our new blogs delivered right to your inbox