A/B Testing: Beginner's Guide 2023 (Steps, Examples, Tools)

As the world shifts to online shopping, optimizing your business to be at its best has become more critical than ever. But there are so many marketing assets to manage with such limited time -- ad copy, promotional campaigns, landing pages, etc. 

We all know that relying on instincts is suboptimal, but collecting sufficient data to make informed decisions can be arduous. Fortunately, there is a method to help you make data-backed changes at scale: A/B testing.

In this article, we'll go through everything you need about conducting effective A/B testing:

• What should you A/B test
• How to conduct A/B testing
• Best A/B testing tools
• And more!

What is A/B Testing?

A/B testing (i.e., split testing or bucket testing) refers to comparing two or more variations of an element (website, product, campaign, etc.) to randomized audiences. The goal is to identify the best-performing version, which will then be rolled out to the broader customer base.

While usually associated with online marketing, A/B testing is actually a common testing technique in advertising, medicine, agriculture, and even politics! Before you read on, it's essential to understand some standard terms of A/B testing.

How A/B Testing Works

A/B testing involves creating a hypothesis and identifying the relevant metric. Then, you show the original version (A, the control version) to one random group of customers and the variation (B) to another. Once the test is done, you use the most effective version for all customers.

Here's an example of a split test that I ran at Howuku. I hypothesized that readers would prefer a shorter article with more apparent how-to steps (hypothesis). This should result in improved time-on-page (metric).

So, I set the original version as the control and the new version as the variant. 

At the end of the test, I found out that the original version actually outperformed the variant. In other words, my well-merited hypothesis had been disproved, and I avoided an SEO crisis through A/B testing. 

Benefits of A/B Testing

A/B testing provides a wide range of benefits, whether you're a marketer, product manager, or e-commerce store owner.

• Improve business metrics: Split testing helps you optimize essential performance metrics such as user engagement, cart abandonment rate, and sales.
• Personalization: A/B testing allows you to assess the most effective variations for different customer segments, allowing you to craft personalized messages and strategies. 
• Safeguarding from mistakes: As illustrated in the earlier example, A/B testing helps prevent changes that may be detrimental to your business. 
• Optimize ad and email campaigns: Testing elements such as headlines, images, and call-to-actions (CTAs) can help you identify the best-performing variation for higher click-through rates (CTR). 

Overall, A/B testing is an indispensable tool to help businesses make data-driven decisions at scale. With all guesswork eliminated, you can be confident that any changes will move the needle on your desired metric. 

What Should You A/B Test?

In a perfect world with unlimited time and budget, you should be A/B split testing every single thing. Obviously, that's not possible, so we recommend experimenting with elements that have the most impact on user behavior:

Web page design: Split test your landing pages, product pages, homepage, and blog layout for various target audiences, especially after a web revamp project. 

Call-to-actions: Experiment with various offerings, different pain points, and button colors or layouts. 

Headlines: 80% of readers never make it past the headline, be it email subject lines or landing page headers. Identify the winning headline, and you'll make strides in your CTR.

These are just a few examples of the many things you can and should A/B test. The key is identifying the elements that contribute the most to your desired metric. Then, explore possible alternatives of those elements and make refinements from there. 

Steps in A/B Testing

There are three broad steps in the A/B or split testing process. They are:

1. Setting up an A/B test

2. Implementing an A/B test

3. Interpreting and reporting results

1. Setting up an A/B Test

Before you begin any testing program, you must clearly understand your business goals and the primary metric to uplift. 

Identifying Your Goals and Hypothesis

Briefly go through your marketing strategy and identify underperforming areas. Decide on a goal, which is usually to uplift revenue or conversions. Then, identify relevant elements that may affect those metrics (e.g., content, time, user experience) and craft a hypothesis.

The hypothesis is a prediction of what will happen after making a change. For example, "providing freebies rather than discounts will result in higher conversions."

Your hypothesis should also be SMART -- specific, measurable, achievable, relevant, and time-bound. This ensures that your marketing efforts are directed into relevant areas, not simply improving conversions for vanity's sake. 

Choosing an Appropriate Test Design

There are several different test designs that you can use, and each comes with its strengths and weaknesses. The most common test designs include:

Split URL test: Same as A/B tests, but you create two different URLs for the control and variation. Website visitors are then randomly directed to one of the URLs. 

• Pros: Suitable for drastic changes in web-design 
• Cons: May negatively impact SEO if not set up properly

Multivariate test: In multivariate testing, you simultaneously test multiple variations of a single element. 

• Pros: Testing several variations to hone in on the best version without setting up multiple split tests.  
• Cons: Complicates your testing process and potentially confuses your marketing team. 

Determining Sample Size and Duration

There are a few factors to consider when determining the sample size and testing duration:

1. Traffic volume: The more website visitors you have, the easier it is to achieve statistically significant results in a shorter period.
2. Conversion rate: The higher your website's conversion rate, the fewer visitors you need to achieve statistically significant results.
3. Confidence level: A higher confidence level requires a larger sample size. A standard confidence level used in A/B testing is 95%.
4. Margin of error: A smaller margin of error requires a larger sample size.

We recommend using a sample size calculator like this one. But if you insist on doing it manually, VWO has a great guide on calculating sample size the scientific way.  

2. Implementing an A/B Test

Once you have identified your testing goals and sample size, it is time to implement the testing program. 

Creating and Implementing Variations

Depending on your goals, creating the variations for your A/B test can be as simple as changing the color of a button or as complex as redesigning an entire page. You can easily create variations using free A/B testing tools such as Howuku. 

Collecting and Analyzing Data

As visitors interact with your website, the A/B testing software (e.g., Howuku) will collect data on their behavior and the performance of each variation. This data will be used to determine which version performed better.

Once you've attained the required sample size or statistical significance, you can stop the testing and proceed to the next step: interpreting the results. 

3. Interpreting and Reporting Results

Communicating Results and Making Decisions

Once the testing is done, you can interpret the results and determine which version performed better. If one version has a significantly higher conversion rate than another, this version will likely be more effective at achieving your goal.

However, if there is no significant difference between versions, further testing may be necessary to determine which performs better.

Finally, it is essential to document and report your A/B test results to the relevant stakeholders. This should include a summary of what was tested, how it was tested (including sample size), what metrics were used to measure success (e.g., conversion rate), and any insights or recommendations based on the results.

Successful A/B Testing Case Study: Brookdale Living

Brookdale Living is a renowned UK-based senior living company that wanted to improve its website's conversion rate. Specifically, their "Find a Community" page did not generate leads nor encourage site visitors to learn more.

Brookdale Living contracted marketing agency Fathom, who redesigned the page to make it more aesthetically pleasing and added social proof, among other things. However, they were unsure about using a static hero image or a customer testimonial video. 

Conventional wisdom would dictate that videos would be more effective in connecting with consumers. And if you conducted a survey on which content type people preferred, chances are they'd picked video over a dull stock image. 

It's a no-brainer -- videos are more effective in capturing attention and conveying a message, right?

Nope. In fact, the static image variant outperformed the control and video variation, resulting in a 3.92% lift and an estimated $106,000 additional revenue. 

As marketers, it can be tempting to rely on past experiences and well-intentioned best practices. But at the end of the day, only the results matter; what works for your business may not work for others, even if it flies in the face of common sense. 

The Fathom and VWO team hypothesized that the reason was most likely that site visitors had slow internet speed. Thus, a video would only result in a painful watching experience, and a simple image would be more effective. 

Read the entire case study here. 

Popular A/B testing tools

As you know by now, A/B testing is an effective way to optimize websites and improve conversion rates, but choosing the right tool can be difficult. In this section, we will compare three popular A/B testing tools to help you determine the best fit for your business.

Howuku (free plan available after 14-day free trial)

Howuku is a cost-effective A/B testing solution that provides a comprehensive list of features, such as heatmaps, visitor recordings, and survey feedback. Their all-in-one platform is excellent for scaling at marginal costs and streamlining most tools into one place. They also provide a freemium plan after their 14-week free trial, which is perfect for those learning the ropes on A/B testing. 

Optimizely (pricing available upon request, annual billing only)

Optimizely is a leading A/B testing platform that features advanced experimentation and personalization tools. Despite its powerful capabilities, Optimizely is easy to use and provides a wealth of resources and readily available customer support. However, you can expect to pay upwards of $50,000 yearly, so this choice is best for enterprise users. 

VWO (free plan available, annual billing only)

VWO is another popular A/B testing alternative that contains all the tools you need for an effective testing program. They also provide a free plan (up to 50,000 monthly visitors) which is great for those still starting out. However, their paid plans for more monthly visitors are only billed annually and can be quite pricey, but still less than Optimizely. 

Helpful resources for further learning

A/B testing is a mix of science, marketing know-how, and lots of experimentation. The best way to learn how to conduct effective split or multivariate testing is to try it yourself. That said, there are helpful resources that offer valuable insights on conducting effective A/B testing.

A/B Test Like a Pro by Firebase: This video series by Google is great for those interested in learning A/B testing on Firebase. Do note that Firebase only works for mobile apps. 

A/B Testing: The Most Powerful Way to Turn Clicks Into Customers by Dan Siroker: Who better to learn A/B testing from than the co-founder of Optimizely? This book by Dan Siroker contains real-world examples and applicable lessons that are valuable for both new and seasoned marketers.

A/B Testing Course by Google: This free online course by Google covers everything you need to know about A/B testing, such as ethics, experiment design, and result analysis. However, the course syllabus leans more towards computer science as it is part of their programming nanodegree, but it is still worth enrolling in, even if you're from marketing. 

Conclusion

In conclusion, A/B testing is a powerful tool for optimizing websites and improving conversion rates. By testing different variations of a website, businesses can determine what changes lead to better performance and make data-driven decisions about their online presence.

To get the most out of A/B testing, it's essential to use the right tools. One such tool is Howuku, a no-code A/B testing platform that makes it easy for businesses. to run tests without relying on technical expertise. Moreover, Howuku also automatically determines the statistical significance of tests, so even users without a background in mathematics can easily conduct A/B tests.

If you want to improve conversions and increase revenue, try out Howuku's A/B testing for free, with no credit card required. 

FAQ

What is A/B testing and why is it important?

A/B testing compares two versions of a webpage or element to determine which performs better. By testing two or more variations, businesses can make informed decisions about optimizing their websites for better performance and conversion rates. 

How do I set up an A/B test?

To set up an A/B test, you need to identify the hypothesis or goal of the test, select the elements to test, decide on the sample size and duration of the test, and finally set up the test using a tool such as Howuku. 

How do I calculate the statistical significance and confidence interval of an A/B test?

You can either do so manually or with an online calculator. However, most A/B testing platforms will automatically determine the statistical significance of your testing campaigns. 

What are common elements to A/B test?

Common elements to test include headlines, images, call-to-action buttons, layout, color scheme, and copy.