A Detailed (Step-By-Step) Look at Effective Ecommerce A/B Testing

David Hoos — May 31, 2018 Follow @davidhoos

— May 31, 2018

Let’s start with the good news.

You’re running a fairly successful ecommerce site. You’re making a decent amount of revenue and things are chugging along nicely.

But it’s not all bubblegum and roses.

Frankly, you’re having challenges.

Your store’s ecommerce conversion rate – and thus your conversion funnel – isn’t that great.
Carts get abandoned more than you’d like.
And your return on ad spend is only average.

You know things could be better, but you’re not sure what steps to take.

We’ve got some good news for you, and it’s called A/B testing.

You’re probably familiar with the term.

You’ve heard about the simplest examples of A/B testing like changing button colors and calls to action. But most ecommerce businesses don’t know how to do it effectively.

This is unfortunate, given that, when done well, A/B testing – or split testing – is one of the most powerful ways to improve some of the most important metrics in your business.

That’s what this article is all about.

We’re going to give you a detailed look at A/B testing, why you should do it, some examples, how it works, what you can test with it, when to use it, and much more.

uber

This is your detailed roadmap to A/B testing.

Paige Gerber, Director of Content Experience, UberFlip

Always. Be. Testing. I can’t stress enough how important it is to test your messaging.

A/B testing is the only way to find out what your customers are looking to see before they hit the checkout and make a purchase.

Once you hone in on the copy, images, and call to actions that convert you’ll quickly see a direct correlation between understanding what your customers want and return on ad spend.

What Is A/B Testing?

Let’s start by making sure we’re all on the same page.

Most people think of A/B testing only in terms of small tweaks to button colors or calls-to-action or headlines, but it’s much deeper than that.

So what is A/B testing? Put simply…

It’s a method of determining which design, content or functionality is more successful with your site visitors. It allows you to test a variation of your page (or element on a page) that may affect your consumers behavior.

For example, it can involve:

Testing two different content layouts for the same product to see which layout produces more sales.
Or different product taxonomies for your site to see which one makes it easiest for customers to purchase from you.
Or moving navigation items around to see which results in more sales.

But here’s the thing…

A/B testing isn’t a one-time deal. It’s not one and done, like a college basketball player going pro.

Effective A/B testing ideally involves repeatedly testing improvements until you get the best possible version.

It’s an iterative process with each test building upon the results of the previous tests.

Like physical fitness, when A/B testing is done consistently and with focus, it allows you to incrementally improve your overall website design to better align with your consumer behavior and business goals.

When you start, you may not see massive gains; however, when you look back after several months, you’ll be amazed at the overall improvements.

Want more insights like this?

We’re on a mission to provide businesses like yours marketing and sales tips, tricks and industry leading knowledge to build the next house-hold name brand. Don’t miss a post. Sign up for our weekly newsletter.

Why Should You Do A/B Testing?

As an ecommerce store owner, you’ve got dozens of tasks on your plate.

Listing products.
Optimizing products for SEO.
Responding to customer inquiries.
And a thousand more.

Why should you bother with A/B testing?

Is it really that effective?

Yes, it really is.

A/B testing allows you to significantly improve your customers’ shopping experience.

Happy customers translates into better click-through rates, more conversions, more revenue, greater loyalty, fewer complaints, and much, much more.

Not convinced? Take a look at Amazon.

How has Amazon managed to evolve from a site that only sold books to the largest ecommerce retailer in the world?

Much of it is due to CEO Jeff Bezos’ insistence on constantly testing, experimenting, and innovating.

As he noted in Fortune magazine:

“Experiments are key to innovation because they rarely turn out as you expect and you learn so much…If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce.”

Bezos understands that, fundamentally, success comes from making your customers as happy as possible – making their shopping experience seamless and pain-free.

But to identify what makes them happy, you need to constantly hypothesize, test, experiment, and repeat.

Consider all the ways Amazon has made online shopping pleasant.

1-click ordering.
Dash buttons that allow you to reorder products simply by pressing a button.
Two-day shipping.
A hugely powerful recommendation engine.

The list goes on and on.

These innovations didn’t occur randomly through some sort of ecommerce version of Darwinism.

Rather, they were the result of rigorous testing and then taking action based on this evidence-based approach.

And Amazon isn’t alone in their commitment to generating outstanding results through conversion optimization.

The most innovative companies in the world deploy it constantly, and you’re regularly exposed to multivariate tests whether you know it or not.

Have you ever noticed how Netflix regularly changes the cover images on television shows and films?

There’s a reason for that. The official Netflix Tech Blog notes:

In fact, every product change Netflix considers goes through a rigorous A/B testing process before becoming the default user experience.

Major redesigns…greatly improve our service by allowing members to find the content they want to watch faster.

However, they are too risky to roll out without extensive A/B testing, which enables us to prove that the new experience is preferred over the old.

And if you ever wonder whether we really set out to test everything possible, consider that even the images associated with many titles are A/B tested, sometimes resulting in 20% to 30% more viewing for that title!

With an astonishing amount of choices available in just about every industry, Amazon and Netflix realize that generating customer loyalty is critical.

They want to give people a reason to come back again and again.

A/B testing paves the way for that loyalty.

Jamie Turner, Author, Speaker, CEO at Jamie Turner Live

The first secret to ecommerce success is testing your way there. That’s always a top priority.

The second is to spend 10% of your time/budget testing new techniques and technologies.

And the third is to be sure to share your successes and failures with everybody internally. That way, you can be sure you have a staff with a knowledge base.

How Does A/B Testing Work?

Now that you have a general idea of the what and why of A/B testing, let’s dive into the specifics.

There are two primary types of A/B testing: client side and server side, with client-side being the more common.

Client-side testing involves sending the exact same version of a page to every visitor and then using Javascript to make changes and tweaks within the visitor’s browser before the visitor is shown the resulting page.
Server-side testing is when your web server shows visitors different page variations, altering them on the server before they are sent to the visitor’s browser. No modifications to the page are happening in the browser.

Regardless of which type is used, the general methodology is the same.

Using analytics tools, such as Google Analytics and HotJar, you evaluate the performance of each page variation.

For example, you may send 50% of your traffic to variation A and 50% of your traffic to variation B.

Then you analyze the data to determine whether variation A or variation B is more effective according to a set objective (conversion), whether that’s sales, email opt-ins, click-through rates, etc.

When it comes to Netflix, they may test the movie Pulp Fiction with two different covers to see which one generates the most views. Amazon may test the number of fields in their checkout process. You get the point.

However, there is a caveat.

A/B testing hinges on what’s called statistical significance.

In order to be confident that test results are accurate and not the result of an anomaly, a certain number of visitors (the sample size) must be reached.

If this threshold isn’t met, A/B tests can be inaccurate, leading you to implement changes that can actually hurt your site.

The implication?

If your site doesn’t get enough traffic, you may want to consider first investing in paid advertising or search engine optimization to increase your overall traffic numbers.

Once you’ve reached the necessary levels of traffic, you can implement A/B testing at any point and see a significant return on your investment.

If you need help determining at what point you’d see an ROI, here is a calculator you can use to determine how an A/B testing can impact your website.

Better Decision Making Made Easy

Fast-scaling businesses use metric dashboards to guide their meetings and decision making – taking gut preference out of the process in favor of data-driven results.

Use this dashboard to see the same results for your brand.

Where Do You Use A/B Testing?

A/B testing is all about removing consumer pain points. Whenever you can remove friction from the shopping process, it significantly increases the chances of conversions. Again, this is where Amazon stands out.

They’ve done everything possible to remove common online shopping hassles such as:

Filtering to find a specific product that matches your needs.
Comparing similar products, or those that were purchased alongside the currently viewed product.
Determining the overall quality of a product (they implement reviews, questions, photos, etc.).
Repeatedly filling in billing information.
Waiting a week for a package to arrive.
And much more.

In the context of your own ecommerce store, you can use A/B testing when comparing:

Different page layouts.
Navigation organization (where your menu is, etc.).
Headline effectiveness.
Body copy.
CTA copy and design.
Website photography and product images.
New visual styling for a page.
New pricing strategy.
Different promotions/offers.
And much more.

The beauty of A/B testing is that you can systematically work through each of these areas, incrementally and consistently improving each one to increase conversions.

Stephen Slater, Sr. SEO and Digital Advertising Manager, TopRank Marketing

Just because it’s launched doesn’t mean it’s done. Always test something new.

If your not testing and evolving from day 1 you are going to lose.

Keep testing. Keep getting better.

So many brands launch and expect everything to work right out of the gate. That’s rarely the case.

Test and get better. Then do it again.

There are so many A/B testing tools out there. Pick one and know it. Google Optimize, Unbounce, Optimizely… Pick one and test everything.

A/B Testing Examples

A/B tests can come in many shapes and sizes.

Here are a few wireframes highlighting the types of A/B tests you might see as part of an optimization program.

Navigation A/B Test

In this test example, we aim to see whether consumers are able to better navigate a site and reach checkout by exposing the menu at the top level rather than through a drop-down menu.

Product Detail Page A/B Test

In this test example, we aim to see if placing more educational information about the product higher on the page will affect the add-to-cart rate.

Form A/B Test

In this test example, we aim to see whether consumers are more inclined to complete a form with fewer fields and simpler placeholder text within the fields.

When Should You Use A/B Testing?

Random A/B testing is always a bad idea.

It results in you spending an inordinate amount of time testing areas that matter very little while ignoring the biggest drivers of growth.

And because you’re testing random variables, it makes it difficult to determine which changes have had the greatest effect.

Before you begin testing, you need a roadmap.

This roadmap will guide you specifically through what to test and in what order.

How do you create such a roadmap?

By doing thorough research on your customers and how they behave on your site.

Some of the research tools include:

Analytics – Identify your most popular pages, CTR, traffic sources, pages with a high bounce rate, paths through your site, etc.
Heatmaps – Discover exactly where and how people navigate your site. See where they click, where they pause, where they get confused, etc.
User Tests – Personally watch users navigate your site to discover where points of friction occur. Proper user testing will also include the testing subjects speaking out loud about what they are experiencing as they complete tasks on your site.
Surveys – Directly ask your customers about their shopping experiences. Did they encounter any problems? What would they like to see improved?

After doing a thorough analysis of your customer’s behavior, you can begin to develop hypotheses about your current website and how you might improve the overall customer experience.

These hypotheses then form the basis of your A/B testing roadmap.

Each of them will be systematically tested. Those that are proven statistically valid will be implemented and those that aren’t will be discarded.

What Are The Benefits Of A/B Testing?

A/B testing works so well because it’s entirely based on numbers.

There’s no relying on gut feelings or instinct. Flashes of “inspiration” are only accepted if the numbers prove them out.

Changes are only implemented if they are validated by the data.

Additionally, A/B testing dramatically reduces the risks normally associated with redesigning your website.

Instead of a taking a big gamble on an entire site redesign, which ultimately could lead to your conversions tanking, you’re making small, incremental improvements.

Each of these improvements is geared toward improving the customer experience.

Finally, A/B testing cuts through the opinions of those paid the most.

Your position or salary don’t determine which changes are implemented. Only the data does.

What Are The Challenges of A/B Testing?

We’d be lying if we said A/B testing was a magic bullet and that one test will fix everything and cause your revenue to skyrocket.

The reality is that it’s an ongoing, incremental, iterative process.

It’s like going to the gym. A single workout isn’t going to do much for your physique. Random workouts aren’t going to help much either.

If you want results, you need a plan that you consistently implement.

The same goes for A/B testing.

Once you develop your roadmap, you need to follow it closely.

Areas of improvement will be tested systematically, and each test will build upon the results of the previous test.

It’s not sexy and it’s not as fast as just making the change, like in a fad diet. It’s a workout. And it’s proven to be effective.

And there are specific challenges involved:

1. It requires a fair amount of experience to do effectively.

There are a number of specific tools used in A/B testing, and some of these tools have a steep learning curve.

Additionally, there needs to be a fundamental understanding of web design, statistical analysis, and other domain-specific knowledge.

2. It’s a process.

Contrary to what many articles would have you believe, A/B testing is not a silver bullet.

A single test probably won’t do much for your revenue.

As mentioned above, it’s about developing a long-term plan and then sticking to that plan. It’s about targeting specific goals and improvements over time.

3. Traffic is required.

We touched on this above but it bears repeating. While the amount of site visitors will vary depending on the test and the objectives, you need at least a fair amount of traffic to ensure that your tests are statistically valid.

If your site has a low traffic volume, you’ll probably need to work on raising that before you can begin A/B testing.

Given The Challenges, Is A/B Testing Worth It?

So, given the challenges and struggles listed above, is A/B testing still worth it?

While we certainly understand the hesitance, we’ve seen again and again just how powerful A/B testing can be.

When done properly, it can have a huge impact on some of your most important Key Performance Indicators (KPIs):

1. Improved return on ad spend (ROAS).

The more friction you remove from your website, the more paid traffic turns into sales, effectively increasing your return on ad spend.

2. Reduced customer acquisition cost.

Similar to how testing helps improve ROAS, the money you spend on other methods of customer acquisition will benefit from the reduced friction and will shrink your customer acquisition cost (CAC).

3. Increased lifetime customer value (LTV).

Testing makes your website more sticky and engaging which helps drive more return business. In other words, the easier you make the shopping process, the more customers will want to come back again and again.

This increases your customer lifetime value.

4. Increased email signup conversion rate.

Your email list can be a huge revenue driver.

Testing different ways of inviting new subscribers will help you maximize your email list sign-ups which increases the people you can promote your products or services to.

5. Increased average order value (AOV).

Increased testing can help you find out what product bundles or upsell opportunities could work best, increasing your average order value.

At scale, this directly impacts your bottom line revenue.

6. Reduced cart abandonment rate.

Every time a person abandons their cart, that’s money walking out your digital door.

Using A/B testing to optimize your checkout process can remove friction, confusion, and mistrust, and reduce your cart abandonment rate.

7. Increased ecommerce conversion rate (CR).

Smoothing out any barriers on the path to purchase using A/B testing is the most effective way to increase your conversion rate.

As you can see, despite the challenges, A/B testing can make a massive difference in the growth and overall revenue of your ecommerce business.

How Can You Find The Right A/B Testing Solution?

When it comes to A/B testing, experience and skill is just as, if not more important than the tools themselves.

First, you need to determine whether you want to do the testing in-house or hire an outside agency to do the testing.

Evaluating Skill Sets

Regardless, of your choice, here are some of the key skills to look for in testing and optimization are:

Data research.
Statistical analysis.
UX design.
Behavioral psychology.
Project management.
Persuasive writing.
Conversion strategy for conversion optimization.
User research.
Information architecture.
HTML and CSS.
jQuery and JavaScript.
And more.

CRO is a robust process requiring a variety of skills.

Each of these skills contributes uniquely to the CRO process, and the absence of any of them can lead to ineffective A/B testing.

Using The Right A/B Testing Tools

There are also a number of specific tools required to do A/B testing.

The two most commonly used tools are Optimizely and VWO.

Both offer a full suite of A/B testing tools, including the ability to easily deploy tests, analyze the results, evaluate website heat maps, and much more.

Examples of Tools You Can Use For A/B Testing:

Optimizely.
VWO.

Conversion Sciences use some of these.

A few things to note about Optimizely:

Optimizely is a feature-rich, enterprise-level A/B testing tool, and is priced accordingly. It is widely considered the industry leader for larger sites.

That said, for many small to medium-sized businesses it can be like using a semi-truck when a pickup will do. We recommend considering whether you will realistically utilize its full suite of features before you select this tool.

A few things to note about VWO:

Visual Website Optimizely (or VWO) is an effective small to medium sized business A/B testing tool.

It provides a solid mix of features that will do all of the core tasks that a professional A/B testing tool should. While it does not offer as many premium features as Optimizely, it is often a more cost-effective alternative for many growing businesses.

Google also recently released a free tool called Google Optimize, which can be good for beginners but doesn’t provide the robust set of tools you need to truly do A/B testing effectively.

Whichever tool you choose, spend time researching the capabilities, pricing, and overall level of support for each.

Also, if you plan on doing your testing in-house, be sure you feel some level of comfort with the software you choose.

Using Conversion Rate Optimization Service Providers for A/B Testing

In addition to the proliferation of SaaS applications designed to help you implement A/B testing, a number of service based businesses provide A/B testing and conversion rate optimization as a ‘done-for-you’ service.

Be warned though: it has become fashionable for many marketing agencies to add CRO or A/B testing to their laundry list of services, yet have little more than beginner knowledge.

There is a simplistic way to do A/B testing and a more methodical and professional way to do it. Make sure you’re getting the latter.

Some things to look for in A/B testing services are:

Whether they offer CRO as an add-on to other marketing services or focus exclusively on CRO.
Case studies that demonstrate previous CRO success. These case studies should make clear the specific processes they used to achieve their results.
Whether they’re data-driven. CRO should always be data driven. Any methodology that relies on intuition, flashes of “insight”, or hunches, isn’t CRO. It’s random guessing that most often turns out to be incorrect.
If they have a defined CRO process. A true CRO firm should have a specific, defined CRO process that they follow again and again. If they don’t, it’s a sign that they probably don’t know what they’re doing.
Do they focus on the short term or long term. Optimization is about playing the long game. Shooting for quick results involves taking shortcuts, or not waiting for enough test results to make a proper recommendation, which almost always disrupt the process and produce inaccurate results. The best CRO firms are squarely focused on implementing a robust plan over the long run.

Bottom line: Don’t assume that because a digital agency says they do CRO, they actually do CRO. Do your research. Ask the right questions. The research is time well spent.

For an in-depth guide on how to evaluate a CRO agency, read this whitepaper.

Conclusion: Don’t Neglect A/B Testing

A/B testing is one of the most powerful and effective ways to drive ecommerce growth.

Some of the biggest, most profitable companies in the world have achieved their massive success in large part due to A/B testing.

Generally speaking, if Amazon does something, other ecommerce sites should pay attention.

A/B testing worked for Amazon.

It will work for you.