Ecommerce Personalization Blog
Ecommerce tips, strategies, and news – all without ever having
[Guide]: Powerful AB Testing for eCommerce
What are the most effective ecommerce ab tests? While you can use any AB testing framework with Barilliance, we've found a number of best practices that generate the best customer experiences.
First we define testing concepts such as AB testing, muti-armed bandit, and audience sizes. Second, we will detail the benefits AB testing promises. Finally, we will break down AB testing best practices, and illustrate with examples.
If you'd like to skip straight to the AB testing best practices, click here.
What is AB testing? (in an eCommerce environment)
AB testing is a specific type of controlled experiment. AB tests are used to determine the best option out of a possible range. In an eCommerce context, this could be determining which offers to show, what products to recommend, or even which product images or layouts work best for a given success criteria.
Like other multi-arm bandit tests, there are two distinct parts. First, there is an exploration phase where multiple options are considered. Once a winner is determined, the second phase of exploitation starts. In this phase, the winner is chosen and the loser is discarded.
Properly constructed AB tests are comprised of the following parts:
“AB tests are used to determine the best option out of a possible range. ”
While defining an AB test is nice, it is helpful to give a few illustrative examples for how AB testing can benefit eCommerce stores.
AB Testing Benefits: How To Use Testing to Improve eCommerce Stores
Recall that AB tests determine the best option out of a possible range. With this in mind, we can quickly go through a number of AB testing opportunities.
We could test what offer resonates best with returning customers.
We could test which products recommendations convert the best for a given product sku.
Customer experiences can also be tested. For example, we could test to see if a welcome pop-up performs better than a message bar.
AB Testing Benefit 1: Improve ROI from Current Traffic
As a fundamental tool in conversion rate optimization, testing maximizes your ability to monetize existing traffic.
Using AB testing, you can:
2016 homepage
2018 homepage
AB Testing Benefit 2: Improve customer experiences
As a tool for personalization software, testing unlocks data informed improvements to your customer experiences.
While Barilliance utilizes machine learning to automatically optimize personalized experiences, you also have the choice to override machine based experiences with rule based variants.
This is especially helpful when you have upcoming events, holidays, new products, categories, or campaigns.
In these instances, it is important to leverage testing to get the best results.
AB Testing Benefit 3: Improve off-site customer interactions
Finally, AB testing can, and should, be applied to all customer experiences. To effectively execute omnichannel strategies, testing capabilities are absolutely necessary.
A few off site AB testing benefits include
Which AB Tests should I perform first?
Every business has constrained resources.
In testing, the quality and number of tests you can run will be limited by:
Given these constraints, the next question is what tests should you perform?
While your business context will be unique, here are a few guideposts to help you identify the best AB tests to run.
1. High traffic touch points
What parts of your customer experience do the most customers experience?
For example, if one product category accounts for 75% of sales, you should prioritize AB tests in that area first. On the other hand, if other categories currently receive marginal traffic, wait to optimize these parts until you've secured wins in higher trafficked areas.
2. Important customer segments
Second, what customer segments are important to your business?
We recommend using RFM Analysis to determine the most important customer segments for your business.
3. Identify bottlenecks
Finally, consider what bottlenecks exist in your business.
Bottlenecks are places where every customer must bass through. For most eCommerce stores, this includes your checkout process, product pages, and possibly ads.
AB Testing Best Practices: How to conduct an AB Test the right way
There are many ways to be successful with eCommerce optimization. However, we've found these best practices lead to more consistent, faster results.
1. Define your success criteria
Ask yourself, "What will make this test successful?" You should identify the single best KPI to evaluate a test.
Typically, your evaluation metric will be a composite metric, like return on marketing investment. It should be predictive of long-term outcomes.
The Harvard Business Review has an excellent writeup detailing a number of challenges Bing faced in determining which metric to prioritize.
While at first it might seem that revenue generated would be a good metric, they found that they could manipulate revenue by adding more ads. Unfortunately, the metric didn't fully capture the costs of these improvements in revenue - namely a worse customer experience and ultimately less people using Bing.
The team came up with an alternative. They wanted to focus on the customer experience first, and so the primary metric was to minimize the number of queries needed to complete any one session, yet maximize the number of sessions per user.
2. Balance explore vs optimization - consider big levers.
There is a constant tension between exploring new possibilities and exploiting previous findings.
With AB testing, it can be tempting to shy away from big redesigns and instead focus only on smaller optimizations. However, the huge breakthroughs are more likely to come from big levers - large changes in your offer or design.
Make sure you leave room for these larger experiments.
3. Optimize your sample sizes
One common misconception is that your control and test audience sizes need to be the same.
The truth is, your control represents the winner of all previous AB tests. It is a tried and true effective experience that results in sales.
Because of this, it is most common to dedicate more traffic to the control group, while looking for improvements with a smaller percentage of traffic.
On the other hand, if you want to increase the throughput of tests you can conduct, you can increase the traffic dedicated to test groups.
4. Don't trust averages. Segment your results
Your AB tests will have mixed results. It will positively impact some visitors, and negatively impact others.
If you just trust the aggregate numbers, you run the risk of alienating whole customer segments.
The answer is to break down the results by customer segment. If you don't have established segments defined in your personalization tool, I suggest running through a RFM Analysis to identify high priority segments.
Ultimately, you want to isolate a defined customer group that the experiment works on, and then make the winner show to that segment and that segment only.
Barilliance connects your offline and online customer data. You can define as many segments as you like, automatically enroll customers based on their actions, and trigger any number of marketing campaigns. Learn more here.
Successful AB Test Examples
We have a running collection of successful eCommerce AB test examples here.
In that guide, we break down in detail the example tests. Here, we will briefly showcase tests from larger eCommerce stores.
How MOO uses AB testing to improve conversions
Moo is a leading eCommerce store in the business card and stationary space.
Above is an annotated screenshot of their current home page above the fold. Below we have the same page, just one year ago.
Comparing the two pages, there are six major improvements the team at Moo have implemented. For the full break down on what these improvements are and why they work, click here.
Next Steps...
To properly perform AB testing, you need a personalization partner.
We wrote a detailed guide on how to select a personalization partner for your eCommerce store here.
However, in short, you want a technology that is able to connect your data in one place, identify profitable segments, and provide personalized experiences and AB tests targeted towards those segments.
If you'd like to see why hundreds of enterprise eCommerce stores have chosen Barilliance as their AB testing solution, request a demo here.
You Might Also Like
