Ten Best Practices in Multivariate Testing

Kim Ann King

Multivariate testing enables you to test many changes on your website simultaneously. Evaluating the impact of combinations of factors and variations often reveals significant interaction effects that can have a dramatic impact on your website’s conversion goals. SiteSpect has developed the following 10 best practices to help you get the most out of your testing program.

Best Practice #1: Form a Great Testing Team

Your testing team should have a project manager, technical talent, executive sponsor, and above all, a mandate for improvement. This means the testing team needs to be charged with improving the website’s condition in a measureable way and be able to clearly demonstrate and communicate results to stakeholders.

Best Practice #2: Get Your Stakeholders On Board

Like any corporate endeavor, without management’s support for your testing initiative, you can essentially guarantee failure. Be sure to work with management from the very start. Illustrate to them exactly what you wish to achieve, communicated in bottom-line financial predictions that will help your company better compete in the marketplace.

Best Practice #3: Write a (Formal) Testing Plan

There needs to be a structure through which requests for testing are considered; SiteSpect suggests a testing planning template, which covers at a minimum, the following questions:

1. What are the elements being tested? For example, an image, a headline, navigation path, on-site search?
   
2. Why are the elements being testing and what are the expectations for the test? This is where you form your hypothesis; for example: “We believe that increasing the font size, color, and placement of the headline will have a positive effect on user engagement, and increase time on the page by 10%.”
   
3. Who are the people, processes, and technology behind the test?
   
4. By when are the results needed?

Best Practice #4: Think about Measurement

Your web testing program should integrate and collaborate closely with your overall web analytics efforts. Metrics and KPIs that have been established within a web analytics program should dictate to a large extent which tests should be undertaken. This takes extra consideration and possibly technical expertise to ensure data available through analytics tools (such as audience segments) can be applied to tests.

Best Practice #5: Clearly Define “Success” and “Failure”

Success means different things to different stakeholders. It can range from obvious financial gains, to increased user engagement, to fewer support calls. But even a “failed” test is helpful, because you’ll better understand what didn’t work. The only real failure is a test that is not properly designed, carefully executed, done within a vacuum, and/or without executive buy-in.

Best Practice #6: Test Your Test

Testing can be hard and it won’t solve all the issues of the website in a month, quarter, or even a year. This is a long-term shift in the thinking of the company to become data-driven. Certain technical implementations of tests may be trickier than others and will involve the IT department. Set expectations and be patient.

Best Practice #7: Clarify Your Testing Timeline

Testers often forget to take day parts and weekends into account. To ensure you’re not giving too much weight to lunchtime, early morning, late night, or weekend visitors, allow for enough time to collect a representative sample. We recommend a 7+1 strategy so you test over an entire week, but also add one day at the beginning of the test before you actively track results.  This allows the test versions to take hold online across the sample and for any last-minute changes.

Those with more time available can opt for a two-week test, in this case a 14+1 timeline.

Best Practice #8: Communicate Your Test Results with Actionable Analysis

When communicating the results to the stakeholders who originally requested the test, go into significant detail. The rest of the organization should receive a broad overview. Whenever possible, communicate the financial impact of the test and include actionable recommendations along with your results.

Best Practice #9: Test Different Audience Segments

The most revealing tests will be those that utilize targeted audience segmentation. Knowing that a certain change increased conversions by 5 percent for all visitors is somewhat helpful, but knowing that same change resulted in a 20 percent increase for a key target audience is far more valuable.

Best Practice #10: Mine for Deeper Opportunities

Once the testing program is up and running, and the team has a few wins under its belt, in-depth data analysis and statistical modeling is the next level to strive for. Companies that are already utilizing robust analytics have an advantage, but this is not an absolute requirement. Analysis and comparison of offline data or qualitative voice-of-customer data alongside test results can yield insights that are not immediately obvious through the testing or web analytics tools alone.  There is a significant competitive advantage for companies who truly leverage their online data.

These testing best practices are derived from a whitepaper sponsored by SiteSpect. To obtain a copy, please visit www.sitespect.com/testingsuccess.

(262)