How to Run A/B Tests That Matter

You can A/B test anything if you really want to. However, the truth is that an awful lot of tests deliver inconclusive results, or worse still don’t make much of a difference to what actually matters – the bottom line.

To illustrate that, let’s take one common example. By testing the subject line of an email, we don’t necessarily learn much. Of course we might improve the performance of that specific campaign by sending the winning variant to the remainder of our audience (although in most cases the numbers used to make these decisions are nowhere close to large enough), but beyond that we aren’t any the wiser when it comes to the next time around.

The reason clever businesses (and I am going to include Swrve customers in that group!) A/B test isn’t just to improve a single campaign. It is to enable those organizations to implement informed changes to the mobile app experience on an ongoing basis. Changes that will engage users, retain users, and ultimately – drive revenues. And do so permanently.

So, what elements of your mobile app should you test in order to deliver maximum ROI to your business? Our customer success team has worked with many of the world’s leading app businesses, so we know what works.

• Your onboarding process, or what a user sees the first time they open the app
  
  
  
  • How many initial screens are most effective?
    
  • Within these screens – what content are users most responsive to?
  
  
• Getting ‘free trial’ users to subscribe
  
  
  
  • Which user groups should be targeted?
    
  • What is the optimal time to deliver campaigns?
    
  • Which campaign creative is most effective?
  
  
• Push notification opt-in
  
  
  
  • Should you request this access during onboarding, or wait until the user is familiar with the app?
    
  • Should you provide the user with details of what kind of push they might receive? In the example below, a customized request for push notification opt-in on the left is compared to the defauly iOS experience on the right.

Measurement is the be all and end all when it comes to giving your customer a top notch mobile experience. And only they can tell you what they want – so listen, learn and keep improving. That’s what A/B testing is all about.

And the good news is we’re here to help! To learn how to run A/B tests that give you real insight, and how to avoid common pitfalls when testing, why not tune into Alex Gray, our Director of Sales Engineering on “How to Run A/B Tests That Matter”. April 14th 3pm GMT- register here!

(27)