Startup Marketing Essentials: #9 Establishing a Testing Culture

Tim Matthews

A/B testing is en vogue. Everyone’s doing it, or at least taking about it. Putting it into practice requires more than talk and tools, though. You need to create a testing culture to really make it work.

For those who are unfamiliar with A/B testing, welcome to the party. The idea is simple: test out two variations: A and B. Pick the winner – sometimes called the champion – and go with it. Then do it again, aka iterate. The champion is the new A, and you create a challenger B. And so on until you reach the point of diminishing returns and move on to something else to test, e.g., your signup forms rather than your email subject lines. A more advanced form is called multivariate testing, where you are testing combinations of multiple A’s and B’s at the same time. (I could geek out all day on A/B testing, but I’m going to leave it at that for now. If you want to learn more on testing and measurement, I wrote a more detailed piece you can find here.)

A/B testing is actually quite old. So it’s more accurate to say it’s back in fashion. Direct marketing greats like Les Wunderman perfected the art through the 1950’s and 60’s. It’s really effective, and I’m happy to see it so popular again. Because it works.

So, back to the culture. A dirty little secret about A/B testing is that it’s a lot of work. At least double for everything you do, and even more when you think about all the data collection and measurement required to make it work. And what happens when you ask people to do things that require more work? (Just exercise five days a week and you will lose weight!) They don’t do them. Or, they get discouraged because it doesn’t work. Or both.

To avoid these pitfalls, you need to establish testing as a part of you culture. You need to lead the way and require your people have conversations based on data and results rather than opinion. You need to change the mindset from task completion to continuous improvement. Here are a few techniques I found useful in making this shift in my organization.

• Set metrics for everything – Yes, everything. Not just conversions. What type of content is read more? Do videos get more views on YouTube or Wistia? Helicopters or T-shirts as giveaways at trade shows? Now, I know what you are thinking. More metrics means more toil means you will just burn people out. Quite the opposite. It will empower them with data. We’re not doing helicopters at the hackathon, Tim. T-shirts pull 37 percent better…In other words, arming employees with data allows them to stop doing busywork or wasting time on projects that don’t move the needle.
  
• Test, test, test – You are never done. This can be demotivating, so be careful. You may reach the limits – go asymptotic (How’s that for high school math recall?) – of certain tests. Rather than going for that last 0.001 percent gain, try something else.
  
• Encourage experiments – Marketers like to do cool new stuff. Experiments can be fun (Okay, I admit, you may need to work on the art and English majors on your team.). But let people know they can try out cool new stuff. If it works, great. If not, good for trying (more on that next).
  
• Establish a safe zone – It’s okay to fail. This is key. Experiments fail. Scientists fail on their way to great discoveries. You often learn something from a failure, or at least you debunk something that was assumed to be true. Culturally, you need to let people know you’d rather have them fail on two experiments on the way to one great one, than not try at all. (I think I’ve channeled both my mom and Teddy Roosevelt here, but you get the point.) You may find this more challenging than you expect. Marketing folks, in my experience, are more likely to exaggerate their accomplishments, and may be uncomfortable admitting failure.
  
• Encourage employees to speak or blog about their tests – The great thing about testing is that you have data. And data makes you an expert! Okay, maybe not an expert, but it is a vehicle to share ones expertise. Encourage your employees to speak at meetups and blog about their experiments. You will be helping them develop their careers, which is also part of your job
  
• Train employees to know what questions you will ask – This last one is my favorite. Get your voice in their head. What would Tim ask? You want your employees to challenge their own assumptions and results. After a while, you will find employees have answers to everything you ask. They may even finish your sentences.

There’s a reason A/B testing is all the rage. It works really well and can be a very effective marketing weapon against your (lazier) competition. So if you are not doing it, time to start. Just keep your people in mind as you do.

Have a cultural tip to share? Or maybe a better headline to challenge the one at the top of this post? Let me know. I’d love to hear from you.

(135)