I’ve noticed that advertisers have increasingly realized that measuring incrementality is just as important as measuring attribution. 

But there are some channels where this realization has yet to fully take hold — and one of those channels is email. The only way to evaluate the effectiveness of an email campaign is to isolate the sales resulting from the campaign from the sales that would have occurred anyway. It’s complicated, but it’s also the only way to guarantee that resources are properly allocated.

There are a few reasons why most companies aren’t analyzing the incrementality of their campaigns. The main one is that it’s hard. Testing campaigns in the real world invites the opportunity for bias and pollution, requiring careful experimental design. Also, it’s difficult to separate attribution and incrementality. 

There’s also the fact that different channels require different tools. Using placebo-based testing doesn’t work for search, for example. Lastly, good incrementality measurement involves both real and opportunity costs. Real costs include buying necessary technology and paying your analysts. Opportunity costs include not advertising to users in holdout groups.

Pick a test campaign. The simplest option is a one-email campaign for, say, a product launch. Try to pick something that doesn’t involve a lot of other email campaign stimulation before or after the test campaign. 

Separate test and control populations. A randomly selected control population of 10% should be good enough (50% is ideal, but the opportunity costs are hard for advertisers to accept). The right number will depend on the size of your email list and your audience’s general propensity to convert. Then run the campaign.

Give your audience time to respond. If you do a lot of email marketing, you should know roughly how long it takes your customers to respond to emails. This informs the look-back window and therefore the run time of the test.

Match purchase activity to the test and control populations. Check the convertor rates, conversion rates, product purchase rates and revenue purchase rates. Comparing the rates for test and control will give you the incrementality rate.

Analyze your results. Once you have the incrementality rate, the fun begins. Your email campaign management platform can tell you not just who was in the test and control cells, but who ignored your email, who opened it and who clicked on it. That means you can calculate the incrementality for customers who just saw the email title, those who saw the body of the email and those who clicked.

By using this methodology across different campaigns, you’ll be able to understand the incrementality of email as a channel and also how it varies by seasonality or other factors. 

All marketers should have a plan for accurately measuring incrementality, not just for email but for all of their channels. The strength of their sales funnel depends on this data.