Make testing work

Email marketers know that it takes data to determine just the right timing, cadence, offer, and design.

Important marketing questions can be answered by gut feel, but that usually gives the fastest – rather than best – answer. Just look at WhichTestWon’s Test of the Week. More often than not, my gut is wrong. What about you?

The challenge always comes where the rubber hits the road: Day in and day out, how do I as a marketing leader get the data I need? When I speak with clients, they often talk about how difficult it is to test when there are so many daily fires to put out. As an industry and a service provider, the message is that we need to produce tools and approaches that make testing easier to incorporate into the daily routine. But what is the takeaway for a brand marketer?

A recent experience working with a non-profit client can help illuminate. The question: Should the email donation button be blue or brown?

  • Brown blended with the background nicely; but does that mean it would be lost to recipients?
  • Blue popped against the background; but was it too much?
EmailTestbox code to switch images and track clicks

EmailTestbox code to switch images and track clicks

Conventionally, the client would have had to hold out a group, conduct the A/B test, then send the balance of the campaign based on initial results. However, using EmailTestbox , the client had the opportunity to send out the campaign all at once; two small pieces of HTML code would do the work of switching out the button images and collecting response. When statistically significant, EmailTestbox would then show the winner to all subsequent openers.

The result? Brown wins! Recipients who rendered the brown button were ~50% more likely to click to donate than a random recipient, and ~200% more likely to click to donate than a recipient who rendered the blue button

Because the test was in real-time, this statistically significant result was captured within hours of the send, ensuring that all subsequent openers saw the brown button.

The lesson for brand marketers?

  1. Minor changes matter. The only change between the two creative versions was the button color, but it produced dramatically different results. Start small, and iterate.
  2. Apply technology where it makes sense. This was a simple A/B test, so using a real-time testing application like EmailTestbox may have been more firepower than needed. However, it also prevented additional work and setup on the campaign side. If your testing goals can be accomplished low-tech, by all means go for it! When you have more complex testing needs (e.g., multivariate), bring technology to bear.

Related Posts

Testing is rather like going to the gym. We know it’s good for us, we know we should do it more often, but…our heart’s just not in it. So what’s stopping us? First, some are unconvinced of the real benefits. Second, testing can drop down the priority list when 101 other tasks demand your attention. Third, there’s […]

At the Risk of Repeating Yourself… Have you noticed something different in your inbox lately? Actually, it’s not so much “different” as “the same.” That is, marketers resending email campaigns that you didn’t open, click or convert on the first time. Resending email campaigns to boost results is a hot topic now, and we have […]
Chat with us