Jeanne Jennings: Scientific Method: Thou Shalt Not Make Changes Without Testing Into Them

“I have more than enough time to do everything I want to do to make my email marketing program more effective.”

-- said no email marketer ever 

The key to success is working smarter, not harder. This is one reason that I love scientific method – if you follow it, you are pretty much forced into a smarter way of working.

I touched on scientific method in my last blog post here; today I want to focus on just one tenet of it, the idea that you never make changes to your program without testing into them. This forces you to work smarter in 2 ways: 

1. You’re saving time, because you are constantly repurposing past messages – resending them with only minor necessary changes -- rather than starting from scratch with every email.
2. If you have hypotheses about how to improve performance, you are including these in a second version of the email, to test against the control -- allowing you to ensure that you are consistently improving your email marketing performance.

Basically, this tenet requires you to do a split test (control versus test) on any discretionary (not necessary) change before you roll out with it. Whichever version wins moves on to be the control for the next send.

When you ignore this tenet, you risk your program performance ‘blowing in the wind’ – high, low, high, low, all over the map. There’s not logic, no checks and balances to make sure that the changes you make are boosting bottom line performance and not decreasing revenue (or whatever you are using as your key performance indicator).

So why would marketers ignore this tenet?

Some people believe it requires single, not multi, variable testing and argue that that will slow their learnings and keep them from optimizing performance quickly. It does not.

Single variable testing is making just one change; this allows you to gauge the impact of this one change on performance.

Multi-variable testing means making 2 or more changes – in many cases with my clients, we’ll use a completely different creative execution or leverage Taguchi theory – and then see which one wins. In multi-variable testing you won’t know which elements of your test version drove the performance, but you will know which version ‘won’ overall. And that’s enough to honor this tenet.

Other marketers reject this key tenet of scientific method because they don’t truly understand how to honor it. They try to apply it in a very literal sense and then throw the whole thing out when that isn’t rational. It’s important here to make a distinction between ‘housekeeping’ changes, which are necessary, and ‘discretionary’ changes, which are not.

Let’s use an example to illustrate the difference. 

Here’s an email I received from the American Institute of Architects in June of last year (2016), announcing the call for submissions for this year’s awards. 

Let’s say that this email was good at driving submissions. If that’s the case then they should use it as a control this June (2017), for the 2018 Awards.

Necessary changes, which you would make without testing, would include:

• Changing the ‘2017’ date to ‘2018’ wherever it is featured with respect to the award year
• Changing the copyright date from ‘2016’ to ‘2017’
• Perhaps they are adding a category for Leadership in Energy & Environmental Design (LEED or Green Design). That could also be added in without testing.
• Since the image is the Twenty-five Year award winner from 2016, you would also want to update this to be the 2017 Twenty-five Year award winner.
• If any branding guidelines had changed (AIA rolled out a new typeface and logo back in 2015), you would need to update these elements of the email without testing them.

Let’s say that you believe the email would perform better if the headline (“Call for submissions…”) was rich text, instead of part of the image.

That would be a hypothesis that you could test, either alone or with other changes, in a test version of the email. It is not a change you would make without testing, because it might depress response instead of improving it.