Can You Afford Not to Test?
By Pippa Nutt, Director, Online Strategy

Let me stop you right there: you can’t. Unless, of course, you are the lucky wrangler of a multimillion-dollar brand budget, and are therefore excused from any bottom-line ROI reasoning. As an aside, if you fall into the latter (lucky you) – I’m jealous.

But I digress. The concept of testing is not new. Today, almost every marketing publication is shouting for its readers to test, test, test. But while the message is loud and clear, for the most part, it seems to be all talk and no action.

If ongoing testing is not part of your marketing strategy, you could be missing out on substantial ROI gains. This is a big mistake – huge. A 10% lift in conversion rates would undoubtedly have an impact on your bottom line, would it not? That is the foundation for testing: focus on converting more of the traffic you are driving, as opposed to paying to drive more traffic.

My personal viewpoint is that, if more marketers had a better understanding of the true value of testing, and how attainable this process has become through online integration, they would quickly come to realize you simply cannot afford not to test – a lot. All the time. Starting now.

What’s involved?

There are many free and subscription-based online tools, like Website Optimizer, that enable precise measurement of split A/B and multivariate tests. I strongly recommend adding these to your tool kit as they, for the most part, take the statistical analysis out of your hands, which saves time and frees you up to focus more on defining the testing matrix, analysis and applied learnings.

Another huge advantage of these tools is the fact that they put the power back in the marketers’ hands. For the most part, you can leave your IT team out of the process (and buy your own celebratory beers at the conclusion of your first test – no groveling required). Just think how much more nimble we as marketers could be if we didn’t have to submit a three-page request form every time we wanted to make a website change.

What can I test?

The options are practically limitless, although some tests are more strategic than others and will provide more valuable insight into your marketing programs. Getting down to basics, there are really two core types of tests: split A/B and multivariate. Most of you are probably familiar with the former, which is essentially the process of testing two unique landing pages or websites against each other.

Multivariate testing, my personal favorite, involves testing multiple on-site variables at the same time – each variable in conjunction with itself and the original.

Deciding which test to perform when is pretty straightforward. A split A/B test commonly precedes the multivariate. Once a clear static winner has been identified, you can expand into a phased, multivariate testing strategy.

Split A/B tests are less common in my world as budgets are generally not limitless (but we can still dream!) and, over the years, we have come to develop and fine-tune a considerably narrow definition of what works online and what doesn’t. In other words, we don’t usually need to test two totally different concepts against each other to know which will outperform. That being said, split A/B tests can be very useful in certain situations – for example, as a divide between senior brand management and your web usability team. If you can’t reach consensus, and you can afford to do so (and the political climate supports this democracy), test both concepts out of the gate.

Where do I begin?

Test what, you ask? Anything you want. Really, the options are limitless. The best way to start the process is to begin brainstorming. Write down a list of “what if” scenarios. For example:

• What if my call to action was positioned in the top right-hand corner?
• What if my “submit” button was red as opposed to gray?
• What if my USP’s were listed in a different order?
• What if the benefits were redefined?
• What if I had less copy?

Next, turn those questions into a theory or statement. This becomes your hypothesis for testing.  For example:

• Heat map studies reveal that the top right-hand corner is the most “visible” landscape on a web page. By repositioning my call to action to this real estate, I can boost my click-through or conversion rates by XX.
• Although our product has various uses, most of our customers see it as a solution to X. By revising the product benefit hierarchy, we may appeal to a broader audience, which will boost click-through and/or conversion rates by XX.

Our product is a highly involved purchase decision. People go online to research their options and educate themselves, so we need to ensure all the information they would possibly need is here on the landing page. Reducing the amount of copy will negatively impact click-through and conversion rates by X and increase the bounce rate.

How often should I test and for how long?

The more variables you have in your testing matrix, the more combinations will need to be served up and the longer it will have to run to form any conclusive results. If you are new to multivariate testing, I would recommend starting small, perhaps testing two variables against your original.

In terms of how often, testing should become like routine maintenance – always needed and continuous. Don’t look at testing as a one-off initiative – strive to test at least five variables every 10-12 weeks. Your campaigns will benefit dramatically.

What results can I expect?

If we knew the answer to this question definitively, there wouldn’t really be a need for testing. Results are going to vary. The goal is to validate the hypothesis you outlined earlier. This validation can be a positive impact, negative, or even neutral and inconclusive. The ultimate goal of testing, however, is to identify the series of variables that have a positive impact on your conversion rate so you can get more out of your marketing investment.

How do you identify a clear winner? A general rule of thumb is to identify those variables and combinations that have a 90% or higher chance of beating your original. These are what we call “safe bets” that you can confidently make static on your site.

This is where you begin to see the light at the end of the tunnel. If you test five to seven variables in one phase of multivariate testing, and you are able to find three that have a 90% higher chance of beating the original, you could stand to boost your on-site conversion rates by more than 10% (depending on the impact each variables makes). It may not seem like a lot, but if you’re driving 1000 or more visits a day to your site, converting an extra 10% can make a big difference – and your bottom-line cost per lead or acquisition will become more efficient.

Understand that, regardless of the outcome of the test (i.e., positive or negative impact), the learnings will still hold value. It’s also common to see inconclusive results. Don’t let this dissuade you. Instead, use this information/these learnings to structure the next phase of your test.

Summary

It all starts with a hypothesis.

If I had a dollar for every time we’ve put a hypothesis in front of a client who responded with “but I already know the answer to that”, let’s just say I wouldn’t have to be writing articles like this. And while I won’t deny the hidden enjoyment that comes from devaluing hours worth of focus-group testing, our goal is to always produce unforeseen, positive results. The proof is in the numbers, and the numbers never lie.