1. Put the shotgun down, website testing is not turkey hunt!
If one thing was made apparent by the WhichTestWon testing awards this year, its that most web marketers are toting around a shutgun, rather than a sniper rifle when it comes to testing. Very few of the entries provided meaningful learnings, beyond “one page is better than the other.” If we had asked people WHY Recipe C beat the control, not many would have had a definitive answer.
So I implore you to stop testing lots of things at the same time (shotgun approach) and instead, pick and isolate variables carefully and deliberately (sniper rifle.) When the test is done, you’ll be able to proudly say, Recipe C beat the control, and the reason WHY is that a medium button that is red and includes the word FREE gets more clicks.
Why does this matter? Because you can take this learning and apply to your next test-
2. Test, Learn, Change, Test, Learn, Change, Test Learn… Iterate More!
Iteration is the secret to delivering the best possible product. Picture what it would be like if car manufacturers did not iterate on their designs? We would all be driving around in clunkers that were no more reliable than they were in the early 1900s. Luckily, this is not the case!
The same is true with testing your sites, landing pages, and other online creative. Rather than using the shotgun approach by testing repeated wildly differing designs- put on your white lab coat and treat your web optimization campaign like a real scientific experiment. Run focused, controlled, carefully designed experiments- varying only one thing for an AB test or a handful of things in a multivariate test.
This approach may not be quite as exciting as the former, but in the end, you will learn more, and have a better likelihood of success. Once you have completed one test, be immediately ready to run the next, iterating (or building) on what you have learned.
3. Add the word OFAT to Your Testing Vocabulary
Just because OFAT sounds horrible (old and fat?) it is neither bad, old nor fat. OFAT stands for One Factor At a Time- in other words one variable at a time. Google erroneously took the term ‘AB test’ and popularized it to mean testing wildly differing designs (in fact different web addresses.) AB testing was originally synonymous with OFAT- meaning in a split test, you isolate one variable and change it to measure the effect of the change.
4. Take a Class or Read a Book
In my job, I get to observe the testing practices of a great number of marketers and agencies. I can say with a degree of certainty that many of them (including those “certified by Google”) would benefit by taking the time to study testing best practices. Even though they are technical and not specifically geared towards web testing reading up on experimental design and design of experiments can help you create better tests with more reliable results. A side benefit is that they will possibly require less traffic to get statistical validity as well. There are a set of well understood principals from offline testing which can be ported over to the online world with great success!
6. Learn from example
Lately, a number of inspiring sites have popped up with examples of real test results, expert commentary, and other resources. One of my favorites is Anne Hollands “WhichTestWon” which hosts a weekly blog asking their namesake question. Marketers submit two versions of a prior test, and visitors to the site are given the opportunity to pick which they thought won! These sites can help you in two ways- 1.) You get a view into the mind of other marketers and how they chose to design experiments, and 2.) They can be a great source of ideas for testing on your own site!
7. Take a walk offline
Some businesses count as many as 70% or more of their conversions from offline sources (telephone orders.) The simple fact is you cannot afford to ignore these when you are running a test. Would you drive a car with the windshield 70% obscured?? Sadly, many many marketers do the equivalent testing online. If you get a lot of orders by phone, you need to hook some phone conversion tracking capabilities to your test. I am not sure how many testing vendors support phone tracking integration, but I know at least one that does (wink).
As for the phone tracking vendors, a search of Google will illuminate many- varying widely in cost. Most will work for testing purposes. The important thing is to make sure that each unique recipe tested is correctly assigned to a unique phone number. Then when a call comes in, the phone system needs to be told to report a conversion for that page version. It sounds confusing because it is. Luckily, all of this complexity happens in the background.
8. Pay Attention to Segments to Maximize Testing Satisfaction
Ask yourself what would happen if Santa Clause delivered the same present to every child, regardless of age, nationality, or gender. Would most kids be happy with their gifts? I doubt it! Instead, Santa segments his audience, making sure little boys get toy cars, action figures, and the like, and little girls get dolls, stuffed animals, jewelry, etc.
Don’t make a mistake by assuming all of your website visitors are going to respond to test options in the same way. Many marketers are running tests with darkened glasses on here, blindly ignoring the nuances that commonly exist amongst different traffic segments. Make a goal in 2010 to invest in a testing platform that can show you how visitors from Google performed compared to visitors from Facebook, and which version of the test won for each group.
