Recently I was thinking about the state of testing and experimentation on the web, and realized through talking with customers- there are still a lot of myths out there about testing. I hope to clear up a few of them here:

Myth #1: Testing will negatively effect my SEO
This concern has been around forever, and continues today. Even people using the free Google website optimizer tool have shown concern about losing their rankings. That being said, Google has come out and said that since testing is all about increasing conversions, not tricking search engines, it is not a problem.

In our experience, the biggest risk comes when you are changing URLs to run a test. This is usually done with a fast redirect, meta refresh, or javascript redirect- all of which can lead to confusion for the search spiders. Luckily, most of the testing software out there takes this into consideration and sidesteps the problem by using javascript to render different page versions dynamically.

Myth #2. Conversion is the correct metric to judge success
When you run a test, its essential to consider what metric you will use to determine the success or failure of a test. Traditionally, we group split testing, MVT and other testing types under the category of “Conversion Improvement.” However, conversion is often the wrong metric to judge the success or failure of a test campaign.

Here is why: Lets say you are testing your online store, and in that store, you have many products at differing prices. So you embark on a mission to improve conversion, and the savvy marketer inside you says- “hey I bet if we make our products free, conversion will go up.”

By running a special promotion, you are able to drive conversion rates to all time highs- close to 100%! But guess what happens to your business? You go out of business because of the sudden lack of revenue. In this case, it would have been better to judge your success based on Revenue per Visitor- which would have shown that even with a 100% conversion rate, your revenue per visit was actually dwindling to ZERO dollars.

This is a silly contrived example, but- it illustrates the point. If you focus solely on conversion, you run the risk of increasing conversion at the cost of revenue.

3. Testing is a “one and done” activity
You should embrace testing and optimization as an ongoing activity, rather than something you do once or twice a year and consider it good. Over the course of more than 5000 web experiments, one thing we have learned is that testing a LOT pays off. Frequently, you will learn a few things in each test, and its the cumulative results that ultimately deliver the biggest results.

Don’t be afraid to fail! Often times, the first test will see only marginal or minimal gains, but as long as you learn something, it puts you one step closer to the higher conversion rate that you know you deserve. Remember- the only test that is a loser is one that you don’t learn anything from.In 2009, commit to an ongoing, iterative test program where you can quickly test over and over again, taking what you have learned and plugging it into the next test. Strive to minimize downtime between tests so as to increase your odds of success over the long haul.

4. Once I optimize my site, I am done.
This is just plain wrong, for a number of reasons- things in the world are always changing! Right now we are in the throes of the worst economy in years. Do you think the same sales pitch that worked 18 months ago is still going to motivate people today? I doubt it!

We recommend people re-validate their results every 6-8 months.  This ensures that you are always maximizing your conversion- and will help you understand the effect subtle changes in the marketplace have on your customers.

5. A/B testing is going to disappear
Since the onset of Multivariate testing, pundits have continued to argue that A/B testing is going the way of the dinosaurs. If you are of this mentality, I have news for you. Dinosaurs are destined to stay in the museums, and A/B Split testing is destined to stay in every marketers toolbox.

Here is why: A/B testing is simple, easy to understand, and works great for sites with lower traffic levels. It is the tried and true way to run tests, and the one approach that all marketers and corporate higher ups understand. That being said, multivariate and Taguchi testing is more powerful and can yield more interesting results- at the expense of increased set up time, and decreased acceptance and understanding by clients and management.

Do you have any more myths??  If you do, please post them in the comments!