Katrina in your test results?

Even if you live nowhere near the gulf coast, hurricane Katrina could be having a major impact on your test results. This is a classic example of a huge environmental noise factor, something that can shift the buying sentiments of an entire population. If you live near the affected area, or cater to a local business, this will be obvious.

But what about us who live hundreds or thousands of miles away?

The press coverage has been more than constant- its been downright transfixing. Anyone who knows someone living near New Orleans or who has been through a hurricane probably spent a little extra time watching the news today... and a little less time surfing the web or "taking care of business." Not to mention the pervasive online news coverage and blogs pulling eyeballs and dollars away from our sites.

To make matters worse, the threat of spiking gas prices has many people hitting the brakes on purchase decisions. We noted a significantly lower conversion rate today on at least a couple of our "key indicator" tests. It seemed more like a Saturday than a Monday.

We use the term noise to refer to any uncontrollable factor that can affect the outcome of our tests, whether it is a positive or negative change. Split testing helps to disperse the effect equally within the test battery, but it doesn't always eliminate an uncontrolled bias from entering the test results.

Take for example, a site that sells flood insurance. If this was you, today was probably a great day for sales, perhaps even record breaking. Would today be a good day to get repeatable test results? I think not. The winning creative today probably would be the one with the shortest and easiest order form. Today, many people wouldn't need much convincing. Normally, the best pulling message might be a completely different appeal altogether. Basing your test results on today's data, even if statistically significant, would lead you to erroneous assumptions.