1. Decide and Document your Goals

Before you ever start testing, you should have an idea of what you plan to get out of it.  You should not be testing just to join the “cool kids.”  Going into your testing program you should have a written goal.  It may be broad, but having that goal will change how you proceed through each of the steps of creating and launching your test.

2. Assemble your Team, Communicate what you are doing internally

Identify who will be involved with the test, and make sure each person understands their specific role.  If you will need custom creative assets developed, identify the stakeholders and get some time carved out in their schedule.  Let other groups in your organization know you will be performing testing on the site so they do not get surprised be an unexpected change to the site.

3. Research problem pages on your site with analytics

Use web analytics tools to spot trouble areas in your site. Things like high drop off or bounce rates, low conversion rates, and poorly performing landing pages should stick out like a sore thumb and could be ideal places to focus your testing campaign.  Also look for pages that get a significant amount of traffic, as these may be prime candidates for testing as well.

4. Choose the page on your site to test

The next thing to do is determine what page or pages you want to include in your test.  In many cases, this will be obvious- most people start with either specific landing pages, or (eek) their home page.  Home pages are notoriously difficult to test because of competing goals and noisy traffic but if you analyze specific traffic segments you can still get good learnings.  Regardless, use your analytics research to help choose where to focus your efforts.

5. Decide what variables to test

This is incredibly obvious, but of course you have to decide what things to test.  We have devoted a number of past blog posts to this topic, but it is worth repeating: focus on stuff that appears “above the fold” like headlines, images, page layouts, inclusion of navigation, special offers, promotions, scarcity elements, and risk reversal. Below the fold focus on order and buy buttons, forms, and more risk reversal (near the buttons.)

Image by Getty Images via Daylife

6. Create radical options

This has also been written up before, but if you are going to run a test, run a TEST.  Don’t be shy!  Minimal changes will result in minimal differences in performance- leading to drawn out test times and the possibility of a NULL test result (no clear winner no matter how much traffic you get.)  Don’t test 10 different shades of blue for that order button- test 10 different colors from across the spectrum!

7. Identify external factors and plan to block them

There are many external factors which can sway test results.  An easy to understand example is “day of week.”  People tend to behave differently on weekdays vs. weekends.  This can show up in a test with different results on weekends than on weekdays.  To block this data from affecting your overall results- make sure you run the test for at least a full week or two full weeks.  Alternatively you can analyze your results for just weekday visitors or just weekend visitors although not all testing software supports this.

8. Develop a Hypothesis

Just like you did in science class growing up, you should form a hypothesis of what you think will happen.  This statement summarizes what you are testing, why you are testing it, and what you suspect will happen.

9. Decide whether to run a multivariate, split or URL test

Multivariate tests have the advantage of speed and understanding: you’ll get more testing done in less time- particularly with a fractional factorial MVT.  You’ll also get the advantage of seeing how various changes work when combined together.  The downside is that management has less understanding and getting buy-in may be tougher.  Split tests let you focus on one or more variables on a page test alternative treatments… think headlines, buttons, hero shots, etc.  If you wish to test really big ideas, like completely different page layouts- try testing multiple different URLs!

10. Make sure your site is “modular”

The best way to run any type of experiment is to focus on something called “variable isolation.”  This means that each thing you are testing should be tested in an independent manner- thus making certain that any performance lift (or loss) you observe are directly tied to the changes you have made.  The easiest way to get to variable isolation is to design your web pages in a “modular” fashion, which makes it easy to target specific things- like the headline, images, or info box.

11. Consider upgrading your HTML

Modern XHTML is better for testing than table based layouts.  Coding your page with XHTML and CSS will give you cleaner code which renders much more quickly in modern browsers than older table based layouts.  It has the advantage of being easy to test with CSS style tweaks (as opposed to submitting entire blocks of html) and also will help your SEO.

12. Identify the success metric and proper value to judge the winner

Looking back at #1, you should be able to extrapolate from your goal to pinpoint the specific metrics you need to base the test on.  Although many people talk about “conversion rate” as the best metric to judge a result- this is often not the case.  If you are selling products or run an e-commerce site, metrics like Revenue Per Visitor (RPV) or Average Order Value (AOV) will provide a much clearer view of the test winner.  Trying to increase ad clicks means that you should be using outbound click tracking, and if your goal is to just increase site stickiness- then track engagement or bounce rate.

13. Make sure your Adgroups and other traffic sources are passing identifying parameters

If you want to be able to segment your test results by adgroup (yes you do) you’ll need to make certain your Adwords account is set up to pass an identifying parameter in the URL.  You probably already do this for your standard analytics- but if not, make sure you do it for testing purposes.  Often you will see visitors from different adgroups responding to your tests differently.

http://www.yoursite.com/landing.html?adgrp=blue_widgets

14. Implement a Testing Solution

Select a testing software solution to implement the test on your site.  Different solutions integrate differently (this could be an entire post.)  Some require you to inject javascript calls throughout your HTML, while others are far less intrusive.

Image by Runs With Scissors via Flickr

15. Test the Test: Make sure everything is working before you launch

Before you launch or auto schedule the launch of a test, make certain everything works as you expect, including the tracking.  There are plenty of examples out on the web of people who have launched tests only to find out later that there was a major problem with the tracking or a variable not displaying correctly.  Here again, different software providers offer different capabilities for checking thing.  Also, some providers update results in real time, making it easy to spot problems quickly.