Slides from the session appear below:

If one thing was made apparent by the WhichTestWon testing awards this year, its that most web marketers are toting around a shutgun, rather than a sniper rifle when it comes to testing. Very few of the entries provided meaningful learnings, beyond “one page is better than the other.”  If we had asked people WHY Recipe C beat the control, not many would have had a definitive answer.

So I implore you to stop testing lots of things at the same time (shotgun approach) and instead, pick and isolate variables carefully and deliberately (sniper rifle.)  When the test is done, you’ll be able to proudly say, Recipe C beat the control, and the reason WHY is that a medium button that is red and includes the word FREE gets more clicks.

Why does this matter?  Because you can take this learning and apply to your next test-

2. Test, Learn, Change, Test, Learn, Change, Test Learn… Iterate More!

Iteration is the secret to delivering the best possible product.  Picture what it would be like if car manufacturers did not iterate on their designs?  We would all be driving around in clunkers that were no more reliable than they were in the early 1900s.  Luckily, this is not the case!

The same is true with testing your sites, landing pages, and other online creative.  Rather than using the shotgun approach by testing repeated wildly differing designs- put on your white lab coat and treat your web optimization campaign like a real scientific experiment.  Run focused, controlled, carefully designed experiments- varying only one thing for an AB test or a handful of things in a multivariate test.

This approach may not be quite as exciting as the former, but in the end, you will learn more, and have a better likelihood of success.  Once you have completed one test, be immediately ready to run the next, iterating (or building) on what you have learned.

3. Add the word OFAT to Your Testing Vocabulary

Just because OFAT sounds horrible (old and fat?) it is neither bad, old nor fat.  OFAT stands for One Factor At a Time- in other words one variable at a time.  Google erroneously took the term ‘AB test’ and popularized it to mean testing wildly differing designs (in fact different web addresses.)  AB testing was originally synonymous with OFAT- meaning in a split test, you isolate one variable and change it to measure the effect of the change.

4. Take a Class or Read a Book

In my job, I get to observe the testing practices of a great number of marketers and agencies. I can say with a degree of certainty that many of them (including those “certified by Google”) would benefit by taking the time to study testing best practices.  Even though they are technical and not specifically geared towards web testing reading up on experimental design and design of experiments can help you create better tests with more reliable results.  A side benefit is that they will possibly require less traffic to get statistical validity as well.  There are a set of well understood principals from offline testing which can be ported over to the online world with great success!

6. Learn from example

Lately,  a number of inspiring sites have popped up with examples of real test results, expert commentary, and other resources.  One of my favorites is Anne Hollands “WhichTestWon” which hosts a weekly blog asking their namesake question.  Marketers submit two versions of a prior test, and visitors to the site are given the opportunity to pick which they thought won!  These sites can help you in two ways- 1.) You get a view into the mind of other marketers and how they chose to design experiments, and 2.) They can be a great source of ideas for testing on your own site!

7. Take a walk offline

Some businesses count as many as 70% or more of their conversions from offline sources (telephone orders.)  The simple fact is you cannot afford to ignore these when you are running a test.  Would you drive a car with the windshield 70% obscured??  Sadly, many many marketers do the equivalent testing online.  If you get a lot of orders by phone, you need to hook some phone conversion tracking capabilities to your test.  I am not sure how many testing vendors support phone tracking integration, but I know at least one that does (wink).

As for the phone tracking vendors, a search of Google will illuminate many- varying widely in cost.  Most will work for testing purposes.  The important thing is to make sure that each unique recipe tested is correctly assigned to a unique phone number.  Then when a call comes in, the phone system needs to be told to report a conversion for that page version.  It sounds confusing because it is.  Luckily, all of this complexity happens in the background.

8. Pay Attention to Segments to Maximize Testing Satisfaction

Ask yourself what would happen if Santa Clause delivered the same present to every child, regardless of age, nationality, or gender.  Would most kids be happy with their gifts?  I doubt it!  Instead, Santa segments his audience, making sure little boys get toy cars, action figures, and the like, and little girls get dolls, stuffed animals, jewelry, etc.

Don’t make a mistake by assuming all of your website visitors are going to respond to test options in the same way.  Many marketers are running tests with darkened glasses on here, blindly ignoring the nuances that commonly exist amongst different traffic segments.  Make a goal in 2010 to invest in a testing platform that can show you how visitors from Google performed compared to visitors from Facebook, and which version of the test won for each group.

The first annual awards were announced last week, and you have until November 20 to get your entry submitted!  There will be an awards webcast in the beginning of December (we’ll announce it here or you can follow whichtestwon.com to stay up to date.)

Winners will get links from us, whichtestwon.com and a number of other sites that choose to cover the awards.  Furthermore, you get a cool badge to place on your site indicating that you were a winner.  Plus it’s a great resume bullet to say your AB test won an Award!  This has the potential to earn you or your agency a lot of free publicity!

The categories  include things like Best: Homepage Test, Copy Test, PPC Landing Page Test, Email Opt-in Offer Test, Ecommerce Store Test, Blog Interaction or Conversion Test, Video Test, Advanced Testing Tactics.

The rules are pretty simple, but one important thing to note is that your test does not have to use Vertster.  In fact if you have a good test to enter that you did with Google Optimizer, Omniture, or any other software, feel free to enter it!  We’ll keep your results anonymized as well!

So hurry up and enter- the deadline is Nov 20.

Before you ever start testing, you should have an idea of what you plan to get out of it.  You should not be testing just to join the “cool kids.”  Going into your testing program you should have a written goal.  It may be broad, but having that goal will change how you proceed through each of the steps of creating and launching your test.

2. Assemble your Team, Communicate what you are doing internally

Identify who will be involved with the test, and make sure each person understands their specific role.  If you will need custom creative assets developed, identify the stakeholders and get some time carved out in their schedule.  Let other groups in your organization know you will be performing testing on the site so they do not get surprised be an unexpected change to the site.

3. Research problem pages on your site with analytics

Use web analytics tools to spot trouble areas in your site. Things like high drop off or bounce rates, low conversion rates, and poorly performing landing pages should stick out like a sore thumb and could be ideal places to focus your testing campaign.  Also look for pages that get a significant amount of traffic, as these may be prime candidates for testing as well.

4. Choose the page on your site to test

The next thing to do is determine what page or pages you want to include in your test.  In many cases, this will be obvious- most people start with either specific landing pages, or (eek) their home page.  Home pages are notoriously difficult to test because of competing goals and noisy traffic but if you analyze specific traffic segments you can still get good learnings.  Regardless, use your analytics research to help choose where to focus your efforts.

5. Decide what variables to test

This is incredibly obvious, but of course you have to decide what things to test.  We have devoted a number of past blog posts to this topic, but it is worth repeating: focus on stuff that appears “above the fold” like headlines, images, page layouts, inclusion of navigation, special offers, promotions, scarcity elements, and risk reversal. Below the fold focus on order and buy buttons, forms, and more risk reversal (near the buttons.)

Image by Getty Images via Daylife

6. Create radical options

This has also been written up before, but if you are going to run a test, run a TEST.  Don’t be shy!  Minimal changes will result in minimal differences in performance- leading to drawn out test times and the possibility of a NULL test result (no clear winner no matter how much traffic you get.)  Don’t test 10 different shades of blue for that order button- test 10 different colors from across the spectrum!

7. Identify external factors and plan to block them

There are many external factors which can sway test results.  An easy to understand example is “day of week.”  People tend to behave differently on weekdays vs. weekends.  This can show up in a test with different results on weekends than on weekdays.  To block this data from affecting your overall results- make sure you run the test for at least a full week or two full weeks.  Alternatively you can analyze your results for just weekday visitors or just weekend visitors although not all testing software supports this.

8. Develop a Hypothesis

Just like you did in science class growing up, you should form a hypothesis of what you think will happen.  This statement summarizes what you are testing, why you are testing it, and what you suspect will happen.

9. Decide whether to run a multivariate, split or URL test

Multivariate tests have the advantage of speed and understanding: you’ll get more testing done in less time- particularly with a fractional factorial MVT.  You’ll also get the advantage of seeing how various changes work when combined together.  The downside is that management has less understanding and getting buy-in may be tougher.  Split tests let you focus on one or more variables on a page test alternative treatments… think headlines, buttons, hero shots, etc.  If you wish to test really big ideas, like completely different page layouts- try testing multiple different URLs!

10. Make sure your site is “modular”

The best way to run any type of experiment is to focus on something called “variable isolation.”  This means that each thing you are testing should be tested in an independent manner- thus making certain that any performance lift (or loss) you observe are directly tied to the changes you have made.  The easiest way to get to variable isolation is to design your web pages in a “modular” fashion, which makes it easy to target specific things- like the headline, images, or info box.

11. Consider upgrading your HTML

Modern XHTML is better for testing than table based layouts.  Coding your page with XHTML and CSS will give you cleaner code which renders much more quickly in modern browsers than older table based layouts.  It has the advantage of being easy to test with CSS style tweaks (as opposed to submitting entire blocks of html) and also will help your SEO.

12. Identify the success metric and proper value to judge the winner

Looking back at #1, you should be able to extrapolate from your goal to pinpoint the specific metrics you need to base the test on.  Although many people talk about “conversion rate” as the best metric to judge a result- this is often not the case.  If you are selling products or run an e-commerce site, metrics like Revenue Per Visitor (RPV) or Average Order Value (AOV) will provide a much clearer view of the test winner.  Trying to increase ad clicks means that you should be using outbound click tracking, and if your goal is to just increase site stickiness- then track engagement or bounce rate.

13. Make sure your Adgroups and other traffic sources are passing identifying parameters

If you want to be able to segment your test results by adgroup (yes you do) you’ll need to make certain your Adwords account is set up to pass an identifying parameter in the URL.  You probably already do this for your standard analytics- but if not, make sure you do it for testing purposes.  Often you will see visitors from different adgroups responding to your tests differently.

http://www.yoursite.com/landing.html?adgrp=blue_widgets

14. Implement a Testing Solution

Select a testing software solution to implement the test on your site.  Different solutions integrate differently (this could be an entire post.)  Some require you to inject javascript calls throughout your HTML, while others are far less intrusive.

Image by Runs With Scissors via Flickr

15. Test the Test: Make sure everything is working before you launch

Before you launch or auto schedule the launch of a test, make certain everything works as you expect, including the tracking.  There are plenty of examples out on the web of people who have launched tests only to find out later that there was a major problem with the tracking or a variable not displaying correctly.  Here again, different software providers offer different capabilities for checking thing.  Also, some providers update results in real time, making it easy to spot problems quickly.

If your landing page takes more than a couple seconds to render on broadband, you are surely leaving money on the table. This is especially true if the headline and primary value proposition are slow to appear. The longer your pages take to download and render in a user’s browser, the more money you are losing. This is because people have a strong tendency to use their back buttons if a page takes too long to appear.

If you are not sure how long your landing pages take to appear, use the free website speed report to learn how fast (or slow) they are. This site will also give you some specific suggestions for speeding things up.

Most people agree that fast loading pages are a good idea, although they seldom do much about it. However, in case you missed it, a couple weeks ago, Google came out and promised to penalize slow loading landing pages connected to Adwords advertisements.

Google has added a new metric in the Adwords interface that indicates if they think your page is too slow. Slow loading pages will have a negative impact on your Quality Score and increase your minimum bids. I am curious what steps they take to determine if your page is slow, but one thing is for certain…. slow pages will ultimately cost you more money to advertise.

Will this mean an end to the 15 page long sales letters than dominate some niches? I don’t know, we will have to wait and see. These pages are frequently 250-300k and can take minutes to download completely on a slow connection. Another area of concern is landing pages that use javascript libraries like Prototype, Scriptaculous, or Jquery. Will Google’s new policy inadvertently stifle ajax and web 2.0 landing pages?

It is too early to know the answers to all the questions, but the bottom line is this. You will not only enjoy a higher conversion rate by putting your pages on a diet, you may save some money on your Adwords bill as well.

Warm Regards,

Scott Miller, CEO

| View | Upload your own

| View | Upload your own

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

When I look at a page for the first time, I examine it under the premise of a simple framework I call the three C’s of conversion. It goes like this:

Does your landing page:

1. Capture the attention of the visitor?
2. Communicate your primary value proposition to the visitor?
3. Close the sale, lead, or other offer you are making?

Each of these steps are essential to boosting conversion. On my snow-delayed flight out of Chicago, I thought I should add one more C to the equation: the Customer!

Obviously, with no customer, there is no sale!

The reason this is important is simple: upon arrival to your site, the customer has some expectation in mind, something he or she was looking for. In order to capture, communicate and close, your site must be as relevant as possible to this customer.

In the coming days I will be sharing more specific recommendations and insight on analyzing and improving your Cs of conversion.

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

I spoke alongside Jamie Roche of Offermatica, Tim Ash from Sitetuners, and Tom Leung from Google Web Optimizer. Plus Alan Dick of Clawfoot Tub retailer Vintage Tub.

My presentation included two case studies: One featuring online store Tinyride.com and tested the effect of using the Hackersafe logo. The other was a Taguchi test from Business and Legal Reports (BLR.com). Following the presentation, was a short time for Q&A, during which people had some great questions.

If you attended the session, or are just interested. you can download my slides here.

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

What is Multivariate Testing
Wondering about the buzz surrounding Multivariate testing? Check this article for a plain English explanation about why you need to start doing it.

Multivariate Optimization vs. A/B Testing
Whether to use A/B or Multivariate testing is a question most marketers face. This article helps you understand some of the important differences.

Taguchi who? I don’t speak Japanese!
This article describes the origins of the Taguchi Approach and summarizes a Taguchi testing framework.

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

* The ad group that used dynamic keyword insertion recommended a minimum bid of $5.00!
* The ad group that used static keyword insertion recommended a minimum bid of $0.40.

Read the post here.

Does this mean we’ll soon see keyword stuffing and other nefarious tactics on PPC landing pages?? I don’t know, and I kind of doubt it would help anyways, but who really knows how sophisticated this algorytm is.

If you are running Urchin Analytics, Google could actually measure your conversion rate, and include it as part of the score! This is kind of scary, and another reason we’ll never recommend using Urchin to anyone. But then again, if you are actively testing and optimizing your landing page content for conversions, you’ll have a leg up on most of the competition.

The answer is rather simple, and here is my favorite metaphor. Web Analytics is like looking in your rearview mirror for the turn or business you just missed. It’s all about what has already happened.

Multivariate testing (MVT) is like having a GPS for your website. A GPS tells you the optimal way to get to your destination, where to turn, and how long it will take. MVT similarly will tell you the best way to build your site to get the most sales the fastest.

The question you must ask yourself: Does your business run by looking in the rearview mirror? Are you reactive or proactive?

Here are Michaels top 4 reasons for why best practices suck:

1. They rarely work
2. It’s a followers strategy
3. Change comes from within
4. They don’t come with a manual

One of the fascinating things we learn through testing sites is how wrong the “best practices” are in many cases. Every site, sale, or offer is unique, and the only way to know what really works is to test it and find out!