Slides from the session appear below:

If one thing was made apparent by the WhichTestWon testing awards this year, its that most web marketers are toting around a shutgun, rather than a sniper rifle when it comes to testing. Very few of the entries provided meaningful learnings, beyond “one page is better than the other.”  If we had asked people WHY Recipe C beat the control, not many would have had a definitive answer.

So I implore you to stop testing lots of things at the same time (shotgun approach) and instead, pick and isolate variables carefully and deliberately (sniper rifle.)  When the test is done, you’ll be able to proudly say, Recipe C beat the control, and the reason WHY is that a medium button that is red and includes the word FREE gets more clicks.

Why does this matter?  Because you can take this learning and apply to your next test-

2. Test, Learn, Change, Test, Learn, Change, Test Learn… Iterate More!

Iteration is the secret to delivering the best possible product.  Picture what it would be like if car manufacturers did not iterate on their designs?  We would all be driving around in clunkers that were no more reliable than they were in the early 1900s.  Luckily, this is not the case!

The same is true with testing your sites, landing pages, and other online creative.  Rather than using the shotgun approach by testing repeated wildly differing designs- put on your white lab coat and treat your web optimization campaign like a real scientific experiment.  Run focused, controlled, carefully designed experiments- varying only one thing for an AB test or a handful of things in a multivariate test.

This approach may not be quite as exciting as the former, but in the end, you will learn more, and have a better likelihood of success.  Once you have completed one test, be immediately ready to run the next, iterating (or building) on what you have learned.

3. Add the word OFAT to Your Testing Vocabulary

Just because OFAT sounds horrible (old and fat?) it is neither bad, old nor fat.  OFAT stands for One Factor At a Time- in other words one variable at a time.  Google erroneously took the term ‘AB test’ and popularized it to mean testing wildly differing designs (in fact different web addresses.)  AB testing was originally synonymous with OFAT- meaning in a split test, you isolate one variable and change it to measure the effect of the change.

4. Take a Class or Read a Book

In my job, I get to observe the testing practices of a great number of marketers and agencies. I can say with a degree of certainty that many of them (including those “certified by Google”) would benefit by taking the time to study testing best practices.  Even though they are technical and not specifically geared towards web testing reading up on experimental design and design of experiments can help you create better tests with more reliable results.  A side benefit is that they will possibly require less traffic to get statistical validity as well.  There are a set of well understood principals from offline testing which can be ported over to the online world with great success!

6. Learn from example

Lately,  a number of inspiring sites have popped up with examples of real test results, expert commentary, and other resources.  One of my favorites is Anne Hollands “WhichTestWon” which hosts a weekly blog asking their namesake question.  Marketers submit two versions of a prior test, and visitors to the site are given the opportunity to pick which they thought won!  These sites can help you in two ways- 1.) You get a view into the mind of other marketers and how they chose to design experiments, and 2.) They can be a great source of ideas for testing on your own site!

7. Take a walk offline

Some businesses count as many as 70% or more of their conversions from offline sources (telephone orders.)  The simple fact is you cannot afford to ignore these when you are running a test.  Would you drive a car with the windshield 70% obscured??  Sadly, many many marketers do the equivalent testing online.  If you get a lot of orders by phone, you need to hook some phone conversion tracking capabilities to your test.  I am not sure how many testing vendors support phone tracking integration, but I know at least one that does (wink).

As for the phone tracking vendors, a search of Google will illuminate many- varying widely in cost.  Most will work for testing purposes.  The important thing is to make sure that each unique recipe tested is correctly assigned to a unique phone number.  Then when a call comes in, the phone system needs to be told to report a conversion for that page version.  It sounds confusing because it is.  Luckily, all of this complexity happens in the background.

8. Pay Attention to Segments to Maximize Testing Satisfaction

Ask yourself what would happen if Santa Clause delivered the same present to every child, regardless of age, nationality, or gender.  Would most kids be happy with their gifts?  I doubt it!  Instead, Santa segments his audience, making sure little boys get toy cars, action figures, and the like, and little girls get dolls, stuffed animals, jewelry, etc.

Don’t make a mistake by assuming all of your website visitors are going to respond to test options in the same way.  Many marketers are running tests with darkened glasses on here, blindly ignoring the nuances that commonly exist amongst different traffic segments.  Make a goal in 2010 to invest in a testing platform that can show you how visitors from Google performed compared to visitors from Facebook, and which version of the test won for each group.

The first annual awards were announced last week, and you have until November 20 to get your entry submitted!  There will be an awards webcast in the beginning of December (we’ll announce it here or you can follow whichtestwon.com to stay up to date.)

Winners will get links from us, whichtestwon.com and a number of other sites that choose to cover the awards.  Furthermore, you get a cool badge to place on your site indicating that you were a winner.  Plus it’s a great resume bullet to say your AB test won an Award!  This has the potential to earn you or your agency a lot of free publicity!

The categories  include things like Best: Homepage Test, Copy Test, PPC Landing Page Test, Email Opt-in Offer Test, Ecommerce Store Test, Blog Interaction or Conversion Test, Video Test, Advanced Testing Tactics.

The rules are pretty simple, but one important thing to note is that your test does not have to use Vertster.  In fact if you have a good test to enter that you did with Google Optimizer, Omniture, or any other software, feel free to enter it!  We’ll keep your results anonymized as well!

So hurry up and enter- the deadline is Nov 20.

1. The search box itself:

The primary goal of optimizing the search box itself is to make certain visitors on the site can find it and understand what it does. Typical tests would revolve around the placement of the search box, size of the text field, color, and the appearance and text on the button.

2. The search results page:

Once people complete a search, they arrive to a results page which displays matching products or content.  The goal of the results page is for people be able to quickly see the results and then link to the content found.  Certainly, the effectiveness of this page will be highly dependent on the relevance of the results, but on-page factors (which can be optimized) play an important role as well.

We have seen simple changes and optimization increase search result page engagement by over 40%, which is a very big lift for something most people would think has little to do with testing.

The results page offers up many more areas of testing and optimization.  Here are a few to consider:

• How many results do you display?
• Do you use a regular site template or a scaled down page design which makes it easier for the user to focus on results?
• What visual cues do you provide within the results (such as highlighted matching text.)
• Do you allow people to purchase or add to cart directly from the search results?
• Do you show prices in the results?
• Do you show a “related searches” section to help the user refine their search (this has potential to distract them from the main results though.)

When you test these, you should make sure to track both result page engagement (did people click on the search results) as well as revenue per visit.  Subtle changes in the search system can make a big difference in revenue for many e-commerce sites, so never rely on engagment metrics only!

Before you ever start testing, you should have an idea of what you plan to get out of it.  You should not be testing just to join the “cool kids.”  Going into your testing program you should have a written goal.  It may be broad, but having that goal will change how you proceed through each of the steps of creating and launching your test.

2. Assemble your Team, Communicate what you are doing internally

Identify who will be involved with the test, and make sure each person understands their specific role.  If you will need custom creative assets developed, identify the stakeholders and get some time carved out in their schedule.  Let other groups in your organization know you will be performing testing on the site so they do not get surprised be an unexpected change to the site.

3. Research problem pages on your site with analytics

Use web analytics tools to spot trouble areas in your site. Things like high drop off or bounce rates, low conversion rates, and poorly performing landing pages should stick out like a sore thumb and could be ideal places to focus your testing campaign.  Also look for pages that get a significant amount of traffic, as these may be prime candidates for testing as well.

4. Choose the page on your site to test

The next thing to do is determine what page or pages you want to include in your test.  In many cases, this will be obvious- most people start with either specific landing pages, or (eek) their home page.  Home pages are notoriously difficult to test because of competing goals and noisy traffic but if you analyze specific traffic segments you can still get good learnings.  Regardless, use your analytics research to help choose where to focus your efforts.

5. Decide what variables to test

This is incredibly obvious, but of course you have to decide what things to test.  We have devoted a number of past blog posts to this topic, but it is worth repeating: focus on stuff that appears “above the fold” like headlines, images, page layouts, inclusion of navigation, special offers, promotions, scarcity elements, and risk reversal. Below the fold focus on order and buy buttons, forms, and more risk reversal (near the buttons.)

Image by Getty Images via Daylife

6. Create radical options

This has also been written up before, but if you are going to run a test, run a TEST.  Don’t be shy!  Minimal changes will result in minimal differences in performance- leading to drawn out test times and the possibility of a NULL test result (no clear winner no matter how much traffic you get.)  Don’t test 10 different shades of blue for that order button- test 10 different colors from across the spectrum!

7. Identify external factors and plan to block them

There are many external factors which can sway test results.  An easy to understand example is “day of week.”  People tend to behave differently on weekdays vs. weekends.  This can show up in a test with different results on weekends than on weekdays.  To block this data from affecting your overall results- make sure you run the test for at least a full week or two full weeks.  Alternatively you can analyze your results for just weekday visitors or just weekend visitors although not all testing software supports this.

8. Develop a Hypothesis

Just like you did in science class growing up, you should form a hypothesis of what you think will happen.  This statement summarizes what you are testing, why you are testing it, and what you suspect will happen.

9. Decide whether to run a multivariate, split or URL test

Multivariate tests have the advantage of speed and understanding: you’ll get more testing done in less time- particularly with a fractional factorial MVT.  You’ll also get the advantage of seeing how various changes work when combined together.  The downside is that management has less understanding and getting buy-in may be tougher.  Split tests let you focus on one or more variables on a page test alternative treatments… think headlines, buttons, hero shots, etc.  If you wish to test really big ideas, like completely different page layouts- try testing multiple different URLs!

10. Make sure your site is “modular”

The best way to run any type of experiment is to focus on something called “variable isolation.”  This means that each thing you are testing should be tested in an independent manner- thus making certain that any performance lift (or loss) you observe are directly tied to the changes you have made.  The easiest way to get to variable isolation is to design your web pages in a “modular” fashion, which makes it easy to target specific things- like the headline, images, or info box.

11. Consider upgrading your HTML

Modern XHTML is better for testing than table based layouts.  Coding your page with XHTML and CSS will give you cleaner code which renders much more quickly in modern browsers than older table based layouts.  It has the advantage of being easy to test with CSS style tweaks (as opposed to submitting entire blocks of html) and also will help your SEO.

12. Identify the success metric and proper value to judge the winner

Looking back at #1, you should be able to extrapolate from your goal to pinpoint the specific metrics you need to base the test on.  Although many people talk about “conversion rate” as the best metric to judge a result- this is often not the case.  If you are selling products or run an e-commerce site, metrics like Revenue Per Visitor (RPV) or Average Order Value (AOV) will provide a much clearer view of the test winner.  Trying to increase ad clicks means that you should be using outbound click tracking, and if your goal is to just increase site stickiness- then track engagement or bounce rate.

13. Make sure your Adgroups and other traffic sources are passing identifying parameters

If you want to be able to segment your test results by adgroup (yes you do) you’ll need to make certain your Adwords account is set up to pass an identifying parameter in the URL.  You probably already do this for your standard analytics- but if not, make sure you do it for testing purposes.  Often you will see visitors from different adgroups responding to your tests differently.

http://www.yoursite.com/landing.html?adgrp=blue_widgets

14. Implement a Testing Solution

Select a testing software solution to implement the test on your site.  Different solutions integrate differently (this could be an entire post.)  Some require you to inject javascript calls throughout your HTML, while others are far less intrusive.

Image by Runs With Scissors via Flickr

15. Test the Test: Make sure everything is working before you launch

Before you launch or auto schedule the launch of a test, make certain everything works as you expect, including the tracking.  There are plenty of examples out on the web of people who have launched tests only to find out later that there was a major problem with the tracking or a variable not displaying correctly.  Here again, different software providers offer different capabilities for checking thing.  Also, some providers update results in real time, making it easy to spot problems quickly.

1.  Your conversion action happens on another  site, which you have no control over: This is a surprisingly common occurrence in larger companies as well as for affiliate marketers.  The best thing to track in this case is any outbound clicks to the remote site which represent a purchase or lead straight to the checkout process.

2. Your IT department refuses to put conversion tracking code on the thank you page: Just as above, the trick here is to track the final event, whether it is a click on the “checkout” button, or a click on the last step of the checkout process.  The later in the process, the better.  You can also grab revenue for the order at this point, and if you want to be even more accurate, reduce the purchase amount to match the “throughput” of the un-measured steps.

In other words- if you can only track up to the “checkout” point, and you know that 50% of people clicking “checkout” will actually complete the process, reduce the revenue figure for each transaction by 50% to better approximate average order value and revenue per visitor.

3. You are optimizing yield from banner ads or some other outbound link (such as affiliates): In this case, mark clicks on the outbound links.  If you have information about the value of each click, you may pass this to the reporting tool along with the click.

4. You are pushing leads to a remote aggregator: In this case, you may be hosting a lead form on your site, which posts to a remote URL of a lead aggregator or marketplace.  There are two possibilities here- either attach the conversion event to a click on the “submit” button, or bind to the form submission.  Either way will truthfully record the conversion occurrence without requiring a code snippet on the thank you page.

The solution to these scenarios

The only way for this to work is if your conversion tracking system provides for click event tracking.  Another way to do it is by adding a “taking you to your destination” page which can then display the normal conversion pixel, but this is not very desirable.

Apparently the new Yahoo home page was the result of testing 141 different versions!  The goal of the testing was to find out which design resulted in the greatest site stickiness.  This makes sense, considering that Yahoo primarily makes money when people view and click on search results and ads.

I spoke alongside Jamie Roche of Offermatica, Tim Ash from Sitetuners, and Tom Leung from Google Web Optimizer. Plus Alan Dick of Clawfoot Tub retailer Vintage Tub.

My presentation included two case studies: One featuring online store Tinyride.com and tested the effect of using the Hackersafe logo. The other was a Taguchi test from Business and Legal Reports (BLR.com). Following the presentation, was a short time for Q&A, during which people had some great questions.

If you attended the session, or are just interested. you can download my slides here.

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

One of the challenges of designing multivariate tests is getting everyone in your organization on the same page. Thats where Fleck comes in. This browser extension lets you add “sticky notes” to annotate any web page.

We just started use it to mark up pages and indicate where variables are and even provide guidance on proposed variable options. This vision can be shared around with multiple people in multiple organizations before the test assets are actually built and the optimization is running.

Here is an example of a test design for my ebook site TheConversionLab.com:

http://extension.fleck.com/?sh=788d04e7f5d3dad9de1c4bf0b9315f16f3fad065

Note: Fleck is currently in beta, and appears to choke on the CSS layout of this page slightly. It goes to figure that this is the only site I have ever seen Fleck have an issue with.

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

What is Multivariate Testing
Wondering about the buzz surrounding Multivariate testing? Check this article for a plain English explanation about why you need to start doing it.

Multivariate Optimization vs. A/B Testing
Whether to use A/B or Multivariate testing is a question most marketers face. This article helps you understand some of the important differences.

Taguchi who? I don’t speak Japanese!
This article describes the origins of the Taguchi Approach and summarizes a Taguchi testing framework.

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

There is no doubt that a well optimized offer page, e-commerce system, etc. can be an incredible competitive advantage. But let’s rewind for a moment and consider what a perfectly optimized offer is…

As I said at SES three weeks ago, optimized offers combine the perfect benefit, perfect amount of risk reversal, perfect amount of scarcity, and perfect unique value proposition for the state of mind the customer is in at the moment they see they offer.

Now unless your offer is identical to your competitors, and your customer’s are the same, your “optimized” offer is going to be different than theirs. The only time offers truly converge is in a commodity marketplace, where nearly every competitor is identical and everyone competes on price.

Now I am not saying to go posting your secret optimized recipes for the world to see, but its food for thought. Do you really need to be paranoid? I think not.

Warm Regards,

Scott Miller, CEO

Author of “The ConversionLab.com,” The worlds first “How to” guide to testing and optimization. Visit http://www.conversionlab.com/ for a special offer!

It wasn’t until the arrival of the automobile that engines found their place in the world. The engine went behind the scenes and a new tool was born- the ability to get from one place to the next became easy.

We think of multivariate testing as the engine. It is what propels the vehicle- your web site. In the perfect world, this engine would “propel” a web page for each individual visitor. Based on their characteristics, the engine would automatically and seamlessly create a page for exactly what the user was looking for. While many people continue to “test” and “optimize” their sites using things like Taguchi and multivariate testing, wouldn’t it be better if your site were like a car? We think so.

By putting the test algorythm behind the scenes, your site could do it what it was meant to do better. That is get visitors from point A to point B, or in other words, from a lead to a sale.

• The difference between split testing and multivariate testing
• What things to test first
• How Vertster got started
• and more…

Listen to the interview now!

The fact is that AB testing is not disappearing any time soon. Taguchi can show you what things on the page are the most important. Once you know this, using OFAT on them is a smart strategy for “fine tuning” your key elements.

Back to factors… what can you or should you test? Start with the stuff that appears above the fold. Recent research shows that web visitors make a nearly instantaneous decision about your page. Improving your “engagement rate” can have a major impact on overall response. If your page is not engaging visitors, how can it possibly close with a conversion.

Things to consider: Your Logo, Tagline, Headline, Font, Face, Color, Size, Position, Hero Shot, Image Size, Caption, Subheadline, Greeting, First Sentence of Text, Top Bullet Point, First Paragraph, Inclusion of Advertising, Top Navigation Choice, Hackersafe, BBB, search boxes, just to name a few.

But what about us who live hundreds or thousands of miles away?

The press coverage has been more than constant- its been downright transfixing. Anyone who knows someone living near New Orleans or who has been through a hurricane probably spent a little extra time watching the news today… and a little less time surfing the web or “taking care of business.” Not to mention the pervasive online news coverage and blogs pulling eyeballs and dollars away from our sites.

To make matters worse, the threat of spiking gas prices has many people hitting the brakes on purchase decisions. We noted a significantly lower conversion rate today on at least a couple of our “key indicator” tests. It seemed more like a Saturday than a Monday.

We use the term noise to refer to any uncontrollable factor that can affect the outcome of our tests, whether it is a positive or negative change. Split testing helps to disperse the effect equally within the test battery, but it doesn’t always eliminate an uncontrolled bias from entering the test results.

Take for example, a site that sells flood insurance. If this was you, today was probably a great day for sales, perhaps even record breaking. Would today be a good day to get repeatable test results? I think not. The winning creative today probably would be the one with the shortest and easiest order form. Today, many people wouldn’t need much convincing. Normally, the best pulling message might be a completely different appeal altogether. Basing your test results on today’s data, even if statistically significant, would lead you to erroneous assumptions.

For this test, we are using an optimization type which we only make available for Full Service optimizations. In this case, it provides for up to 6 headlines, 3 images, 3 headers, 3 form placements, 3 greetings, 3 body copies, and 3 urgency appeals. This means there are over 4000+ combinations we will be able to look at!

One of the things I try to emphasize is that no matter what optimization system you are using, if you put garbage in, you are going to get garbage out. This means that a truly great creative writer can probably do more with A/B testing than an ordinary person could do with any of these advanced methodologies. So much of the ultimate success of an optimization campaign is controlled by the quality of the creative inputs.

Factor selections and level selections can be broadly divided into two categories- “creative” and “mechanical.” Mechanical variables are things like font size, color, background color, image size, etc. Creative variables are those like headline copy, ugency type, or call to action. In this case, improved copy will be a key to the optimization. Looking over the factor list, most of the factor selections are of the “creative” type, rather than the “mechanics” type. There is one notable exception- The form position.

Here is our test design, flushed out with the level selections:

Headline

1. Why watching this DVD could be worth an extra $10,000 a year
2. Before your next performance review, make sure you’ve watched this
3. How watching this DVD could keep you off the pink slip list… for good.
4. How to get a raise on your own terms
5. Starting -date- get paid what you’re worth
6. At last a surefire tactic to get a raise on -date-

Image

1. DVD only
2. Lifestyle- Woman
3. Lifestyle- Couple

Logo Header

1. UAC Logo only
2. PB Logo only
3. Both Logos

Form Position

1. Right Side
2. Left Side
3. Bottom

Lead In

1. Here is what…
2. Dear Bookkeeping Professional…
3. None

Body Copy

1. CONTROL
2. Show you first
3. What you will learn first

Urgency Appeal

1. Limited Time Offer
2. Free for a limited time
3. None

Over the coming days we will share some of our thoughts and strategy as we optimize this landing page using the Taguchi Method, and post live results to our blog. Our goal is to increase the lead conversion rate of the landing page at http://www.bookkeepingbiz.com/knowaccounting/ Currently it performs at a relatively anemic 15.5% on traffic from Google Adwords. (This is the conversion rate to date since 6/1/2005).

Using the Taguchi Method, we have identified 7 variables or “Factors” on this page design which we will test:

1. Headline
2. Image
3. Header
4. Form Position
5. Lead In/Greeting
6. Body Copy
7. Urgency Appeal

Check back to this blog regularly, as we reveal some of the secrets of increasing conversion rates, and what we learn on this optimization. In the next post, we’ll find out about level selection for the test!

Here are Michaels top 4 reasons for why best practices suck:

1. They rarely work
2. It’s a followers strategy
3. Change comes from within
4. They don’t come with a manual

One of the fascinating things we learn through testing sites is how wrong the “best practices” are in many cases. Every site, sale, or offer is unique, and the only way to know what really works is to test it and find out!