I went to unsubscribe from a opt-in list and spotted a great test idea. This particular advertiser, was clever enough to add an option for staying on the list.
I would love to see a test of this and find out how much this second option actually reduced list unsubscriptions.
If you have tried unsuccessfully to access this blog in the last couple days, we have an unknown problem, which is currently being addressed. Please check back soon.
After going through this horrible economic downturn (which we’ll be feeling the effects of for some time) companies have learned different ways to survive.
The companies making it through the economic crisis are going to be more efficient and more streamlined as they have learned to do more with much less, to squeeze more out of areas that they did not focus on before. Corporate resources are called upon to deliver more and work harder.
Budgets have been cut back and marketing spend has been reduced, so savvy companies are learning that their websites need to deliver the goods. It used to be that if you wanted more leads or sales, you just turn up the dial on the adspend and get more.
Now there are ways to increase leads and sales from your website even amidst the budget crunch and ad spend cutbacks.
1. A/B and Multivariate Testing – If you have not done this, then you are not putting your best foot forward or at least you are not putting your best website forward. Why settle for a poor performing website when you can tweak it for maximum performance.
2. Segment optimization- Not every visitor to your site is the same. There are questions you should be asking yourself about your visitors. Where do they come from? What keyword brought them in? What time of day do they visit the site? Are they coming from your advertising? Once you know this information, you can take action that will increase the performance of your website. Find the right combinations that work for everyone coming from Google for example. Closely related to Personalization below.
3.Personalization- Almost all websites are showing one message for every visitor. But what if you could show the right message for every visitor? Blue widgets for the people who found the site through a blue widget keyword, red widgets for the red, North American widgets for those who came in from Seattle and glow in the dark widgets for those who came to the site in the middle of the night. Well you can and you should, this increases conversion rates above and beyond what testing can do.
Living through the furnace of economic trials will make stronger companies who are smarter with what they have. Don’t be surprised if your competition has a website that doesn’t seem to rank well or they don’t spend much to promote it, but it seems to be kicking your butt. They’ve probably figured out how to make the site work for them.
1. Put the shotgun down, website testing is not turkey hunt!
If one thing was made apparent by the WhichTestWon testing awards this year, its that most web marketers are toting around a shutgun, rather than a sniper rifle when it comes to testing. Very few of the entries provided meaningful learnings, beyond “one page is better than the other.” If we had asked people WHY Recipe C beat the control, not many would have had a definitive answer.
So I implore you to stop testing lots of things at the same time (shotgun approach) and instead, pick and isolate variables carefully and deliberately (sniper rifle.) When the test is done, you’ll be able to proudly say, Recipe C beat the control, and the reason WHY is that a medium button that is red and includes the word FREE gets more clicks.
Why does this matter? Because you can take this learning and apply to your next test-
2. Test, Learn, Change, Test, Learn, Change, Test Learn… Iterate More!
Iteration is the secret to delivering the best possible product. Picture what it would be like if car manufacturers did not iterate on their designs? We would all be driving around in clunkers that were no more reliable than they were in the early 1900s. Luckily, this is not the case!
The same is true with testing your sites, landing pages, and other online creative. Rather than using the shotgun approach by testing repeated wildly differing designs- put on your white lab coat and treat your web optimization campaign like a real scientific experiment. Run focused, controlled, carefully designed experiments- varying only one thing for an AB test or a handful of things in a multivariate test.
This approach may not be quite as exciting as the former, but in the end, you will learn more, and have a better likelihood of success. Once you have completed one test, be immediately ready to run the next, iterating (or building) on what you have learned.
3. Add the word OFAT to Your Testing Vocabulary
Just because OFAT sounds horrible (old and fat?) it is neither bad, old nor fat. OFAT stands for One Factor At a Time- in other words one variable at a time. Google erroneously took the term ‘AB test’ and popularized it to mean testing wildly differing designs (in fact different web addresses.) AB testing was originally synonymous with OFAT- meaning in a split test, you isolate one variable and change it to measure the effect of the change.
4. Take a Class or Read a Book
In my job, I get to observe the testing practices of a great number of marketers and agencies. I can say with a degree of certainty that many of them (including those “certified by Google”) would benefit by taking the time to study testing best practices. Even though they are technical and not specifically geared towards web testing reading up on experimental design and design of experiments can help you create better tests with more reliable results. A side benefit is that they will possibly require less traffic to get statistical validity as well. There are a set of well understood principals from offline testing which can be ported over to the online world with great success!
6. Learn from example
Lately, a number of inspiring sites have popped up with examples of real test results, expert commentary, and other resources. One of my favorites is Anne Hollands “WhichTestWon” which hosts a weekly blog asking their namesake question. Marketers submit two versions of a prior test, and visitors to the site are given the opportunity to pick which they thought won! These sites can help you in two ways- 1.) You get a view into the mind of other marketers and how they chose to design experiments, and 2.) They can be a great source of ideas for testing on your own site!
7. Take a walk offline
Some businesses count as many as 70% or more of their conversions from offline sources (telephone orders.) The simple fact is you cannot afford to ignore these when you are running a test. Would you drive a car with the windshield 70% obscured?? Sadly, many many marketers do the equivalent testing online. If you get a lot of orders by phone, you need to hook some phone conversion tracking capabilities to your test. I am not sure how many testing vendors support phone tracking integration, but I know at least one that does (wink).
As for the phone tracking vendors, a search of Google will illuminate many- varying widely in cost. Most will work for testing purposes. The important thing is to make sure that each unique recipe tested is correctly assigned to a unique phone number. Then when a call comes in, the phone system needs to be told to report a conversion for that page version. It sounds confusing because it is. Luckily, all of this complexity happens in the background.
8. Pay Attention to Segments to Maximize Testing Satisfaction
Ask yourself what would happen if Santa Clause delivered the same present to every child, regardless of age, nationality, or gender. Would most kids be happy with their gifts? I doubt it! Instead, Santa segments his audience, making sure little boys get toy cars, action figures, and the like, and little girls get dolls, stuffed animals, jewelry, etc.
Don’t make a mistake by assuming all of your website visitors are going to respond to test options in the same way. Many marketers are running tests with darkened glasses on here, blindly ignoring the nuances that commonly exist amongst different traffic segments. Make a goal in 2010 to invest in a testing platform that can show you how visitors from Google performed compared to visitors from Facebook, and which version of the test won for each group.
I’m excited to announce our involvement in the 2009 WhichTestWon Awards! Next week, I invite you to join me on Tuesday when I reveal the Best Tests of 2009 at WhichTestWon’s Awards. These are not just Vertster tests, but come from many entries and many different testing systems!
I’m co-presenting the live webinar awards with Anne Holland. Our fellow judges are Avinash Kaushik and Bob Bly. Plan to attend next week, on Tues Dec 8th, 2:30pm ET/11:30am PT
You’ll get to see samples — winning and losing test panels — plus data from the best tests of 2009 including:
If you love testing, this should be one of the most inspirational events of the year. The webinar is Next week, Tuesday Dec 8 at 2:30 PM EST.
On-site search is a big deal, especially for e-commerce sites. Many sites rely on internal search engines to drive thousands or millions of dollar in revenue, yet there is very little written about optimizing them to improve conversion, revenue per visit, or simple click-through rates. On-site search consists of two transactions of interest, both of which can be optimized to improve usability, and the visitors ability to find what they are looking for:
1. The search box itself:
The primary goal of optimizing the search box itself is to make certain visitors on the site can find it and understand what it does. Typical tests would revolve around the placement of the search box, size of the text field, color, and the appearance and text on the button.
2. The search results page:
Once people complete a search, they arrive to a results page which displays matching products or content. The goal of the results page is for people be able to quickly see the results and then link to the content found. Certainly, the effectiveness of this page will be highly dependent on the relevance of the results, but on-page factors (which can be optimized) play an important role as well.
We have seen simple changes and optimization increase search result page engagement by over 40%, which is a very big lift for something most people would think has little to do with testing.
The results page offers up many more areas of testing and optimization. Here are a few to consider:
When you test these, you should make sure to track both result page engagement (did people click on the search results) as well as revenue per visit. Subtle changes in the search system can make a big difference in revenue for many e-commerce sites, so never rely on engagment metrics only!
1. Decide and Document your Goals
Before you ever start testing, you should have an idea of what you plan to get out of it. You should not be testing just to join the “cool kids.” Going into your testing program you should have a written goal. It may be broad, but having that goal will change how you proceed through each of the steps of creating and launching your test.
2. Assemble your Team, Communicate what you are doing internally
Identify who will be involved with the test, and make sure each person understands their specific role. If you will need custom creative assets developed, identify the stakeholders and get some time carved out in their schedule. Let other groups in your organization know you will be performing testing on the site so they do not get surprised be an unexpected change to the site.
3. Research problem pages on your site with analytics
Use web analytics tools to spot trouble areas in your site. Things like high drop off or bounce rates, low conversion rates, and poorly performing landing pages should stick out like a sore thumb and could be ideal places to focus your testing campaign. Also look for pages that get a significant amount of traffic, as these may be prime candidates for testing as well.
4. Choose the page on your site to test
The next thing to do is determine what page or pages you want to include in your test. In many cases, this will be obvious- most people start with either specific landing pages, or (eek) their home page. Home pages are notoriously difficult to test because of competing goals and noisy traffic but if you analyze specific traffic segments you can still get good learnings. Regardless, use your analytics research to help choose where to focus your efforts.
5. Decide what variables to test
This is incredibly obvious, but of course you have to decide what things to test. We have devoted a number of past blog posts to this topic, but it is worth repeating: focus on stuff that appears “above the fold” like headlines, images, page layouts, inclusion of navigation, special offers, promotions, scarcity elements, and risk reversal. Below the fold focus on order and buy buttons, forms, and more risk reversal (near the buttons.)
Image by Getty Images via Daylife
6. Create radical options
This has also been written up before, but if you are going to run a test, run a TEST. Don’t be shy! Minimal changes will result in minimal differences in performance- leading to drawn out test times and the possibility of a NULL test result (no clear winner no matter how much traffic you get.) Don’t test 10 different shades of blue for that order button- test 10 different colors from across the spectrum!
7. Identify external factors and plan to block them
There are many external factors which can sway test results. An easy to understand example is “day of week.” People tend to behave differently on weekdays vs. weekends. This can show up in a test with different results on weekends than on weekdays. To block this data from affecting your overall results- make sure you run the test for at least a full week or two full weeks. Alternatively you can analyze your results for just weekday visitors or just weekend visitors although not all testing software supports this.
8. Develop a Hypothesis
Just like you did in science class growing up, you should form a hypothesis of what you think will happen. This statement summarizes what you are testing, why you are testing it, and what you suspect will happen.
9. Decide whether to run a multivariate, split or URL test
Multivariate tests have the advantage of speed and understanding: you’ll get more testing done in less time- particularly with a fractional factorial MVT. You’ll also get the advantage of seeing how various changes work when combined together. The downside is that management has less understanding and getting buy-in may be tougher. Split tests let you focus on one or more variables on a page test alternative treatments… think headlines, buttons, hero shots, etc. If you wish to test really big ideas, like completely different page layouts- try testing multiple different URLs!
10. Make sure your site is “modular”
The best way to run any type of experiment is to focus on something called “variable isolation.” This means that each thing you are testing should be tested in an independent manner- thus making certain that any performance lift (or loss) you observe are directly tied to the changes you have made. The easiest way to get to variable isolation is to design your web pages in a “modular” fashion, which makes it easy to target specific things- like the headline, images, or info box.
11. Consider upgrading your HTML
Modern XHTML is better for testing than table based layouts. Coding your page with XHTML and CSS will give you cleaner code which renders much more quickly in modern browsers than older table based layouts. It has the advantage of being easy to test with CSS style tweaks (as opposed to submitting entire blocks of html) and also will help your SEO.
12. Identify the success metric and proper value to judge the winner
Looking back at #1, you should be able to extrapolate from your goal to pinpoint the specific metrics you need to base the test on. Although many people talk about “conversion rate” as the best metric to judge a result- this is often not the case. If you are selling products or run an e-commerce site, metrics like Revenue Per Visitor (RPV) or Average Order Value (AOV) will provide a much clearer view of the test winner. Trying to increase ad clicks means that you should be using outbound click tracking, and if your goal is to just increase site stickiness- then track engagement or bounce rate.
13. Make sure your Adgroups and other traffic sources are passing identifying parameters
If you want to be able to segment your test results by adgroup (yes you do) you’ll need to make certain your Adwords account is set up to pass an identifying parameter in the URL. You probably already do this for your standard analytics- but if not, make sure you do it for testing purposes. Often you will see visitors from different adgroups responding to your tests differently.
http://www.yoursite.com/landing.html?adgrp=blue_widgets
14. Implement a Testing Solution
Select a testing software solution to implement the test on your site. Different solutions integrate differently (this could be an entire post.) Some require you to inject javascript calls throughout your HTML, while others are far less intrusive.
Image by Runs With Scissors via Flickr
15. Test the Test: Make sure everything is working before you launch
Before you launch or auto schedule the launch of a test, make certain everything works as you expect, including the tracking. There are plenty of examples out on the web of people who have launched tests only to find out later that there was a major problem with the tracking or a variable not displaying correctly. Here again, different software providers offer different capabilities for checking thing. Also, some providers update results in real time, making it easy to spot problems quickly.
We just announced some new features for internet marketing agencies, http://bit.ly/15ECOy (press release.) We are excited to bring the Vertster multivariate platform to our agency partners with some major features that allow agencies to deliver A/B and multivariate testing to clients.
1. Private Label – Skin the software to have your colors, your CSS, your logo, even your own domain.
2. Flexible Pricing - Do you have a lot of clients? Or just a project from time to time? Either way, the pricing structure lets you work within your business model.
3. Support and Resources - Need help setting up a test? Need help in selling a client? Need whitepapers or collateral? We have it and we can private label that as well!
4. Client Ready Graphical Reporting - Run reports for your clients showing their conversion lift, conversion rate, and winning versions as well as many other important KPI’s.
5. Increase Your Revenue - With our pricing model it’s easy to monetize the service. Also you will be providing a valuable service that clients will be willing to pay for. Get ready to open up a whole new revenue stream!
6. Login Control - Want to give your customers access to the reporting capabilities of Vertster? No problem- we’ll keep them from breaking things and doing other unauthorized things that could hurt their results.
If you are from an online marketing agency or an interactive media agency and want to learn more call us at 801-571-8200. We would love to show you a demo of the new features designed specifically for you.
We are testing junkies and we do it for a living on the web. But we always appreciate a good multivariate or split test, whether it’s on the web or not.
This interesting article popped up recently about a test done in Scotland for lost wallets. What variables combine to entice the finder of the wallet to return it?
The variable was a picture and the options were, an elderly couple, a family, a baby and a puppy.
The picture was supposed to elicit an “Ah that’s cute!” moment and “Somebody is going to want that back!” feeling. The desired action? Return the wallet!
Which picture do you think got the most returned wallets? Keep in mind that each wallet was completely void of cash and didn’t contain any credit cards, only typical wallet items. But the picture was prominently displayed.
If you guessed the baby, you guessed right. Here are the return rates.
80% of the baby
53% of the puppy
48% of the family
28% of the elderly couple
A wallet with a charity donation receipt was also tested and had a 20% return rate and wallets with no special items(The Control) had a 15% return rate.
It just goes to show that when you want someone to take a desired action, whether it be on a website or on the street, you need to play to their emotions. Emotions play a huge role in marketing and if you can hit the right triggers, you can elicit big responses from your multivariate tests!
I’d like to try the wallet test again, but this time let’s add some money to the mix!
Oftentimes when you are “testing” a test, you should clear your cookies each time through- to make sure you do not have old values stashed away that could prevent you from seeing your test operate correctly. Luckily, there are some great extensions available for the Firefox browser that make this a trivial operation. If you are using one of the other browsers, follow our complete instructions below:
1. Firefox with the Web Developer toolbar: First of all, this is definitely the easiest way to clear cookies for the Firefox browser. If you do not have the Web Developer Toolbar installed, head over to the Firefox Add Ons site and grab it.
To use it, click on the Cookies menu on the left, and select “Delete Domain Cookies”. Pow. All the cookies for the site you are currently on will be removed!
2. Clearing a Specific Cookie on Safari: Safari makes you do a little more mousework to get to the listing of cookies in the system, and you have to specify which you want to remove. To get started, go to the preferences menu, and open the “Security” Options. Partway down the page is a button labeled “Show Cookies.” Click this.
Once you have clicked the “Show Cookies” button, you will be shown a listing of all of the cookies your browser has stored, along with a handy search box. Type the current domain name into this search to see a listing of cookies set by the current site. From here, you can select and delete the cookies.
3. Clearing a Cookie on IE 8: Internet Explorer before version 8 made it a real hassle to delete cookies for a given domain, but now version 8 includes a component called Developer tools which is modeled after the Firefox Web Developer toolbar. This makes the process dramatically easier. To begin, select to tools menu and choose “Developer Tools.” This can also be opened by pressing F12.
Once the developer tools window opens, choose the “Cache” menu option on the top and you will the the choice to “Clear Cookies for Domain.” Click this and pow, the cookies will be eliminated for the current domain you are visiting. Super easy!
4. Clearing a Specific Cookie on Chrome: Chrome makes it fairly easy to clear cookies for a single domain, and works similarly to Safari. To start out with, click on the wrench icon on the right side of the address bar. This will drop down a number of choices.
Choosing “Options” brings up another dialogue and if you scroll down a bit, you will see a button for “Show Cookies.” From here there is a convenient search function for quickly locating cookies you wish to remove. Enter the hostname you are looking for and then select the cookies from the list and choose “Remove.” Simple.
Conclusion: It has gotten easier to delete cookies for a specific domain over the years- and now is a relatively painless process. It used to be that many people would delete all of their cookies at once, which is not ideal given that you may loose autologins, personal choices, and other site functionality.
Chances are you need to test your web applications and sites in each of these four most popular browsers, so refer back to this guide in the future to remember how easy it really is to remove specific cookies.
Copyright 2012, Vertster, Inc.
