Recent Entries:
Vertster.com
Conversion Rate Marketing Resource Center
| Top Story: Multivariate Optimization vs. A/B Testing |
|
||
|
Before you begin a testing or optimization strategy, you first should decide on a method. Generally speaking, there are two broad techniques: A/B Split Testing, and Multivariate Testing. Despite what some of my colleagues in the testing industry say, Split testing is not "The dinosaur" of testing. It is not going away soon. Instead, you should think of split and multivariate testing as two different tools to keep in your box. There are times for each. Understand that both split tests and multivariate tests compare a set of page "recipes" or creative designs. This set is called the "test battery." Two methodologies: Optimization vs. Testing Although you will often see the word "testing" used interchangeably with "optimization", they are really quite different. The concept of testing is most often associated with split testing, whereas optimization is normally linked with "multivariate optimization." The difference, although subtle, is important. Testing is something that normally involves manual interaction, a hypothesis, experiment, and analysis of the results. Split testing compares two or more options of the same variable. Analysis is relatively simple- one version will win or not. Optimization is a process which seeks the best possible selection of variables to maximize some output (referred to as the success metric.) A multivariate optimization considers many variables, with many options, with a goal of determining the best combination of them. Analysis is not immediately clear. However, the output of the experiment will be an "optimized" page design- calculated automatically by the mathematical algorithm. I advocate including both Split Testing and Multivariate optimization as part of your overall testing campaign. They each have their own pros and cons, and there are definitely times when one is preferable to the other. Advantages and Disadvantages of Split Testing Split testing is useful for quick tests, and those that involve only simple changes. Split tests can be launched rapidly because they do not involve substantial creative development before the test can begin.. I frequently say that you can think of a new test at lunch time and be running a test within a few minutes of getting back to the office. The biggest disadvantage of split testing is that it is only able to look at one change at a time. Advantages and Disadvantages of MVT There is no doubt that multivariate optimization has the potential to give you the biggest bang for your testing buck. The ability to test several thousand (or even a million) combinations in one run would require a lifetime of split testing to get an equivalent learning. Multivariate algorithms also tell you what things on the page are causing the most influence on your effectiveness. Powerful information for further testing. Perhaps the most compelling advantage of multivariate testing is that the algorithm automatically determines the best page design for you. Unfortunately, setting up a good multivariate test takes time. Because you are testing multiple variables and multiple options for each, there is a lot more creative work that must be completed before the testing can begin. The design of the test and development of creative options must be done with great care. And, even though the output of the experiment a "theoretical" best page design, this does not mean you should turn your brain off. In many cases the best design may need a little further tweaking. |