Multivariate Testing

Taguchi who? I don't speak Japanese!

In the 1940’s and 1950’s, a Japanese scientist and engineer named Genichi Taguchi became known for a radical new method of improving manufacturing quality. This technique is known as the Taguchi Method of Process Improvement.

The Taguchi method is a derivative of Design of Experiments (DOE). Through sophisticated analysis of the results, the experimenter is able to learn the relative impact of each variable, including how important it is to the overall output of the experiment.

Multivariate Testing using the Taguchi Method is particularly well suited to optimizing marketing communications. It performs better than other algorithms in noisy environments (those with lots of uncontrollable variables.) Taguchi is also a good choice because it gives insight with relatively smaller sample sizes.

A typical Taguchi experiment goes through several phases:

Screening:

This is the initial test phase of a Taguchi optimization. The goal of the screening phase is to take a superficial look at many variables. The objective of the screening phase is to determine which factors are influential and should be candidates for further optimization.

Optimization:

Involves drilling in on just the variables which have been shown to be the most important through the screening round. The algorithm will provide the “theoretical” best combination of variables. This should be interpreted as a direction to head in the fine tuning round. Due to inherent marketing noise, it may or may not prove to be the best.

Fine Tuning/Verification:

Seeks to improve the performance a few additional percentage points by fine tuning the creative design. Fine tune your optimization by taking the theoretical best design, the best recipes which have actually run in your test, and any additional combinations you wish to investigate, and put them in a split test against each other, to determine the best performing recipe.