Split Testing Strategies

What is a control offer, and what if you don't have one?

A marketing test is normally made up of two or more "candidates," called creatives. The creatives may be different in their look, appeal, offer or price.

The test can either be run using an A/B split, or following a "one after another" approach, proceding until one of the test members demonstrates clearly superior performance.

When testing a new creative against an existing ad, the older candidate is referred to as the control. It is the element in your test that you have used the longest... the control is what you have always done. You should also have good documentation of how well it performs- an ongoing accounting of its conversion rate.

The new candidate (or candidates because you can have more than one,) is sometimes referred to as the "test." This is somewhat confusing- because the whole process is called the test, and the test is made up of a test member and a control member. That's a lot of "tests!"

In order to test against the control, you should have a thorough understanding of how well it works, and a documented history of its' performance. Without being able to quantify its' effectiveness, it will be impossible to determine if the test member is really better.

Of course, before you can run a test you have to consider what to test. If you haven't read it yet, you may find our article ["To really improve your results, you have to know what to test"] (/split-testing-strategies/to-really-improve-your-results-you-have-to-know-what-to-test) helpful!

What if you don't have a control?

If you do not have a control, then your first objective in testing will be to try out some different ideas, and establish one. The circumstance of not having an established control is actually quite common. It is handled by designating all available candidates as "tests."

Your objective is to evenly test them against each other, and determine a control for future activities.

One reason for not having a control would be if you have not kept accurate records of your current ads performance (i.e., you have no idea exactly how well it converts.) This is very common, as most business owners don't understand how or what to use as a measure of success online.

Another common occurence: launching a new product, offer, or campaign. In this case, there is no history, and no ads have ever run before. Launching a new campaign is a fun time- you can get creative and try three completely different ideas.

Track the three candidates, and eventually you will have yourself a winner. Having a winner means you have identified a control. This is what you will be using from now on, and this is the one that others will be measured against!

Conclusion

A control is the proven element of any test. Its' performance is well documented, and it represents the standard by which other candidates are measured.

Just because you don't have a control, doesn't mean you can't test. It simply means that the best performer of your first test will become the future control.

Lack of a control affects how you split the audience between the test elements. Without a control, there is nothing to lose, and evenly splitting traffic between the candidates is the best way to go. Use an even 50%/50% split for a two member test, or 33% each for a three member split.

With a proven control, limiting the potential for losses becomes a priority. In a two member test, sending 50% of your potential profits to a poor performer is a real risk. To combat this situation, plan on splitting your audience such that they see the control the majority of the time. Using a split that directs 80% or 90% of the audience to the proven candidate reduces your risk to current results. It may take a little longer to get large enough numbers to identify the better performer, but in the long run, you will be better off.