Conversion Rate Optimization 101 - Part II: Multivariate Testing

According to FireClick 97% or more of your hard earned web traffic is wasted. Such poor online marketing performance is not acceptable or sustainable. A solution for this problem can be relatively simple and affordable. This three-part blog post is designed as your hands on guide into the world of Conversion Rate Optimization (CRO) and dramatic increases in the online marketing ROI. What is Multivariate Testing? In Internet marketing, a/b or split testing is a process of measuring the performance of a few different versions of a web page to determine which one has the highest conversion rate. Split testing is very useful for determining which page layout performs the best, but it is very limited in its ability to find a version of the page that fully maximizes a conversion rate potential. Multivariate testing, on the other hand, can be viewed as a process of a deep page testing where you are measuring the impact of simultaneous changes of several page components on the overall page conversion rate. The number of possible page combinations that can be created by changing multiple page components can quickly escalate into the thousands and sometimes even into the millions. Testing such a large number of page combinations is a science that requires automation and effective multivariate testing technology. About Multivariate Testing Methods In the not so distant past, multivariate testing solutions were available exclusively to those who had deep pockets, technical resources, and high volume of web traffic. In last year or two the situation has dramatically changed making it possible for companies of any size to use multivariate testing. Multivariate testing technology is still evolving and it is critical to recognize that not all multivariate testing methods are the same. On a high level one can identify three main product categories: Full Factorial Testing Tools The full factorial method is a fancy name for testing of all possible page variations. Since every page variation is tested until statistical reliability is achieved, a large amount of visitors (time) is necessary to complete the test. This type of testing is so slow that for a great majority of websites a multivariate test with more than a dozen page variations can be very impractical. In this product group we recommend use of free Google Website Optimizer. This is a great entry level multivariate testing solution that will give you an insight into the value of split and multivariate testing. Fractional Factorial Testing The more sophisticated (and much more expensive approach) is so called the fractional factorial method. Instead of testing all possible page variations, only an array (i.e. subset) is tested and mathematical modeling is used to predict the overall winner. This method can produce a 10x improvement to the full factorial method but still requires high volume of traffic. - The best known products in this product group include:    Accenture Digital Optimization (formerly Memetrics, Inc.)    Interwoven Optimost    Omniture Offermatica    SiteSpect    Vertster    Widemile Adaptive Multivariate Testing Hiconversion, Inc. has invented a patent pending adaptive multivariate testing methodology that moves performance needle for another factor of 10 in comparison to the fractional factorial method. As a result, it is now possible to run a reasonably sized multivariate test for web pages with only 100-200 visitors per day. How To Select a Tool That Fits Your Needs Although the multivariate testing methodology is a very critical element of any multivariate testing solution, it is not the only criteria that you should use in making a tool selection. The table below summarizes other product features that can impact effectiveness of your test in significant way: Experiment Setup The following are the practical steps that you should follow during the experiment setup. Step 1. Determine If You Need a New Page Design Sometimes your existing landing page is so poorly designed that 'perfuming the pig' will not be the best use of your time and resources. We suggest that you take a critical look at a page you want to test, and conduct a simple 'smoke test'. All you have to do is to verify that all three critical page elements exist and that they are all placed above the fold (see below). If you page is not satisfying these basic requirements we would recommend that you create a new layout. Step 2: Determine Duration of Your Test Different multivariate testing methods will perform at different speeds, but you will still have to create an optimization experiment that can be completed within a reasonable period of time. First, let us help you calculate the total number of page combinations that you may need to create during your multivariate test. The math is very simple, the total number of page combinations is equal to: the consecutive multiplication of the number of variations in section one, the consecutive multiplication of the number of variations in section two, and so on... For example, if you design an experiment with 4 sections and 5 variations each, a total number of page combinations is 5 x 5 x 5 x 5 = 625. When we say 5 variations, we mean control (the existing element) and 4 new variations. Second, based on your current web traffic you have to estimate how long it will take to complete the test. Ideally, we recommend no more than 2-4 weeks of testing. Few multivariate testing tools offer a test duration calculator:  Google Website Optimizer Duration Calculator Hiconversion Test Duration Calculator Just to give you a feel for test duration, let’s use the two calculators from above and apply it to a few typical real world situations As numbers indicate, the bigger the number of test combinations the bigger the difference in test duration between two multivariate testing methodologies. Step 3: Design Sections and Variation Often website owners are paralyzed by a luck of ideas about what elements of the web page they should be testing and how to test them. Let us share with you few resources that will help you jumpstart your experimental design. Marketing framework The most complete method for designing a multivariate test can be described by the Conversion Sequence formula developed by MarketingExperiments.com. Where c = Conversion, the other sequence elements refer to:    m – the match between the offer and visitor Motivation.    v – the clarity of the Value Proposition.    i – Incentives used to counter Friction.    f – the level of Friction in the sales process.    a – Anxiety caused by the process. While Motivation has the highest coefficient in this formula, it also represents an external factor in the marketing cycle that is beyond your control. That makes the clarity of your Value Proposition the most important internal factor. However, many marketers try to improve results by changing page elements like font colors and sizes, button shapes, images, incentives, and so on, when the first step should really be focusing on strengthening their value propositions. Practical Tips There are many tips that you can find yourself on the Internet. Few of our favorites are:     Google Website Optimizer 101 – a quick-start guide to conversion rate optimization 10 Landing Page Optimization Tactics Landing Page Utopia: What you can learn from the world's greatest landing page Pitfalls According to many analysts, the most common errors in testing and analysis include:    Bias: Approaching a test with a clear predisposition toward a particular outcome  Impatience: stopping a test too soon based on early results  Extrapolation: Drawing overreaching conclusions from limited test data  Follow-through: Failing to prepare follow-up tests Step 4: Define Conversion Goals Defining your conversion goals is usually a simple task of identifying the most desirable action that should be taken by the test page visitor. Some tools will allow selection of multiple actions on the same test page. This can be very convenient for landing pages that offer multiple calls for action leading to different pages in the sales funnel. Others will force you into defining conversion goal as tracking of visitors who reach a pre-defined conversion page after visiting the test page. This is quite straightforward but can limit your ability to measure multi-call conversion actions. Step 5: Optimization Enable Your Test Page Enabling your test to participate in the multivariate test is a technical step that can be more or less complicated. For example, if you decide to use Google's Website Optimizer you will need to insert few different types of scripts into your test page and tracking scripts into conversion page, as shown below: The scripts above are test and section specific so each time you decide to make a change in your test, or to run a new test you have to update scripts into your test and conversion pages. This can be a complicated error prone process that fully depends on the availability or IT resources. Hiconversion's transparent enabling method simplifies enabling process by requiring you to insert script into a generic page locations only one time in the life of the web page. After that you can make changes in your test or run a whole different test without any need to touch your web page source code. The third method of optimization enabling is called reverse proxy method. Here you have to deploy a proxy server in front of your existing web server. Although this approach can provide the highest optimization speed, many companies are very reluctant to make changes in the website hosting setup. Running a multivariate test If you are using Google's Website Optimizer or Hiconversion Pro, all you have to do is to start your test by pressing a 'play button'. Taguchi factorial design Some Taguchi based solution will require an addition step of factorial design. Namely, before beginning of the test you will need to identify which subset, called an array, of page combinations will participate in the first wave of testing. To learn more about Taguchi method please visit Jonathan Menendez's blog. For example L8 arrays are used for 7x2 (7 elements and two variations) and L9 for 4x3 (4 elements and 3 variations) MVT. Know when to stop your test One of the pitfalls of multivariate testing is that marketers get excited about results too early. Often they immediately stop the test and apply unreliable winning page combination. For those of you who overwhelmed by statistics, here are few rules of thumb:    25 to 50 conversions are required to be somewhat confident in a given landing page's reported conversion rate (roughly 80% confident) we recommend 50 to 100 conversions which will put you in 80%-95% confidence range 100+ conversions are required to achieve 95% or higher confidence If you desire to learn more about how to calculate statistical confidence of your results, please visit an excellent writup produced by Market.com.

Not all multivariate testing methods are equal

Adaptive Multivariate Testing sets a new performance standard for conversion rate optimization

Website optimization, i.e. optimization for high page conversion, is generating a lot of marketing buzz.  The products in this category are often called multivariate testing solutions. In this write-up you will learn about a new innovative methodology introduced by www.hiconversion.com that is dramatically improving performance and expanding the reach of this important technology.

Why you should care about testing methods?

The primary advantage of multivariate testing is its ability to test a large number of page versions while varying multiple elements at the same time.

Prior to Hiconversion there were two common multivariate testing methods. They are defined as the full factorial method and the fractional factorial method. Now, there is a third approach - adaptive multivariate testing.

Novice users can easily be confused about how these methods work and the differences between them. 

What follows is a high level overview of these three methods. Particular attention is paid to the number of web visitors required to arrive at statistically significant results. Prior to Hiconversion the need for high volume web traffic prohibited  the great majority of websites from using this technology.

Full Factorial Method

The full factorial method is a fancy way to describe an approach that tests all possible combinations and then determines a winner.

Useful Info

Number of page combinations: If you decide to test two elements of your landing page by assigning three different variations to each one of them, you create a total of 16 possible page combinations:

Adding more elements to the test dramatically increases the number of combinations. For example, a test with five elements and three variations of each generates 1,024 combinations (4x4x4x4x4).

Statistical accuracy: Another aspect of the page testing challenge focuses on the statistical accuracy of measurements. Without dragging you through a boring tutorial on statistics you can assume the following guidelines:

- 25-50 conversions will give you an 80% confidence level in the accuracy of your results
- 100+ conversions will produce a 95% confidence level

For a landing page that has a 2% conversion rate, this implies that you have to test at least 5,000 visitors per each test combination to achieve a 95% level of confidence.

What follows is a visual representation of the number of visitors necessary for full factorial testing of two variable elements. Each variable element is represented by its own one dimensional axis. The third dimension illustrates the number of visitors necessary to achieve statistical reliability for each page combination tested. The total number of visitors needed to perform this test is represented by the cube below:

The full factorial method has its advantages and disadvantages:

Pros
- Free tools such as the Google Website Optimizer
- Reliable results – there is no question about which page combination is the best,
  you have tested them all

Cons
- Prohibitive page traffic requirement - Since every element must be considered
  and tested, a large amount of data (time) is necessary to achieve results.
- Limited scope – to make it practical given the amount of visitors and time required   to test all combinations companies are forced to run very small tests with very
  few variable elements; this limits test potential and marketers’ freedom to
  explore new ideas

Fractional Factorial Method

The more sophisticated (and much more expensive approach) is called the fractional factorial method. Instead of testing all possible page combinations, only an array (i.e. subset) is tested and mathematical modeling is used to predict the overall combination winner. This method is very sensitive to the potential interaction between different elements. That is why several waves of testing are necessary before good results are achieved.

The picture below illustrates the reduced number of web visitors required by this methodology.

Pros
- Ability to test a large number of elements (combinations) – use of mathematical
  modeling in combination with effective test array design makes it possible to
  effectively evaluate a large number of page elements and element variations;
- Quick learning – After each wave of testing a marketing professional can learn about
  the importance of each element enabling a more iterative approach to testing.

Cons
- Sensitivity to element interaction – The ideal case for this methodology occurs when
  the impact of each page element is independent from other elements. In real life
  scenarios, one variation often works better in combination with another element
  variation. This affects the accuracy of mathematical modeling forcing multiple
  waves of testing.
- Requires a large number of visitors – Running each wave of tests until statistical
  confidence is achieved means that the number of required visitors escalates making
  this method unfeasible for mid to small websites (some multi variate testing
  vendors point to a requirement of 100,000+ visitors/mo).
- Requires expert knowledge – designing test arrays and analyzing test results
  requires specialized multivariate testing knowledge
- High price – the existing fractional factorial solutions tend to cost thousands of
  dollars per month.

Adaptive Method

Adaptive multivariate testing moves performance to a whole new level. The primary feature of this methodology is its ability to adapt to visitor behavior in real time while converging toward a winning page combination.  Actual testing begins with a few random page combinations that are presented to live visitors. Visitor reactions are then processed in real time creating statistical data that is used by the adaptive algorithm to generate a new page combination that will be shown to successive live visitors. During this iterative process, the adaptive algorithm is able to effectively detect elements that impact page conversion in a positive way and to filter out elements that have a negative impact. As a result, the process starts converging toward a winning page or group of effective page combinations quickly.

The picture below illustrates the adaptive method and its ability to effectively converge to the best page combination with a dramatic reduction in number of required web visitors.

Useful Info

- The adaptive method’s effectiveness is achieved through continued learning and a
  self-correcting loop. Unlike the traditional fractional factorial method where mathematical
  modeling sits and waits until a full array of page combination are accurately tested
  (weeks of time and many thousands of visitors), the adaptive method continually learns
  and adjusts in real time.

- The foundation of this method is based on the well known gradient search technique.
  The gradient is a mathematical term that measures angle or ‘steepness’ of the line or curve
  at a certain point. Applied to the website optimization problem, continued gradient
  calculations help to determine in real time if a sequence of two or more page combinations
  creates ascent (converging toward a better performing combination), descent (moving
  toward lesser performing combinations) or staying flat (no improvement or decline).

Adaptive multivariate testing embodies the best aspects of the other two methods: the exactness of results achieved through full factorial testing, and the reduced number of required web visitors produced by the fractional factorial method.

Pros
- Ability to test a large number of elements (combinations) with a minimum number
  of web visitors – the adaptive method requires an order of magnitude less visitors  
  than any other solution making it possible for mid to small websites to effectively
  run reasonably large multivariate tests;
- Increases overall conversion rates during the testing period – After brief initial
  training the algorithm is able to start converging toward the best solution while
  consistently testing page combinations that are better than the control. As a result,
  the conversion rate of the entire test, meaning the cumulative conversion of
  all page combinations that participated in the test, will be better than the control
- No need for the multivariate testing expert knowledge – The adaptive algorithm
  creates iterations for you. You do not need to know anything about the design of
  multivariate testing arrays.
- Affordable pricing – Hiconversion’s product is delivered at price point that is easily
  acceptable for companies of any size 

Numbers do not lie

To quantify the difference in performance between the three methods discussed, we applied all three methods to the following real life multivariate test:

 Number of test combinations:     196
 Current conversion rate:              30%
 Expected improvement:               20%

*See Google calculator

Conclusion

By breaking performance barriers, adaptive multivariate testing makes it possible for businesses of any size with a relatively low number of web visitors to improve their results through website optimization.