Adaptive Multivariate Testing sets a new performance standard for conversion rate optimization
Website optimization, i.e. optimization for high page conversion, is generating a lot of marketing buzz. The products in this category are often called multivariate testing solutions. In this write-up you will learn about a new innovative methodology introduced by www.hiconversion.com that is dramatically improving performance and expanding the reach of this important technology.
Why you should care about testing methods?
The primary advantage of multivariate testing is its ability to test a large number of page versions while varying multiple elements at the same time.
Prior to Hiconversion there were two common multivariate testing methods. They are defined as the full factorial method and the fractional factorial method. Now, there is a third approach - adaptive multivariate testing.
Novice users can easily be confused about how these methods work and the differences between them.
What follows is a high level overview of these three methods. Particular attention is paid to the number of web visitors required to arrive at statistically significant results. Prior to Hiconversion the need for high volume web traffic prohibited the great majority of websites from using this technology.
Full Factorial Method
The full factorial method is a fancy way to describe an approach that tests all possible combinations and then determines a winner.
Useful Info
Number of page combinations: If you decide to test two elements of your landing page by assigning three different variations to each one of them, you create a total of 16 possible page combinations:
Adding more elements to the test dramatically increases the number of combinations. For example, a test with five elements and three variations of each generates 1,024 combinations (4x4x4x4x4).
Statistical accuracy: Another aspect of the page testing challenge focuses on the statistical accuracy of measurements. Without dragging you through a boring tutorial on statistics you can assume the following guidelines:
- 25-50 conversions will give you an 80% confidence level in the accuracy of your results
- 100+ conversions will produce a 95% confidence level
For a landing page that has a 2% conversion rate, this implies that you have to test at least 5,000 visitors per each test combination to achieve a 95% level of confidence.
What follows is a visual representation of the number of visitors necessary for full factorial testing of two variable elements. Each variable element is represented by its own one dimensional axis. The third dimension illustrates the number of visitors necessary to achieve statistical reliability for each page combination tested. The total number of visitors needed to perform this test is represented by the cube below:
The full factorial method has its advantages and disadvantages:
Pros
- Free tools such as the Google Website Optimizer
- Reliable results – there is no question about which page combination is the best,
you have tested them all
Cons
- Prohibitive page traffic requirement - Since every element must be considered
and tested, a large amount of data (time) is necessary to achieve results.
- Limited scope – to make it practical given the amount of visitors and time required to test all combinations companies are forced to run very small tests with very
few variable elements; this limits test potential and marketers’ freedom to
explore new ideas
Fractional Factorial Method
The more sophisticated (and much more expensive approach) is called the fractional factorial method. Instead of testing all possible page combinations, only an array (i.e. subset) is tested and mathematical modeling is used to predict the overall combination winner. This method is very sensitive to the potential interaction between different elements. That is why several waves of testing are necessary before good results are achieved.
The picture below illustrates the reduced number of web visitors required by this methodology.
Pros
- Ability to test a large number of elements (combinations) – use of mathematical
modeling in combination with effective test array design makes it possible to
effectively evaluate a large number of page elements and element variations;
- Quick learning – After each wave of testing a marketing professional can learn about
the importance of each element enabling a more iterative approach to testing.
Cons
- Sensitivity to element interaction – The ideal case for this methodology occurs when
the impact of each page element is independent from other elements. In real life
scenarios, one variation often works better in combination with another element
variation. This affects the accuracy of mathematical modeling forcing multiple
waves of testing.
- Requires a large number of visitors – Running each wave of tests until statistical
confidence is achieved means that the number of required visitors escalates making
this method unfeasible for mid to small websites (some multi variate testing
vendors point to a requirement of 100,000+ visitors/mo).
- Requires expert knowledge – designing test arrays and analyzing test results
requires specialized multivariate testing knowledge
- High price – the existing fractional factorial solutions tend to cost thousands of
dollars per month.
Adaptive Method
Adaptive multivariate testing moves performance to a whole new level. The primary feature of this methodology is its ability to adapt to visitor behavior in real time while converging toward a winning page combination. Actual testing begins with a few random page combinations that are presented to live visitors. Visitor reactions are then processed in real time creating statistical data that is used by the adaptive algorithm to generate a new page combination that will be shown to successive live visitors. During this iterative process, the adaptive algorithm is able to effectively detect elements that impact page conversion in a positive way and to filter out elements that have a negative impact. As a result, the process starts converging toward a winning page or group of effective page combinations quickly.
The picture below illustrates the adaptive method and its ability to effectively converge to the best page combination with a dramatic reduction in number of required web visitors.
Useful Info
- The adaptive method’s effectiveness is achieved through continued learning and a
self-correcting loop. Unlike the traditional fractional factorial method where mathematical
modeling sits and waits until a full array of page combination are accurately tested
(weeks of time and many thousands of visitors), the adaptive method continually learns
and adjusts in real time.
- The foundation of this method is based on the well known gradient search technique.
The gradient is a mathematical term that measures angle or ‘steepness’ of the line or curve
at a certain point. Applied to the website optimization problem, continued gradient
calculations help to determine in real time if a sequence of two or more page combinations
creates ascent (converging toward a better performing combination), descent (moving
toward lesser performing combinations) or staying flat (no improvement or decline).
Adaptive multivariate testing embodies the best aspects of the other two methods: the exactness of results achieved through full factorial testing, and the reduced number of required web visitors produced by the fractional factorial method.
Pros
- Ability to test a large number of elements (combinations) with a minimum number
of web visitors – the adaptive method requires an order of magnitude less visitors
than any other solution making it possible for mid to small websites to effectively
run reasonably large multivariate tests;
- Increases overall conversion rates during the testing period – After brief initial
training the algorithm is able to start converging toward the best solution while
consistently testing page combinations that are better than the control. As a result,
the conversion rate of the entire test, meaning the cumulative conversion of
all page combinations that participated in the test, will be better than the control
- No need for the multivariate testing expert knowledge – The adaptive algorithm
creates iterations for you. You do not need to know anything about the design of
multivariate testing arrays.
- Affordable pricing – Hiconversion’s product is delivered at price point that is easily
acceptable for companies of any size
Numbers do not lie
To quantify the difference in performance between the three methods discussed, we applied all three methods to the following real life multivariate test:
Number of test combinations: 196
Current conversion rate: 30%
Expected improvement: 20%
Conclusion
By breaking performance barriers, adaptive multivariate testing makes it possible for businesses of any size with a relatively low number of web visitors to improve their results through website optimization.
