Multivariate Testing

Multivariate Testing an Optimizers Best Friend

Multivariate testing is the process where more than one component of a website, app or other digital property is tested in a live environment. This differs from A/B tests which are usually performed to determine the better of two content variations with a single variable differential; multivariate testing theoretically tests the effectiveness of limitless combinations. The limits on the number of combinations and the number of variables in a multivariate test are governed by statistical significance where to get a valid result a larger sample of visitors is required.

Multivariate testing is employed in order to ascertain which variation produces the best improvement in the defined goals of a digital property, whether that be user registrations, successful completion of a checkout process or a white paper download (conversion rate optimization). Increases can be seen through testing different copy text, layouts, images, background colors and more. However, not all elements will produce the same increase in conversions and by looking at the results from different tests, it is possible to identify those elements that produce the greatest increase in conversions consistently.

Testing can be carried out dynamically with the digital property displaying the different variations of content in equal proportions to users. Statistics on how each user behaved after seeing the content under the test must then be gathered and analyzed. This type of user behavior is optimized via our proprietary testing and tracking formulas; Behavior-Based Media Mix Modeling and our Real-Time Dynamics.

Multivariate testing can be seen as allowing the users of a digital property to vote with their actions for which content they prefer and will stand the most chance of their proceeding to a defined goal. The testing is transparent to user, ensuring that each user is shown the same content on every visit.

Most digital properties benefit from continuous optimization as user response to creatives and layouts differ by time of day, day of week, particular season, channel of origination, etc.

Multivariate testing is currently a high growth area in online marketing and a staple in Digital Strategy as it helps business owners to ensure that they are getting the most from the users of their websites, app, digital displays, etc. Areas such as search engine optimization and pay per click advertising bring visitors to a site and multivariate testing allows digital marketeers to ensure that visitors/users are being shown the correct offers, content and layout to convert them to sale, registration or other action.

There are two main approaches to multivariate testing on digital properties. One being Tagging; a process where the creator of the digital property inserts code to inject content variants and monitor visitor response. Page tagging typically tracks what a visitor viewed, for how long that visitor remained on the site, what actions they took and what pathways they navigated.

The second approach does not require page tagging. By establishing a DNS proxy or hosting inside the digital property’s own datacenter, it’s possible to intercept and process all user traffic to and from the property undergoing testing, insert variants and monitor user response. In this case, all logic sits server side rather than browser side and after initial DNS changes are made, no further technical involvement is required from the digital property point of view.