Continuous integration isn’t only pushing developers to release more often, it’s enabling them to use A/B testing more effectively. Companies like Omniture, SitesSpect and Webtrends have all pushed out A/B testing software to help developers make better choices. And that’s the promise of A/B testing: no more guesswork.
What exactly is A/B testing, and how can it be used in software? Simply put, it’s the act of putting percentages for use on digital assets. In Farmville, for example, an A/B test could be set up to see how many users purchase an in-game pig when it is offered to them, versus how many purchase an in-game cow. A/B testing software would push out either a pig or a cow to Farmville players based on their profiles, or just based on a 50/50 chance, as determined by the developers.
Bob Garcia, optimization solution sales director for North America at Webtrends, has been working in the A/B and multivariable test market for many years now, and he said that proper use of such tests can have a dramatic impact on the bottom line.
He told the story of a recent test consulting engagement Webtrends had with a major motorcycle retailer. “We worked with them on tests for their checkout process. They started with little tests in their existing site, like changing button location, the size of the text, and adding content. They did this incrementally. Some of the tests didn’t derive statistical differences, but then they went and built an entirely separate, parallel checkout process. When they did that, they saw an improvement, which has translated into US$2.5 million in extra revenue by reducing abandonment rates. That’s a great use case where A/B testing is really most useful, in testing really big concepts.”
Thus, most of the work in A/B testing is designing the alternative layouts and deciding what to test. Garcia said A/B testing is much more effective on large changes rather than simple ones like button relocation and text adjustments.
Testing a new market
Hugh Reynolds, CEO and founder of Swrve New Media, is expecting A/B testing to be the new market for middleware. He was a cofounder of gaming physics middleware company Havok, and with his new company, he’s offering SaaS middleware for the Facebook game crowd.
For Swrve New Media, that aforementioned pig can be just about anything. Reynolds said his team has been focusing on expanding the capabilities of Swrve New Media so that it can now handle A/B testing duties within subsets of users, rather than simply spraying tests across an entire user base.
Reynolds said Swrve New Media is the perfect tool for project managers because they can tweak application performance without introducing bugs, requiring redeploys, or even requiring a developer or IT assistant. “We’re great for product managers who want to adjust things,” he said. “If the product guy comes in and says, ‘We want to run a test and see if we should give people 1,000 starting coins or 500 starting coins,’ the dev team is going to say ‘No way!’ ”
But Swrve New Media allows such substitutions to be performed quickly from a Web interface. Developers add tags into their code where A/B tests can be performed, and the Swrve New Media interface allows for a simple point-and-click substitution on the basis of user identity, percentages or other factors.
“It’s the product managers who are our champions,” said Reynolds. “It really hangs on how a whole organization behaves. The old model was that product management was out on a peninsula, and they’d walk the drawbridge back to development and maybe be refused admission. Every few weeks, the high priest of analytics pulls the numbers out of a hat and says, ‘This proves everything!’ ”