A newer client of ours asked a great question recently:
“What if we create a completely new site from scratch using the persuasion framework you developed and then A/B test it against our old site — What would happen?”
Let me answer the second part of the question first: The one thing I’m quite certain of is that humans are awful at predicting the future. If someone tells you they know what’s going to happen in the future put it to the test. I’ve got one buddy who claims that he can “often” predict the future, albeit only about 10 seconds forward. “Great!” I told him. Let’s go to Vegas, and I’ll put up the money and we’ll play craps all weekend. That’s surely a fast enough game that your 10 second limitation won’t stop us from getting rich!” We never did make it there — somehow it was never the “right time” to predict the future, I suppose.
But when it comes to optimizing your online efforts and you (or, more typically, the boss) presumes to know what will happen next, hold yourself and your team to a hard objective measure: TEST! It’s the best way to leave the subjective world of opinion and enter the objective world of reality.
But of course, that’s what our client really meant with his question. To rephrase it, one might ask, “Is it more efficient to A/B test large scale changes by jumping into a vastly improved architecture first and then proceed with incremental improvement of that new architecture?” As your instincts might indicate, this can be an ambitious way to jump-start optimization efforts, under the right circumstances.
Most sites have a legacy architecture that actively works against persuasion and conversion, and if your pockets (and your nerves) are big enough, the ultimate A/B test is to say, “Hey, what we have now is a Control benchmark. What we really want to to make sure that the major persuasive scenarios are aggressively planned to be ready for optimization and then go from there.” You’ll note that I’m emphasizing the persuasive planning portion because just throwing a new design up isn’t likely to leave you in a position to learn from future optimization as efficiently as possible. Instead the purpose of the newly revamped site is to accept the old site as a benchmark to measure against, and to put in place the series of key performance indicators and measurement points which will be used in ongoing optimization quarter after quarter, using experience and insight.
There are some additional issues that should be thought through of course: First off, the technical implementations that will continue to send a portion of your traffic to the old site. Second, make sure you have your apples and oranges understood so that you compare the correct new analytics with their counterparts from the old site’s analytics. And third, don’t forget that you’re not testing in a sterile academic environment where the goal is to understand all permutations completely, but rather in the living, breathing biodome of a company with live customers and a live income stream. As improvements are verified more and more traffic should be assigned to those improved points of conversion and persuasion — much like the medical profession where the adage “first, do no harm” is the rule.
Personally, I have to give this client props for the sheer nerve of bringing this topic up. It shows a confidence in his team’s ability to implement and live with change — “the only certainty is change” — as well as a willingness to quickly say “ok, we were wrong in this aspect, so let’s analyze/test/optimize yet again”.
I think this is going to be a fascinating long-term experiment. What about you?