JohnQuarto.com » Headline

Why Test? The Purpose of Testing

John Quarto-vonTivadar — Wed, 18 Jan 2012 04:25:00 +0000

Why Test?

The purpose of testing is not to find out what works, but rather to find out what does not work.

I often encounter clients who have what they consider a large percentage of “failed” tests. Yet these tests reveal a rather large amount of information and insight towards future testing.

In fact, when a test “works” — and I use quotes on that to mean “does what we wanted it to do by supporting the hypothesis in some way” — we often learn less because we over-interpret the success. For example, one frequent test failure pattern I’ve seen is: “we tested a large number of headline copy variations and NONE of them showed any improvement! How can this be?”

Often, the subject matter (in this example, the headline copy) isn’t the problem, it’s the contextual basis under which the test was presented. To paraphrase Shakespeare: “The fault lies not in our tests, but in ourselves”. That is where you go to find actual insight that ends up leading to better tests. Ask yourself questions like, “what assumptions did I build into that test?”, “And are they all valid?”, “If I were sitting across the table from this prospect they would need X, Y and Z at this point to continue — so is my test creating a roadblock to that?” .

You may in fact have roadblocked your way out of teasing incremental improvement out of your tests by means of something vitally important but unrelated to your test. And, remember you are testing not on inanimate particles but on humans who have memory and require cognitive resonance in order to proceed. If you disturb that, it requires no small amount of testing effort to tease out when are then minor issues such as headline copy.

Another issue that is often brought up is “how much traffic do I need for my test to be meaningful?” There are rules of thumb for traffic, the most important of which is that the more homogeneous the traffic, the smaller the variance you can expect in the sample of visitors versus the population of visitors. If you have a site that was geared towards something specific — say, late stage Lung cancer patients — you don’t need nearly as large a set of traffic to get meaningful results than with a broader spectrum of, say, eBay shoppers. That is not a trivial meme to keep in mind as the size of your test samples will be driven by that concern, as well as impacting the frequency of the tests and the overall testing schedule you keep.

On some tests you’re going to need 50,000 visitors to get significance; on others you might only need 500. In fact, the closer any number of variations are to each other in their measured performance during the test, the larger a sample size you’d need for each to achieve the same level of confidence in the results.

Speaking of traffic, another issue that arises is what to do when you expect quite a difference between the original version of your site and something new which you hypothesize to be a strong improvement (because, after all, why would you spend your time working one something you didn’t a priori think would be an improvement?). You try umpteen different variations and … no significant results. If you have the traffic to support it, I recommend running a multivariate test to attempt to deconstruct the results… so that you can learn from them. If you started with the multivariate, remove variables and test as a standard univariate (what we normally call an AB) test. When you get an unexpected result, try doing the opposite to see if you get another unexpected result.

One of the most powerful tricks I’ve found is dead-on simple: The first thing you need to do is to repeat the test. You have to convince yourself — and you can do this numerically — that the sample set of visitors of your test is representative from the population of your visitors as a whole. Or, more simply, did you just get a goofy mix of folks in the first test? One can’t really know this from just one test, though there are ways to sniff out some confidence levels.

I’d suggest repeating with lesser traffic since at the end of the day, when you subject any visitors to a less optimized experience you’re costing yourself some money…what you’re looking to do is to see if the results are different, while costing yourself as little as possible while still getting meaningful results. For example, repeat the test with only, say, 10% of the traffic being exposed to it. Yes, it will take longer to run, but while it’s doing so, you’ll be testing other parts of your site anyway. It’s definitely a balancing act!
Further, back to the issue of “are there rules for total amount of traffic for a test?” touched on earlier, I’d also comment that if someone had, say, 5000 visitors taking a test, I’d much rather see the results of 10 of the same tests of 500 visitors each, than one big test of 5000. The challenge with conversion rates that are low, is that you have to expose a larger number of people to the test to tease out insight into what are typically 1-2-3% conversion rates. This means the signal to noise ratio can be rather poor, but the same techniques used in polling (“Candidate Jones 51%, Candidate Smith 49%”) can be useful.

The take-away? Commit to “testing your tests” by repeating them — because if you get a randomly skewed sample of visitors, it will completely throw off your interpretation of the test results. Repeat your “failed” tests to ensure your results aren’t fooling you, and repeat your “success” tests to ensure you aren’t fooling yourself. Worry more about Directionally Correct, not Metaphysical Certitude.

My New Column at Marketing Land

John Quarto-vonTivadar — Mon, 26 Dec 2011 19:47:56 +0000

I’m happy to announce that I’ve been selected by the folks at Search Engine Land as one of their regular columnists for their new sister site, Marketing Land. I’ll be writing monthly articles on topics concerning metrics and analytics important to marketers and to business owners in particular. All starting in late January 2012.

The Ultimate A/B Test

John Quarto-vonTivadar — Thu, 20 May 2010 20:25:52 +0000

A newer client of ours asked a great question recently:

“What if we create a completely new site from scratch using the persuasion framework you developed and then A/B test it against our old site — What would happen?”

Let me answer the second part of the question first: The one thing I’m quite certain of is that humans are awful at predicting the future. If someone tells you they know what’s going to happen in the future put it to the test. I’ve got one buddy who claims that he can “often” predict the future, albeit only about 10 seconds forward. “Great!” I told him. Let’s go to Vegas, and I’ll put up the money and we’ll play craps all weekend. That’s surely a fast enough game that your 10 second limitation won’t stop us from getting rich!” We never did make it there — somehow it was never the “right time” to predict the future, I suppose.

But when it comes to optimizing your online efforts and you (or, more typically, the boss) presumes to know what will happen next, hold yourself and your team to a hard objective measure: TEST! It’s the best way to leave the subjective world of opinion and enter the objective world of reality.

But of course, that’s what our client really meant with his question. To rephrase it, one might ask, ”Is it more efficient to A/B test large scale changes by jumping into a vastly improved architecture first and then proceed with incremental improvement of that new architecture?” As your instincts might indicate, this can be an ambitious way to jump-start optimization efforts, under the right circumstances.

Most sites have a legacy architecture that actively works against persuasion and conversion, and if your pockets (and your nerves) are big enough, the ultimate A/B test is to say, “Hey, what we have now is a Control benchmark. What we really want to to make sure that the major persuasive scenarios are aggressively planned to be ready for optimization and then go from there.” You’ll note that I’m emphasizing the persuasive planning portion because just throwing a new design up isn’t likely to leave you in a position to learn from future optimization as efficiently as possible. Instead the purpose of the newly revamped site is to accept the old site as a benchmark to measure against, and to put in place the series of key performance indicators and measurement points which will be used in ongoing optimization quarter after quarter, using experience and insight.

There are some additional issues that should be thought through of course: First off, the technical implementations that will continue to send a portion of your traffic to the old site. Second, make sure you have your apples and oranges understood so that you compare the correct new analytics with their counterparts from the old site’s analytics. And third, don’t forget that you’re not testing in a sterile academic environment where the goal is to understand all permutations completely, but rather in the living, breathing biodome of a company with live customers and a live income stream. As improvements are verified more and more traffic should be assigned to those improved points of conversion and persuasion — much like the medical profession where the adage “first, do no harm” is the rule.

Personally, I have to give this client props for the sheer nerve of bringing this topic up. It shows a confidence in his team’s ability to implement and live with change — “the only certainty is change” — as well as a willingness to quickly say “ok, we were wrong in this aspect, so let’s analyze/test/optimize yet again”.

I think this is going to be a fascinating long-term experiment. What about you?

Low Hanging Fruit: Cherry Picker or Lettuce Picker?

John Quarto-vonTivadar — Thu, 20 Aug 2009 19:37:55 +0000

When you think of a cherry picker, do you conjure up images of someone who only picks the easiest or ripest fruit? Or does it perhaps have some artisanal connotation, waiting until only the proper time before action is taken? Is that how you go about optimizing your web site?

Or are you a lettuce picker? The sort of person that toils for long hours in the field and accomplishes an honest day’s back-breaking labor of work that most white collar business execs would consider a less-than-optimal career.

That fact is, when it comes time to harvest, virtually the entire crop must be worked on at the same time. You don’t have time to cherry pick, and anyway the average business isn’t expert enough in how and what to optimize to know which portion of the crop should be cherry picked. Instead, when that crop is ripe, it’s time to get out there in the field and put in a 14-hour day getting it harvested.

Often when we speak wiith prospective clients, they have the impression that there is some magical formula that leads to higher conversion rates and that it can be achieved without any hard work or commitment. The reality, however, is that our most successful clients who enjoy on-going regular improvements of 40-80% in their conversion rates year after year are the ones who are implementing change on a regular basis. They’re lettuce pickers, and not so proud as to let hard work get in the way of increased revenue.

Are you a hard-working lettuce picker when it comes to your website? Are you guessing at what changes will improve your site? Or do you work diligently every week, every month, and every quarter to effect continuous improvement?

What Could Happy Billy Teach You About Your Business?

John Quarto-vonTivadar — Mon, 17 Aug 2009 18:51:19 +0000

During a recent visit with family and with the fuel tank showing a big red Empty, my Mom was insistent that we had to get gas from Billy. “Who the heck is Billy?”, I wondered? (Bear with me, this gets interesting.)

We pull into the most ordinary of Shell gas stations (international readers: you might better recognize this company as Royal Dutch Shell) and are greeted by a very happy personality, Billy, who pumps the gas at this full service station. He has an entire conversation with my mother — not just about the expected “what grade of fuel” but about regular life issues such as weather, health, etc. Like two old friends who bump into each other in a cafe. Then he finally goes and pumps the gas. Then I watch Billy move on to the next car where he proceeds to have another friendly conversation with another customer who he clearly is well acquainted with. And on Billy moves to yet a third customer, just as our fuel gauge reaches Full. Billy’s station is always busy, it seems.

“Mom, how long have you been coming here?”

“Oh, since I got my first Subaru, in 2001.”

“Nine years you’ve been going to the same place. There’s plenty of other gas stations all over the place. And you don’t go anywhere else?”

“Why would I? He’s a polite young man and he always smiles. And it’s full service for only a penny more than the self-service stations. Plus, now I recognize the other customers, too.”

Can you imagine that? A sort of social-networking-meets-customer-retention at a gas station? As you might guess, this Happy Billy no doubt means a very happy Shell station owner, selling what is otherwise a most fungible of commodities wherein people will often drive miles to save a penny per gallon. Instead at this Shell station on Post Road in East Greenwich, Rhode Island, people go out of their way to pay more (in a recession) for one of Billy’s smiles.

Now, I’ll be honest. This wouldn’t work in New York City, where I live. People are in just too much of a rush. But in the correct environment where life is slower and individual customers are seen as individual people this is incredibly effective. I can only hope Billy gets a bonus based on revenues.

I wonder, how many online businesses are willing to think of their customers as fellow citizens of the same small town and to know them with the online equivalent of a Billy smile? When’s the last time you felt that sort of allegiance to your cable company, or the convenience store where you buy milk? And think of the long-term revenue it means for a company that achieves that sort of loyalty.

Does your sales and customer service staff treat your customers to a Billy Smile?

Improvement Starts With (Bad) Habits

John Quarto-vonTivadar — Wed, 10 Jun 2009 14:15:30 +0000

Occasionally we hear from clients after they’ve implemented some recommendation for improvement that they see a temporary dip in conversion. This seemingly goes against logic — after all, if you fix a problem, things should get better, right? — but Mammals aren’t entirely logical nor rational, at least not as often as we’d like to think, and particularly when it comes to learned and patterned behavior.

Sometimes it takes your customers a while to “get used to” the changes you made (think of how long your customers’ buying cycle is), especially when they are surprised to wake up one morning and discover such changes implemented. You knew the changes were coming; the customers are generally taken off guard. A short video illustrates:

[youtube]http://www.youtube.com/watch?v=yH52XesnqJQ[/youtube]

Not only will the dog not go out the door, despite evidence it’s not there (she sticks her head thru, for goodness sake…sounds like logical, rational, tested evidence of no door, right?), but instead is cued to the behavior of the door handle. And it’s not immediately clear that she believes the evidence since she then waits outside for the door handle to be involved in re-entering the house. What she really needs is the repetition of the new activity several times before modifying her own behavior. So there’s a delay between the implementation of the change and its measured improvement.

What’s interesting is that you see this behavior in humans too. How many diet solutions focus on giving one a sense of “feeling full”, not because one needs to eat so much, but simply one has gotten used to eating a certain relative volume of food and therefore reinterprets a smaller volume as “I’m not done yet”. Or, your City fixes some streets and roads and months later drivers still need “Warning: New Traffic Pattern” signs to remind them that something different, even something improved, has come along.

Of course. over time we all become used to the new way of doing things and that’s when the actual benefits of optimization will start to yield your company the long-tail, long-term results.

This is why we so often talk about a “cycle of improvement” and a “culture of testing”, because optimization gets its biggest bang from operating continuously. So the next time you make some improvements to your site, consider the idea that it’s completely normal for a dip in conversion to occur at that point in the persuasive process; use the time to plan your next round of improvements, so that when you come out of the dip you’re ready for the next cycle. Get realistic about your expectations and you’ll be pleasantly surprised.

Three Reasons The Easter Bunny Shouldn’t Be Your Marketer

John Quarto-vonTivadar — Fri, 10 Apr 2009 10:25:39 +0000

Well, it’s the Friday before Easter and everywhere you look there’s the Easter Bunny. Despite the great press coverage, there are three excellentreasons why the Easter Bunny should not be your marketer:

Reason #1: The Easter Bunny hides his products. I mean seriously, this is Bad Marketing Commandment Numero Uno, “Hinder Thy Customers From Finding Thy Products”. Now think about this: Senor Bunny has these colored eggs — (the plastic ones have candy inside, in case you grew up in a Easter-Bunny-Free-Zone) — and his merchandising goes to great lengths to hide these eggs from you. Not only is this bad marketing, it’s a stupid technique for the product itself: hide something you just finished painting in pastel colors?

Reason #2: The Easter Bunny provides a product that is unethical. The Evil Rabbit promotes addiction and ill health, by giving kids candy and eggs (hmm, so maybe The Bunny is a secret marketer for the World Dentistry Council and drug conglomerates peddling cholesterol-lowering meds?). Nevertheless, everyone knows candy rots your teeth, and Aubergine Dye #2 is carcinogenic. And when the kids are too old for candy, I bet Mr. Bunny stays “kewl” by slipping them a few smokes. Any marketer that preys on unsuspecting kids and teens ought to be fired on the spot.

Reason #3: The Easter Bunny’s market share is based on incompetence, of his competitors. I mean, let’s face it: the only nearby competitors in the calendar are a Ground Hog (too shy) and St.Patrick (too hungover). Even with a merger, they’d still just be a drunk guy in a green suit with a clover-eating pet rodent. Not the sort of marketing image that sells. The Easter Bunny is only doing well because he showed up, and looks cute. Now, while it’s true that success is often achieved by those who simply show up, a recession is the best time to grab market share from complacent competitors. Just imagine the run Mr. Bunny would get for his money if you put him up against some real competition, like a Turkey serving Pumpkin Pie and Football, or the commerce god himself, The Fat Man in the Red Suit.

And that’s why this Sunday I shall enjoy not a traditional Ham but rather Easter Hassenpfeffer.

[If you're looking for two quick Easter-Bunny themed laughs, check out Doug Savage's cartoons at: http://www.savagechickens.com/2007/04/easter-egg-hunt.html and http://www.savagechickens.com/2008/03/easter.html . The latter just cracked me up. The former I had think for a bit before I got the reference (hint: famous horror movie)]

[Extra thanks to whoever the author is of the funny cartoon at the top of the post.I don't recognize the signature, although the style looks familiar. So if you know who this is, please let me know!]

Making Tabs Work For You

John Quarto-vonTivadar — Thu, 09 Apr 2009 09:27:18 +0000

UI Designer David Leggett wrote an interesting article recently, “Tabbed Navigation, and What Makes It Useful“. His first, and best, point is a physical observance: in a real-world store one has a sense of the physical size of the establishment the moment one enters the store, whereas online there’s no way to estimate the relative size of the enterpise by a quick visual size of its sheer volume of space.

Is it Walmart-ish? Is it a Mom’n'Pop? And does it even matter, if they have what I’m looking for? That’s just an outright good point to keep in mind, even when tabs aren’t involved.

Of course, lack of physicality also manifests as the “great leveler” that allowed an Amazon.com to compete with Barnes and Noble and all the other bookstore chains. It’s also the driver as to why a singular fellow like Matt Drudge (from DrudgeReport.com) or Craig Newmark (from CraigsList.com) give the Associated Press and other news orgs at the Newspaper Association of America a hissy fit.

While there’s a few points I think I’d debate with David over a beer or coffee — such as, “Tabs Can Connect With Secondary Navigation”, to which I’d ask, “well, how did the visitor get to the point where they needed the Navigation after they got their bearings anyway? Doesn’t that imply a lack of (or broken) persuasive engagement with the content? And therefore tabs are operating more like a crutch for someone with a busted leg: a way to re-enable mobility when it’s broken, rather than a vehicle to increase velocity” — I think the conversation itself would be a lot of fun.

Check it out! It’s a 5 minute read and well-worth your coffee break time.

Video Views Up, When Will Sales Follow?

John Quarto-vonTivadar — Tue, 27 Jan 2009 12:34:40 +0000

Hot on the heels of a recent comScore report, we hear the interesting news that product videos views are up some 40%, year-over-year basis last October. Now, that was done on a single rather small sample, but still this speaks to the increasing influence video will exert on product marketing online.

We’ve commented any number of times that rampant poor product imagery represents a consistent loss of sales for online businesses. Most retailers just end up using the low-quality, “ordinary” images provided by the product manufacturer. Rare is the retailer who invests the money to re-shoot product with an eye towards improved presentation on the web.

Certainly, replacing or augmenting poor product images with product videos can help sales. Is it better? Yes! Particularly if the video has greater quality than the manufacturer images it replaces. Is it enough? I doubt it. To get to the next level where the video has a substantial impact on sales, there must exist a certain persuasive quality to the video, and you don’t get that by simply running stills together at 30 frames per second — no one would claim the typical YouTube video to be on a par with the work of Hitchcock, Kubrick, or Fellini.

Now, no one is expecting retailers to win Hollywood awards for their product videos, but quality video production is waaaaaay more complex than quality still image production. It has to be scripted. Do you use a voice over? Is it a male voice, or a female voice? What about using a model — do we go with the hot one in a bathing suit or with Average Joe Everyman? What’s the ideal length for this sort of product and audience? What will the calls to action be? Think about your typical product showcase on QVC or HSN and how much effort and time go into selling each product.

Technology like EyeView are springing up to measure video analytics (hmm, “vanalytics”, anyone? Too risque?) and even test it. But this, too, begs the question: are consumers even trained that they can click within video? (YouTube certainly seems to think they can be trained). So low early conversion rates may be ok, but give consumers a year or two and those clicks will be up significantly.

Where does this go next to get to this higher quality level? Videos can be used to show product in new, more revealing informative ways such as this sort of 3D imagery by Ortery, which revolves around a product, taking a series of stills, and then automatically creates Flash video of the product ready for upload. How about testimonials, perhaps by creating a product-specific “home shopping network” for one particular product? Imagine having Bazaarvoice integrating customer video testimonials directly into a longer, fuller product video.

What do you think?

P.S. If video and commerce interest you, then don’t forget to subscribe to my friend Xavier Casanova’s blog, VideoRetailer.org which covers the intersection of video and commerce.

Right Now Is Always The Right Time

John Quarto-vonTivadar — Mon, 19 Jan 2009 20:30:22 +0000

[A personal story with a business ending]

Over New Year’s, I went on a dance cruise to Mexico — a group of dancers goes on a regular cruise ship and effectively “takes over” the dancing, especially late night — and everyone (including non-dancers) ends up having a great time. We’d planned this trip back in the summer, with my two wacky cabin-mates absolutely insisting that we upgrade to a Junior Suite something-or-other which had a lot more room and a balcony. This was August, “pre-Recession” for readers with short memories, and so splurging seemed like a good idea.

Despite FutureNow’s having a record quarter ending in September, by early November my more practical side was fretting we should’ve just gotten a regular inside cabin with the fake porthole. Or maybe even save some money by not going at all. I may well have chickened out but The Hummingbird and Dr. Evil (right) reminded me everything was already paid for and non-refundable so there was nothing to do but relax and enjoy it.

You know what? I had a blast. Wouldn’t have traded it for anything.

And not because of the balcony but rather despite the balcony. It was the people I was with that made the experience into a rocking-chair tale fifty years hence; in fact, a fake porthole might even have lent further spice to the story. [Almost at the business part!]

As I’ve thought about the trip the past week, this concept of remembering to live in the present came back to me again and again. How many of the hundreds of people who just stood and watched for hours from the railing wished they’d learned more from their Aunt Ethel’s b-day gift of 6 lessons at Arthur Murray’s so they might’ve joined in? What about the singular, nutty fellow who despite his lack of dance lessons joined in anyway and ended up trading an appetizer of “looking foolish” for a main entree portion of “priceless”?

Are you too busy to take a dance lesson? Or walk your dog? Or take your Mom out for ice-cream? Or – let me segue this to my business theme — work on some project you’ve been putting off, say, increasing your conversion rate? or getting started in testing? or revamping your website?

Recently, I’ve been listening to some Alan Watt’s Zen-ish podcasts and he talks about one translation of the concept of nirvana being “to exhale”. Which is to say, literally and figuratively, action comes when we “breathe out”.

Are you holding your breath waiting for the Recession to end before you try something new? You’ll have suffocated long before then. Exhale, and start acting now to change your company’s situation.

Are you just waiting for your Dev Team to have enough free time before they get around to improving that shopping cart? They’ll never get to it, unless you stop thinking about it and start doing something about it. Exhale, and start.

Not enough budget to start testing and optimizing? When have you ever had enough budget for everything? Stop worrying about it and exhale. Right Now is always the Right Time.