FutureNow's GrokDotCom / Marketing Optimization Blog

The Ultimate A/B Test

John Quarto-vonTivadar — Wed, 30 Nov -0001 00:00:00 +0000

A newer client of ours asked a great question this week: "How about we create a completely new site from scratch using Persuasion Architecture® and then A/B test it against our old site -- What would happen?" First off, let me answer the second part of the question first: The one thing I'm quite certain of is that humans are awful at predicting the future. If someone tells you they can know what's going to happen in the future put it to the test --I've got one buddy who used to claim that he could often predict the future, albeit only about 10 seconds forward. "Great!" I told him. Let's go to Vegas, and I'll put up the money and we'll play craps all weekend. That's surely a fast enough game that your 10 second limitation won't be important!" We never did make it there; somehow it was never the "right time" to predict the future, I suppose. But when you think you know what will happen next, hold yourself and your team to a hard objective measure: TEST! But of course, that's what our client really meant with his question this week. The actual question he was asking I might re-phrase as, "is it more efficient to A/B test large scale changes by jumping into a vastly improved architecture first and then proceed with incremental improvement of that new architecture?" As your instincts might indicate, this can be an ambitious way to jump-start optimization efforts, under the right circumstances. Most sites have a legacy architecture that actively works against persuasion and conversion, and if your pockets (and your nerves) are big enough, the ultimate A/B test is to say, "Hey, what we have now is a Control benchmark. What we really want to to make sure that the major persuasive scenarios are aggressively planned to be ready for optimization and then go from there." You'll note that I'm emphasizing the persuasive planning portion because just throwing a new design up isn't likely to leave you in a position to learn from future optimization as efficiently as possible. Instead the purpose of the newly revamped site is to accept the old site as a benchmark to measure against, and to put in place the series of key performance indicators and measurement points which will be used in ongoing optimization quarter after quarter, using experience and insight, such as with OnTarget™. There are some additional issues that should be thought through of course: First off, the technical implementations that will continue to send a portion of your traffic to the old site. Second, make sure you have your apples and oranges understood so that you compare the correct new analytics with their counterparts from the old site's analytics. And third, don't forget that you're not testing in a sterile academic environment where the goal is to understand all permutations completely, but rather in the living, breathing biodome of a company with live customers and a live income stream. As improvements are verified more and more traffic should be assigned to those improved points of conversion and persuasion -- much like the medical profession where the adage "first, do no harm" is the rule. Personally, I have to give this client props for the sheer nerve of bringing this topic up. It shows a confidence in his team's ability to implement and live with change -- "the only certainty is change" -- as well as a willingness to quickly say "ok, we were wrong in this aspect, so let's analyze/test/optimize yet again". I think this is going to be a fascinating long-term experiment. What about you?

Great Technique Is a Habit, Not a Happenstance

John Quarto-vonTivadar — Wed, 30 Nov -0001 00:00:00 +0000

Have you ever wondered why ballet dancers look so elegant when they move? For the longest time, I always assumed it was because they were, well, ballet dancers (I know, that's circular reasoning) -- but really meaning "just born graceful, a natural dancer". Do you think so, too? As a swing dancer, I noticed that whenever someone new showed up at the Gotham Swing Club dances you could spot right away if she had a ballet background. Even though she might not know how to swing and made all sorts of mistakes, damn she'd make even the most common of mistakes look elegant. And it didn't necessarily mean she picked up swing any quicker than anyone else; she simply looked good while moving through the novice ranks. After talking with some pro ballet dancers, as well as several Rockettes, about how their job works, the one common element that I heard was that they spend several hours every day working on their basics. The same basics they learned on Day One when they were, like, 5 years old. They would never consider to not work on their Basic technique every day. That's why they look so good when they move. Their core technique is so embedded in their body that they cannot help but make it look great. Their good technique has become a habit. Their good technique doesn't derive from what they are; it derives from what they do. Do you to work on your core conversion techniques every day? It's easy to forget and to fall back into bad habits, as I witnessed recently in a group of conversion professionals ! One member of the group had wanted the rest of us to vote for one or another nominee for some award (the details don't matter). He didn't get a great response rate so he used one of the most awful techniques possible: he created a shocking Subject line for his follow-up email, sure to get everyone who read it to open the email: "Microsoft buys Adobe for $24.6 billion" which of course is not only false but also has nothing to do with the nominee award balloting in question. I knew right away that this is a fellow who doesn't have his basics locked down far enough. He may be consciously competent in his regular works-- he likely does fine work when he thinks about doing fine work -- but there's no way he's unconsciously competent, doing great work even when he isn't trying. He's not practicing every day. You may be thinking, "What the heck is wrong with a subject line like that?" Well, to start, companies who use such a technique rarely increase their conversion rate. Oh, you most certainly can increase your Email Opened rate 'cuz of the catchy subject line; it's not hard to craft a Subject line that induces "opens". But the moment people realize they've been bamboozled -- that the implicit promise of more info about the fictitious Microsoft-Adobe merger are not forth-coming -- your chances of converting drop down to the pre-subject-line levels, or worse. You haven't spoken at all to what's-in-it-for-me to induce incentive to convert to the *actual* call to action (in this case, "vote for a nominee"; in most Grok readers' cases, "Buy My product", "Fill Out My Form", etc). Or, you may be thinking, "Well, in a group of Pros, it's ok to slip a bit on the Basics, cuz you all know them". And that's just the point. A Pro doesn't ever "slip a bit" on the Basics because, just like Pro dancers, they cannot do so without intent. Their core technique is so deeply embedded that slipping becomes a matter of conditions beyond their control: and writing the subject line of an email is completely within one's control . So if you find yourself "slipping a bit", you should consider that maybe you haven't been working on your technique regularly enough. Great technique is a habit, not a happenstance. What have you worked on, today, that better clarifies why your customers ought to buy from you?

AB Testing: Too Little, Too Early?

John Quarto-vonTivadar — Tue, 21 Mar 2006 22:49:13 +0000

Lately, everywhere you go analytics industry folks are talking about AB Testing. That’s a good sign, since it means the industry is focusing on an overlooked leverage point in their web analytics investment.

But as so often happens, achieving full buzzword compliance has become the goal rather than the means; what lies behind the words is often lost. In this case, “AB testing” – the buzzword – has become a euphemism for plain old “testing”, which, like ordering liver on a first date, may be good for you, but is certainly not sexy. But throw some “AB” in front of “testing” and your dour liver is magically transformed into paté de foie gras.

This is a bit disturbing, especially when you hear people sprinkling the “AB” condiment to add flavor to anything from a focus group (“Hey, did you AB Test the response to the new company logo?”) to the mundane (“Suzie’s lamp is out, can you AB Test the light bulb?”) to the painfully comical (“Honey, let’s AB test the Lord of the Rings Director’s Cut with the Wide-Screen edition!”).

Mixed in there, perhaps lost among the cacophony of buzzword hype, are the ingredients to some real AB testing and with it a future vision of how to achieve its true objective.

What is AB Testing?

AB Testing is based on a simple principle that we’re all familiar with:

compare and contrast alternatives;
based upon measurement, act accordingly.

Let’s say we want to determine whether Nolan Ryan is a better baseball player than Homer Simpson? How should we proceed? First, we might set a metric for what we mean by a “better” baseball player. We can measure evidence in concrete ways, noting the two subjects’ different batting averages or RBIs or the like. What we’re searching for is the right metric—a formula that would lead us to a correct decision. Such a formula is more precisely termed a “fitness function.”....
______________________________________________________________________________

This is the begining of a 14 page paper John wrote recently about AB testing. We're still not sure how we will be distributing it but if you are interested in reading it then email me ( jeff at futurenowinc.com ) and I'll send you the PDF.

The Great Debate: or "When All You Have is a Reporter, Every Analysis Looks Like a Nail"

John Quarto-vonTivadar — Sat, 15 Apr 2006 08:05:26 +0000

In the Visits/Visitors debate, side with Unique Visitors

by Howard Kaplan and John Quarto-vonTivadar

Analysis doesn't happen in a vacuum. And as we've said a thousand times: you can torture the numbers to confess to ... uh, we mean, rationalize ... just about anything. Matt Belkin of Omniture blogged recently about the differences between Visits and Unique Visitors as they relate to measuring reach and as they factor into the Conversion Rate formula. Matt argues that Conversions per Visit is more important than Conversions per Unique Visitor. We occasionally see our own clients make this sort of slip-up, so it's worth examining the merits of the argument. Read the rest of this article. Read the entire newsletter: Volume 128

Your Unreal Conversion Rate

John Quarto-vonTivadar — Thu, 16 Nov 2006 23:54:00 +0000

I'm an avid reader of Avinash Kaushik's blog, Occam's Razor, and I quietly anticipate each new post. In general, Avinash is incredibly insightful, thought-provoking, and just fun to read.

But as a fan and follower of his work, I was surprised by many of the statements in his recent post, "Measure the Real(tm) Conversion Rate & Opportunity Pie." It’s almost as if Occam cut himself shaving.

It’s possible that Avinash didn’t have time to examine every angle—especially since he was putting presentation slides together for London at the time, and we all know how that goes—so I’d like to address the more dangerous and misleading conclusions that can be drawn from so-called “Real™ Conversion Rate” thinking. The article first caught my eye when it described, among other things, how to "…improve your Conversion Rate in 10 minutes by doing nothing more than applying simple math." Sounds fantastic! After all, here we are as an industry spending tens of millions of dollars measuring marketing efforts that cost tens of billions of dollars when, all the while, a solution needing less time than a cup of coffee sits under our collective noses. “This will be great,” I thought. The problem with the argument presented in the article is that it strikes me as a marketer's version of the Anthropic Principle: the notion that the universe is the way it is because we're in it. Such scientists are that the forces of Nature are balanced just-so because every other variation would not allow Life (i.e., us) to exist to observe it. Applied to marketing and analytics the message becomes: "We're professionals who toil to effectively persuade. All those industry-typical, outrageously-low conversion rates can't be our fault because we work too hard, so there must be something wrong with the math." In other words, let's fit the data to our expectations, rather than the other way around. This fascinating approach to improving conversion consists not of converting more traffic, but filtering out what qualifies as “total traffic.” I like to call this our “Unreal™ Conversion Rate.” It's kind of like deciding the crime rate is too high, and therefore the solution is to make murder legal -- thus lowering the crime rate by simply not counting an entire class of crime. Of course, our government does this every day in under-reporting the budget deficit when it uses current Social Security receipts to offset current year expenses, despite the fact that those receipts represent future debt still to be repaid. Ah, but I digress. The same logic was applied to women’s dress sizes in the U.S. when the Department of Commerce withdrew the standardization of woman’s apparel in January, 1983. It’s no secret that American women—and men, and children—have gotten heavier since then. So, in a blatant appeal to vanity, a “perfect Size 6” became a “Size 4” or lower! Today, there are even negative sizes for our svelte ladies. One might otherwise conclude that the thin have gotten thinner. Yet, sadly, thinking thin doesn’t quite do the trick. The article then states something spot-on: "The fastest way for you to improve your conversion rate is to figure out what is the number of people who are in play for even remotely being converted.” Yes, I totally agree. But then the piece goes on to describe three suggestions for achieving such improvement and, in so doing, demonstrates the H.L. Mencken witticism that "For every complex problem there is a solution that is simple, neat, and wrong." #1 Bounce Rate An interesting approach on bounce rate, defined in the article as "…the minimum time that someone has to commit to your website, just ten seconds, for you to even have a chance of convincing that visitor of anything." At first, this sounds reasonable, at least until you start thinking a bit deeper of some actual scenarios in which it would be in play. The first time someone comes to your site, it may absolutely be true that they do a 10-second-ish scan of your homepage. But what about regular, repeat customers, the backbone of any successful business? When you go to Amazon.com to buy a book, do you stare at the site for 10 seconds, waiting to be convinced? Or, already familiar with Amazon's fairly standard navigation, do you instead quickly type in the book/DVD/gadget you were actually looking for? In fact, wouldn't a repeat customer spending more than 10 seconds at a familiar interface indicate something were wrong rather than right? And wouldn't this suggest several other insights? For instance, there are certain situations where the bounce rate is an indication of something wrong in the selling process rather than customer disinterest. Further, that there are some situations where the mental state of the prospect indicates they know precisely what they want, rather than just idling around. These two points combined suggest that there are any number of scenarios and states of mind of the personas visiting the site—all of which must be accounted for in fulfilling a full spectrum of customer needs and goals. It seems unreasonable that we can infer intent based on bounce rate. (Actually, I'd assert that we can't infer intent by any analytic-centric approach; we can only use analytics to measure the efficacy of modeled intentional paths. But that's a different discussion altogether). So, this huge discount of10-20-30% of traffic caused by "bounce rate" is really just a fudge factor so we don't have to feel as bad for not really planning out complex sets of scenarios. #2: “(If you use Web Logs) Filter out search bots, image requests, 404 errors, website monitoring software ‘visits’ etc.” To be frank, this advice has been around 1999. The article claimed that a conversion "realization dawned" thereby shedding some insight. If you're still mining your own web logs and don't know to filter image requests, or still using software that doesn't filter it automatically, then you don't need a filter, you need a good hard kick in the ass. Maybe it's time to splurge on indoor plumbing, or a color TV. Then again, if you're still doing all that AND having industry-standard conversion rates of about 2.5%, then what does that really say about the analytics industry in 2007?

#3 “Use Customer Intent” This point starts out so well; so delightfully, painfully well: "One of the biggest mistakes business[es] make is thinking that every visitor to the website is fair game.” I just knew the next statement would be something insightful, such as "not every visitor is there for the same reason,” or "visitors might be motivated to achieve different things on your site" or something along those lines. Instead, we are treated to a stereotyped car dealership analogy that belies a fundamental misunderstanding of the sales process. What's next, ‘All Democrats are Commie Pinkos?’ or, ‘The darker the berry, the sweeter the juice?” You can read the example yourself in the article, but the gist is that the big evil car salesman is going to do every unethical thing in his stereotyped book to trick you into buying a car. And you, the noble visitor, have loftier goals that none dare call conversion. So close to making some headway, the article continues: "not every visitor ... is there to buy" (true! true!), followed by the disappointing "not every visit ... is an opportunity to convert" (false! false!). In fact, your visitors are there precisely to convert as long as you realize that in order for your business to achieve its goals, the customer must be allowed to achieve her goals first. Funny thing is, even if marketers and site owners don't get this, most salespeople do—which is exactly why they modify how they sell to match the customer buying process. There are countless ethical, insightful and financially successful car salesmen who know how to sell to folks who just happen to walk into the car dealership looking only for information. Their hush-hush trade secret: they give them only information. They establish rapport and build a relationship by giving the customer what she wants at her particular point in the buying cycle. And by doing so, they create the state of mind in the customer to come back later in the buying cycle when she is truly ready to purchase. I think that the fundamental misunderstanding of the article is best summed up in the following paragraph where it states "Using Market Research or Website Surveys or other methods, attempt to compute why Visitors come to your website.” Wait, you're gonna COMPUTE why Visitors come to your website? If you don't already know why people come to your website, what will "computing" it do for you? I can compute the path of the earth around the sun, but does that tell me why gravity works the way it does? I think not. And what self-respecting business owner needs Market Research or Surveys to determine why people come to the site? To be sure those things can give insight into why people don't come to the site, or why they come to the site and then leave too quickly (after all, that's the proposed benefit to using Bounce Ratio). But if you're in business and you don't know why people come to your site, then you're in a heap of trouble, and a survey ain’t gonna save you. There's no amount of analytics crunching that is going to tell you why people come to your site. Rather, one must do the hard work of putting oneself in the visitor’s place and empathically plan the persuasive scenarios in which they come to the site. Then, aggressively use analytics to measure how well you've implemented those scenarios; THAT's the secret to significantly higher conversion. Otherwise, we’re merely forcing the numbers to rationalize our own poor planning. A “Real™ Conversion Rate” might help your company’s self-image, but it certain won’t improve your bottom line.

Mashable: "MySpace is Better Than Porn"

John Quarto-vonTivadar — Sun, 22 Apr 2007 23:51:42 +0000

Mashable's headline got my attention, too. You can read 'MySpace is Better Than Porn' for yourself but here's an excerpt:

"Well, almost. The Economist (via Computers.net) has pulled up the Hitwise stats to show that social networks are about to overtake sex sites in the US any day now. The metric being measured is percentage share of site visits, with % visits to sites like MySpace, Bebo and Facebook on the increase, and % visits to porn sites going down (no pun intended). "

The post offers more information and a nice chart, but it still left me wondering.

Top 10 Tips for Selling "it" on eBay

John Quarto-vonTivadar — Mon, 16 Jul 2007 09:10:20 +0000

For nearly a decade, companies have hired Future Now to help them understand how people buy online. In this same amount of time, I've spent a good chunk of each paycheck ~~bidding,~~ ~~saving money,~~ doing "market research" on eBay. And, throughout the years, it continues to amaze me how few eBay sellers get it right. One little-known but poorly-kept secret out there in vendorland is that many big companies -- the same ones who come to us for retail advice -- use eBay to dispose of returned, open-box, or otherwise retail-disabled inventory. Did you know that? So it dawned on me: Here I am, an experienced buyer -- who better to ask than me about what makes me bid, bid, bid? Want to SellItNow™ your way to increased eBay sales? Here are some guidelines so you'll be able to sell like the pros (and by pros, I mean folks like these, not just eBay PowerSellers). 10 tips for persuasive eBay listings: #1) A Sticky Headline -- If you can't write a strong headline, you might as well not bother. It's your only hope for getting anyone to ever see what you're selling. (Don't forget to test your headlines.) #2) Better Product Images -- Having better-looking product images than other sellers will do wonders. In fact, 83 percent of eBay shoppers skip listings without images, while sites with galleries get 15% more activity and those with so-called super-size photos show a 24 percent spike in sales.The better photo wins every time. Consider this photo, for example. And remember that lighting control is essential, as well as these two other points about product photos on eBay:

Place a product image by the headline. It's the best way to grab attention to your headline. Remember, you're trying to slow the bidder's eye as she cans hundreds of similar listings. It costs virtually nothing to add a photo by the headline, you'll get way more click-throughs, and it simply looks more professional. If you don't have $0.35 for this critical feature, you'll never get my attention. Don't be penny-wise and pound-foolish!
Show multiple views with close-ups. This article makes the point, so we won't repeat ourselves. It's especially important to show multiple views and close-ups while exaggerating product flaws (see tip #5).

#3) Outstanding, Original Copy -- If you're tempted to just cut-and-paste your way into persuasive sales copy, forget it. Your words matter. First of all, it's obvious when sellers just use the same boilerplate copy from the manufacturer's website, which may not even be good to begin with, that everyone else is using. Besides, using the manufacturer's copy implies to me that the product is brand new and untouched. Show some personality. Showing personality helps potential buyers to see you as real; it builds trust. Why did you buy this product in the first place? Why are you selling it? Have you sold any of these items before? Different people buy in different ways, so the words you choose, and how you choose to dispense them, are everything. Start with spontaneous, emotional copy at the beginning, then get more methodical toward the end when listing product details. The second half of this article offers good advice on how to write for different personality types. #4) So, What's the Catch? -- Why do you have such a good deal? Oftentimes, the most persuasive thing you can do is to be completely transparent about your business model. Are you making tons of money by selling digital cameras in bulk? Did you buy too much for your brick and mortar store, and you're selling the overstock? Is your wife making you sell the XBox before you get a Nintendo Wii, so you've set a low reserve just to move it (this happens). I want to know. Sorry, but saying "Lowest price on eBay, guaranteed!" means nothing. #5) Exaggerate Flaws -- This one may seem counterintuitive for the novice seller but it makes perfect sense and the best eBay sellers do it masterfully. If there's a minor scratch on that DVD player you're selling, zoom in on it enough in a separate photo to the point where it seems ridiculous that you're apologizing for it in the first place. Overestimating flaws builds trust. Trust is what makes people bid. #6) Accept PayPal -- It's been the eBay gold standard since 1998. If you don't accept it, you're not making things easy. I'm skeptical. In fact, offer as many payment options as possible. Get the cash (GTC)! #7) Know How to Price -- Hear about all those folks trying to sell iPhones on eBay for $1,000? How's that working out? It's not. In the world of eBay, your competitors are two clicks away, at most. Overestimate the market for your product, and you'll never get that crucial first bid. Here's how it's done:

Be aware of what it's selling for elsewhere. Find out how much similar products are selling for and you'll have a good idea of what you should expect to get for it. Then...
Take the expected winning bid price and cut it in half. Shave off another 10% off, and you've got your reserve price. Think that's crazy? Too low? Guess again. Studies have shown that bidding is what drives up the price. The more competition among bidders, the higher the winning bid. The only way to kick-start the bidding frenzy is by putting the floor well below the ceiling.
If you're willing to accept a price that's around or below where it's selling elsewhere on eBay, list that price as the "Buy It Now" and throw in "free" shipping. Now you've killed two birds with one stone. You've created a compelling offer and you've avoided the stigma of "shipping rape" (see #10). Free shipping is often the #1 driver when it comes to online promotions.

#8) Link to the Owner's Manual -- (Where applicable) link to the owner's/user's manual/instructions for your product. Feel free to borrow product details from here, just don't use the manufacturer's boring words ;) Just don't use this tip as a substitute for tip #3 above. #9) Have a Star Rating Above 99% -- If you have a star rating below 99%, that means you've upset too many people for me to feel comfortable buying. Sorry, but if you have a 100% rating and you've sold to less than 100 people, I'm still not confident; it's not a true 100%. If you've sold to thousands of people and have a 98% rating, your "success rate" means nothing. #10) No Shipping Extortion -- Last, but not least, some eBay sellers lose their minds when it comes to shipping. Do you think we're stupid? Um, no, it doesn't cost $15 to ship from a one-pound package from Kansas to Brooklyn within 10 days. If it costs $5 to ship it from China, why must I pay $25 to ship it in "4-6 weeks" from California? And, by the way, I'm receiving the package, so I often how much you've paid the very moment I get it. Try this, and the only one you'll fool is yourself. eBay sellers: I've still got one last free corner of space in my apartment. Please help me fill it with stuff! The quicker it fills, the quicker I'll clear it out by selling on eBay and then have all sorts of free space to fill up with new eBay purchases! Do you have any tips to sell more effectively on ebay? Update: Seth reminds us people are irrational.

Fight for Kisses

John Quarto-vonTivadar — Thu, 27 Sep 2007 18:06:02 +0000

A geek buddy passed along a link to an interesting video clip today, commenting on how it was "such good marketing". Now, anytime I hear a techie say this, I'm pretty much assured it'll be *bad* marketing -- or no marketing at all, just a good advertisement. (Hell, that's a whole other post.) Anyway, human nature being what it is, I perked right up; everyone likes to gawk at a car wreck. I don't want to say much specifically until you've had a chance to watch this yourself -- am I'm helping to make it viral? -- but give a look-see and let me know your reaction. Don't read any further until you watch this video:

(If video doesn't load, click here.) OK, now you've seen it. This brought up a few points for me: First, it felt a little longer than expected, didn't you think? I figured once I saw the ad for the Quattro -- a name I'm fond of for obvious reasons -- we were near the end. But no, it continues for some time past that. And the ending gave me a bit of a surprise, as I had only a hint of a feeling we were dealing with a game. Second, even at the end when the game nature became clear, I kept thinking Shick Quattro was probably just a commercial sponsor. Third, I actually went through the bother of going to the main site for this beast, ffk-wilkinson.com. Other than commenting on how extraordinarily painful it was to wait for this site to load -- close to 2 minutes on a broadband connection!? -- it turns out the razor itself is one of the characters in the game. So, Quattro isn't just a commercial sponsor of a game; it seems Shick actually produced the game as theater for showing off its product. Well, that "shortened my leash" on how much I'll allow this game to shave minutes off my valuable free time. Finally, let me defend my geek: The clip was entertaining. But if marketing were entertainment, every day would be the Super Bowl. The funny thing is, although I'm curious to see the game in play, I'm not terribly excited about the prospects of playing it. What about you? Are you interested in this product, or were you simply entertained by the video clip? Do you have a passion to fight for kisses?

"No, but I did sleep at a Staybridge Suites last night... "

John Quarto-vonTivadar — Fri, 02 Nov 2007 17:18:09 +0000

Often we relate stories here on Grok about conversion missteps or persuasion challenges facing companies on- and off-line. Today, I'd like to relate a success story! An Open Letter to Andrew Cosslett, CEO of InterContinental Hotels Group: Dear Mr. Cosslett, As a business traveler, my needs are simple and predictable: I just want some restful sleep, power outlets numbering more than one, and an internet connection that works. Now, in the last 90 days, I've spent more than a third of those nights in a hotel room, so I'm waaaayyyy too familiar with lumpy pillows, concave mattresses, and TV remotes where the previous guest felt entitled to liberate the "free" AA batteries inside. On a recent trip out to the Googleplex in the San Francisco Bay area, I found myself searching about for a quality place to stay. The usual spots had no vacancies, so I chose one of your less commonly known properties, Staybridge Suites. (I happen to love staying at "suites" hotels; the visual separation of a living area from the sleeping area, and an actual kitchen, creates the feel of a condo that a human lives in rather than just a hotel room.) I check in, no problems. It's actually *half* the price of regular hotels in the area. I'm pleasantly greeted by staff and quickly finding my room. But once inside, I'm delighted to see this:

That's right, a hand-written note from the general manager, Ms. Lisa DeLorean. Not a computer-written-in-handwriting-font note, but a real, live, ink-on-quality-stock note. I wasn't even terribly concerned about the words themselves -- the note's pleasant enough -- but this fine business manager took the time to write that note herself, and addressed to me personally, so I know it's not just the boilerplate greetings that tells you the name of the cleaning staff. It actually took me awhile to read the note, as most of the "wow" effect came from just receiving it! Of course, she thanks me for choosing her hotel, but she also thanks me for all the other visits I've made to the affiliated chain members (Crowne, InterContinental, etc., none of which I suspect factored into her bonus those past years), and then she finished with a bang [emphasis mine]: "We want you to be very satisfied with your stay." Not just satisfied, but very satisfied. And I was. The place was indeed restful, power outlets everywhere and free internet. And, yes, fresh batteries in the remotes. I'm sure you've all heard the stat that a dissatisfied customer tells, on average, 12 others about their bad experience. (Well, Lisa DeLorean, I just told 85,000+ GrokDotCom readers about you, your fine hotel, and the classy way you treat your customers. Keep up the good work!) Mr. Cosslett, as CEO of Lisa's parent company, if this handwriting of thank-you notes is corporate policy, congrats to you too! If Lisa did this on her own initiative, you just found your next regional manager. Cuz if you don't, I'm sure another hotelier will snatch up talent like Lisa's -- and fast. Sincerely, John Quarto-vonTivadar, delighted customer P.S. -- At the end of my stay, I tracked down Lisa DeLorean in the manager's area and thanked her for the note. Curiously, she was taken by surprise, and expressed that no one's ever thanked her before for writing these notes and (get this!) she was beginning to doubt if they made a difference. Chin up, Lisa, they most certainly do. If any readers would like to stay at Lisa's facility, here's the 411: Lisa DeLorean, general manager, (650) 588-0770 -- Staybridge Suites, at the San Francisco Airport, 1350 Huntington Ave, San Bruno, California [Oh, and by the way, I have no financial interest in InterContinental Hotels Group or its affiliates, nor do I know Andrew Cosslett, and I never met Lisa DeLorean until the events related in this story.]

Testing Add-to-Cart Buttons: Stuck in the Middle With You

John Quarto-vonTivadar — Fri, 25 Jan 2008 19:08:48 +0000

Bryan walked into my office the other day to point out an interesting item found while surfing: a left-sided add-to-cart button on a product detail page. We chatted back and forth about the conversion issues involved with placing it there -- and in fact, one of our Conversion Analysts, Peter, commented on this very topic in his latest post -- but soon our conversation turned to something much more interesting than left-sided calls to action: the testing of left-sided calls to action. "Do you think they tested it?" Bryan asked. "Hmm, the Joker in me wants to say Yes, but I'm guessing the money bet is No," I replied. Now, that's not because Crutchfield doesn't test. In fact, I've no idea at all what sort of testing culture Crutchfield nurtures; I'm just saying that in our experience, only rarely does this sort of innovation ever come about from testing. Instead, it's sadly de rigeur for it to arise from a designer wanting to try something "different", or an IT staff that doesn't perceive one shopping cart as different from another, or maybe Matilda the Intern just forget an HTML tag. Anyway, the point is to go with the simplest explanation -- which, in 2008, is that most companies still don't test. "I think you're right," Bryan continued, "cuz if they did test it, it probably wouldn't do well." "Maybe some Clown in IT or Marketing just wanted to be 'kewl'." Here's what we're talking about, as shown on Crutchfield.com:

Intuitively, I hope you'll agree with us that right-sided feels like a better than even-money bet (though that in itself is a reason to do a test) -- but what's the point of leveraging your intuition to be "directionally correct" unless you eventually try to back it up with some evidence that you're actually correct? That started me down the road thinking about how to actually test this hypothesis. (I can be wordy, so if you've lost the trail of thought, the question is, "Which converts better? Right- orLeft-sided Add-To-Carts?" and the hypothesis would be, "Right-sided Add-To-Carts convert better than Left-sided Add-To-Carts.") Here's where it gets interesting: The supposition is that most Web surfers are so used to right-sided Add-To-Carts (and right-sided Calls-to-action, generally) that a left-sided one is bound to produce some cognitive dissonance. It might not be consciously noticed -- less so on "narrower" sites and more so on wider ones -- but the placement on the left will "feel" odd. With that in mind, just how do you go about running a test you already know has a skew to it? How would you really determine whether the Clowns or the Jokers win The Great Add-To-Cart Positioning Debate of Aught-Eight? Here's what I would do: First off, start with the most obvious test, because we have to get a quick benchmark of just how far Clown is from Joker. Throw some percentage of traffic at the left-sided Add-To-Cart -- enough for some statistical significance -- and see just how well Right does vis-á-vis Left. (The fascinating thing about intuition is that a fair percentage of the time it's fabulously, gloriously, achingly, wrong -- and if this is one of those times, better to find out early and move on to the next good idea.) Assuming we've shown some evidence of the skew in favor of right-sided shopping carts -- otherwise, why continue reading this post? -- how do we go about removing the skew that comes about from people being "trained" that right-sided is "normal" to answer the real question: If folks weren't biased by convention, which side converts better? To do that, what you'd really want is to look among your customers who've already successfully converted using one particular side and to present them with similarly-sided add-to-carts in the future (hmm, might have to set a cookie!), so you can gauge what the conversion rate is for people who've shown at least some indication that they can successfully convert.** The idea here is that, all else being equal -- something the pre-existing bias hurts -- the true question should be, "Do people actually have a preference for sidedness at all"? By picking only from those who've successfully converted previously, you're making a first attempt to say, "Hey, at least these folks don't seem to be impeded by a systemic bias"; therefore, those who buy consistently using left-sided calls to action might then be expected to convert at approximately the same rate as those who buy consistently using right-sided calls to action. "And surely," you might argue, "those who show a preference for left-sided add-to-carts should convert better when consistently presented with left-sided add-to-carts than Right-Siders who are suddenly presented with a left-sided add-to-cart." See, you've switched the tables. Get it? In short, you try to come up with series of tests -- a Testing Campaign, if you will -- which attempt to disprove the way your original hypothesis was leaning (we figured Right would do better, so let's design tests that indicate when Right does poorer), and let us challenge any underlying bias (i.e., that Add-To-Cartss typically appear on the Right) that gives unfair advantage. Well, those are my thoughts on the subject. What I hope you got out of that is that a "culture of testing" means thinking as deeply about the design of experiments as it does their performance. I'd love to hear more about you. Are you a "Clown" or a "Joker"? Or are you just "Stuck in the Middle"? Would your brand loyalty or the customer's familiarity with your site's User Interface simply override any preference you have for being a Clown or a Joker? - - - - - - - **A few readers will feel reassured to know that, in actuality, you'd still send at least a few visitors who preferred one Side to see an opposite-Side call-to-action once in a while just to keep things honest; enough to get insight from the data, but not enough to cost the company too much from the loss from the expected conversion differential. I figured I'd say that as a footnote before some Sharp Tack out there writes in to scold me. ;) [Author's Note: What's with all the Clown and Joker references, you ask? From the song "Stuck In The Middle With You" by Stealers Wheel (c.1973), comes the lyric "Clowns to the Left of me/Jokers to the Right/Here I am/Stuck in the Middle with You." I was bound and determined to get that song into a post sometime this month, just to stop humming it in my head. There. Now it's your problem. :) ] [Editor's Note: Want more profitable ideas on how to beat assumptions with better testing? Take a look at our free website testing resources, including John's A/B testing white paper.]

My Cup Runneth Over from High Slurp-Factor™

John Quarto-vonTivadar — Thu, 07 Feb 2008 16:22:51 +0000

Have you tried Arizona Iced Tea? They aren't bad at all, and I've really taken a shine to the No-Carb Blueberry Green Tea they produce. For the longest time I could not put my finger on why the product always make me smile, until last night. After so many months, it dawned on me: the containers for the green teas are overfilled. When you open the bottle, there's more product in the container than it should be expected to hold -- even to the point that if you opened it up while exerting pressure on the bottle (careful, Readers-who-Test!), you'd spill blue-ish tea on yourself. Sometimes I have to reach down and take a "slurp" off the top so it doesn't spill -- which no doubt causes the rest of the family to consider that "Q's own personal bottle" of the stuff. An interesting way to establish territorality. Back to point: I feel happy when I open this product because I feel I've gotten more than expected, and certainly more than any competitive product. Obviously that extra slurp's-worth costs them some finite amount of money, but I'm wondering if the delight I feel at getting more is common enough across their customers that it's driving more sales than the cost of the slurp. And if I feel good about a product, I buy it regularly, and therefore my slurps and the slurps of my fellow... Slurpers represent significant lifetime value to the company. Does your company's product or service delight customers more than they expect?

PayPal Should Go Undercover

John Quarto-vonTivadar — Fri, 29 Feb 2008 18:29:58 +0000

PayPal recently announced a streamlining of its payment flow process that doesn't require a PayPal account to use. In other words, you can "check out" via PayPal, reap the security benefits of the merchant store not knowing your financial details, and pay for your item without having created any long-term relationship with PayPal (although they wouldn't mind). Adding PayPal to an e-commerce site can sometimes result in lower conversions -- which makes sense because you're being taken away from the experience you were just having at the merchant site. On the upside, some mid-sized UK merchants using this new process are reporting an increase in their monthly total payment volume, with gains of over 9% on average. But I've got a different request altogether. I use PayPal. A lot. Probably at least $500 a month of online purchases of various things that, at the time, I'm convinced I really need. It always amazes me how confusing the PayPal part of the checkout process is. First I'm on the merchant site. Then I'm off it -- but not so obviously that I notice right away. It's just a white, empty-feeling page with the merchant logo and a familiar PayPal button. Then the interface changes again to make it obvious that it's PayPal. In order to return to the merchant site, I have to click a small-font text link that competes with PayPal-branded buttons for my attention. At this point, I'm still not sure if the purchase "took" -- that confidence doesn't come until I return to the merchant site. Won't some of those e-tailers enjoying that volume increase please, please, PLEASE put just a fraction of that revenue toward hiring a bright developer to create a way to do this undercover? Its seems this could be easily resolved with a bit of (*buzzword alert*) AJAX. Enter your PayPal user name, maybe some kind of modal lightbox pop-up to asks for my password, it goes back behind the scenes to confirm this with PayPal, then seemlessly closes the pop-up and updates my status on the merchant site to say, "Purchase completed via PayPal. Thanks for your business!" I like using PayPal. I just don't want to notice it. Kinda like the electricity in my home; I just want it to be there when I plug in my laptop. What do you think?

The Price of Perfection

John Quarto-vonTivadar — Fri, 14 Mar 2008 22:21:51 +0000

Recently, one of our regular readers blogged about testing with Google Website Optimizer (GWO). In the discussion thread, a respondent worried that he may not be able to use GWO because his company's website has a database-driven content management system. He described himself as a "perfectionist" and it didn't settle well that content was somehow taken "out" of his site and hosted on Google. Further, one of his company's consultants commented to him that GWO just "isn't useful" for a complex database-driven site. First off, we can tell you from experience* that his consultant is mistaken. (See explanation here.) Secondly, everyone thinks their own site is complex. Everyone. (Just like everyone thinks their kid is cute enough to be a model for Gap Kids.) But ecommerce sites are pretty similar -- and simple. It goes something like this:

Get customer to site
Display product to customer
Help customer decide to buy
Accept her money with a thank you
Ship out the goods
Repeat

Customers don't care if what we have behind-the-scenes is simple or complex. All the customer cares about is how simple and enjoyable -- or not -- the experience is for them. Now, back to the issue of perfectionism. This fear of taking an incremental step lest it turn out wrong, even if the step is toward improvement, seems to evoke fear, dread and a certain "deer in the headlights" mentality. Ever hear the adage, "Anything worth doing is worth doing wrong"? It's a great way to think about testing and improvement of any kind, because it deals with the fact that the first step toward improvement always "feels" the hardest. It speaks to the moment when you're most susceptible to false objections like "It's too complex!" or "That's inefficient!" Let's get those first steps out of the way. Let's embrace being wrong, because we will almost surely learn some way to improve. The fact that the improvement won't be immediate or perfect just isn't a viable reason not to try. Asking for it to be perfect first and always is a perfect recipe for "never". If your company does, say, $5m/yr online and you can raise the conversion rate from, say, 4% to 5% (a 20% lift) because of your testing with GWO -- or any testing tool for that matter -- you just added $1 million ($5m x 20%) to the bottom line. If I were a CEO and found that so-called perfection was costing me $1m/yr in lost revenues, plus employee salary, I'm pretty sure I could find less expensive, less perfect employees. I wonder, just how many companies out there are paying millions of dollars a year for perfectionism? And how many imperfect employees, freed from this apotheosis, consistently deliver better results for their companies and their customers? Could this be why three quarters of online retailers don't test even though it's free? . . *FutureNow is an Authorized Consultant for Google Website Optimizer.

Video: How to Do A/B Split-Testing on Lower Traffic Sites

John Quarto-vonTivadar — Wed, 09 Jul 2008 17:00:17 +0000

Dr. Ralph Wilson of Web Marketing Today spent a few minutes interviewing FutureNow's Bryan Eisenberg about testing on sites that have little traffic. You can view the video below. You may also be interested in reading more about the hierarchy of optimization when you are done viewing the video. [youtube]http://www.youtube.com/watch?v=60xuSEc7lxA[/youtube] Bryan and I have co-authored a new book all about testing and helping you figure out what to test. It's called Always Be Testing: The Complete Guide to Google Webiste Optimizer (published by Sysbex/Wiley) and we're expecting it out next month; you can pre-order it now on Amazon.

Free Books & Signing at Search Engine Strategies San Jose

John Quarto-vonTivadar — Tue, 12 Aug 2008 20:29:18 +0000

On Monday, August 18, 2008 at 12:30pm, my co-author friend and colleague Bryan Eisenberg, founder of FutureNow and co-author of the New York Times bestselling books, "Waiting For Your Cat to Bark?" and "Call to Action", will launch "Always Be Testing: The Complete Guide to Google Website Optimizer" at Search Engine Strategies (SES) San Jose ( http://www.searchenginestrategies.com/sanjose). The event will be held August 18-21, 2008 at the San Jose McEnery Center. Bryan will be presenting the session "Pay Per Conversation" together with Brett Crosby, Group Manager of Google Analytics and Google Website Optimizer, on the 18th at 11:15am and immediately after, Bryan will be available to sign copies of his book being provided as a courtesy by Google. Get there early since Google will only be giving away a few hundred books. If you can't make it there you can always sign up for our next Always Be Testing webinar for your chance to win 1 of 3 signed copies (by both of us). Want to know more about the book? Watch the interview below or read one of the online reviews: [youtube]http://www.youtube.com/watch?v=rkYkTKcZSwg[/youtube] Click Z (Mike Grehan): http://www.clickz.com/showPage.html?page=3630480 100 Dollar SEO (Carlos Del Rio): http://www.100dollarseo.com/always-be-testing-a-cookbook-for-web-optimizer/landing-page-optimization DamnIWish (Andy Sernovitz): http://www.damniwish.com/2008/08/use-reviews-to.html [youtube]http://www.youtube.com/watch?v=MSDbafXqwgs[/youtube] Better yet, why don't you buy a copy for yourself and send us your own review to link to.