Everyone knows the question: “What one [book|person|food|movie|etc] would you take with you on a deserted island?”  That is to say, what one thing can’t you live without?

Most of us have a pretty narrow list. Perhaps it’s a book you’ve read over and over again. Or your favorite cousin — the one you would like as a friend even if you weren’t related. Or chocolate chip ice cream (I assume the deserted island has a freezer and electricity!).

Have you ever asked yourself that question about your metrics? What one metric in your business can’t you live without? What measurement of success must you have to run your business, so much so that you’d be catatonic if it went away? Even for businesses in the same category, there’s no one right answer all the time. For some businesses, it’s Conversion rate. For others, it’s Revenue. Yet others prefer a blended number like Net Revenue per Customer. How about Repeat Customer rate? And some folks cheat and come up with a conditional “well if I was selling X, it’d be this, but if I were selling Y, then it’d be that.” No matter how you slice and dice your metrics, choosing one is a heck of a choice to have to make!

Now, we all have a hard time narrowing down a favorite movie or food or book to one . Usually, we can get to two or three “essential” items and hate having to make the final cut. Fortunately, too, the vast majority of us will never actually be stranded on a deserted island, so choosing wrong or flippantly or impractically doesn’t carry a big penalty.

But imagine if you choose wrong for your business. Now the stakes are far more real and dangerous. A “favorite” now has to justify itself and its impact on the business. It can’t hide behind “well we always look at that one” ( a logical fallacy known as “appeal to tradition”) or “I went to the Conversion Conference and heard that Landing Page Optimization is the only way to improve conversion!” (a logical fallacy known as “appeal to authority”) or “ABC Widgets Inc and XYZ Widgets LLC both use this metric, and I sell metrics, so this must be the right one!” (a logical fallacy known as “I’m a lemming and can’t be bothered to think for myself”).

Your One better be, well, The One.

And the act of winnowing down, and choosing a final singular metric upon which all metrics are subservient, is a laborious, thoughtful, even painful process. Imagine a  cross between “Lord of the Flies” and “Lord of the Rings”.

I challenge you to try. If you can narrow down your list to one number, it means that all your other numbers must, per force, be useful only in supporting/refining/augmenting The One. Thinking of it that way, the other metrics you choose to be in supporting roles might well change, and give you a very different view of your business. If they aren’t supporting The One, what are they doing there?

I’m often amazed at how naive companies can be about their online testing efforts. The bigger the company, it seems, the more disconnect there can be between the talented optimization team and senior management. And that, as you might guess, ends up undermining large portions of the revenue and efficiency improvements that come from continuous testing.

This is part 1 of a multi-part column. In future columns, I’ll discuss why you want to celebrate every time you “lose” when testing — which is really what I wanted to write about. But I realized it’s important to lay the groundwork for why continuous improvement through testing is absolutely essential to your company’s online efforts. So that’s where I’ll start.

Most of the arguments I’ve seen “for” testing are made towards the marketing team, and while this is the correct audience to impact ongoing efforts, the marketers can often be too fuzzy when it comes to their own analytics and measurement. Yet, the analytics team is often overly emphasized on tools and pure metrics — and I discussed in my January 2012 article — focused on too many metrics. This column will use an example, with numbers, to review how testing equals business smarts.

Over the course of this series of columns, we’re going to develop a spreadsheet that you can use to model your own company’s efforts towards testing, each month adding more and more features to the underlying model.

This month, let’s start with the basics. By far, the most common frustration I’ve heard expressed by optimization teams is their high failure rate of tests. These are the folks in your organization who should know best that “Losing More Often” — a topic we’ll cover in a future part to this series — is actually the recipe for long-term success in continuous improvement.

Introducing Cato’s Kitchen

Let’s start. Imagine your company, Cato’s Kitchen, produces specialty customized cat food — sort of like private-labeling your own bottle of wine. Cato’s is in a tough market dominated by the big guns of the pet food industry but their concept is certainly a unique approach. And they’re small and nimble, so they can test, change, adapt, and repeat fairly often and at low cost. So their mindset is perfect for improving their online efforts through testing.

Now some numbers: Cato’s team has the bandwidth to do 10 tests a month. And each time they test, the test itself has fixed costs of $1,000 per test, mostly from the simpler tweaks to their website that often accompanies the early phases of testing: low-hanging fruit such as conversion improvement, design tweaks, better calls to actions. Every testing team must begin this way; if your team jumps first to long-term structural change before you handle the simple stuff, find another team.

How about success rates? Cato’s team wins only one out of every ten tests, a win rate of 10%. When they win, it’s worth about $25,000 to the company.

Failure rates? Cato’s team loses about six times out of ten, and when they do it costs about $10,000.  And Cato scratches (pardon the pun) about 3 times out of ten.

Our First Model

Here’s what our first model looks like so far:

Some astute readers will be asking: “Why $1,000 per test as fixed costs?” “Why is the win rate only 10%?” “Why is the average winner 2.5 times as large as the average loser?”  Good questions, all!

The Underlying Assumptions

I’m using approximate numbers based on experience. Because after doing this for umpteen years and with eleventy hundred clients, these are the sorts of ratios I’ve seen be quite common among the most successful testing teams: Their costs for implementing a given change tend to average out to a fairly static fixed amount, typically no worse than 10% of an average losing test, often much lower.

Successful teams’ win rate, while a seemingly low 1:10 on casual inspection, comes from being unafraid of being “wrong” and therefore testing more often (and again, in a future column I’ll get to why you want to test more often and to happily welcome losing tests).

And, finally, successful testing teams typically have average winners that run 2-3 times the size of their average losers.  One of the upsides to the “learning what to test” lesson from continuous testing is that your progressive improvement translates into more efficient winners. This can be reflected in positive improvements to any or all of the key metrics surrounding testing wins:  a higher average winner over time,  a smaller standard deviation among winners, and outright win rate percentage. All else being equal, by far I would prefer the first since it’s the least subject to post-test second guessing. If you start posting a series of progressively profitable wins, it’ll be hard to argue your team isn’t bringing in true value. (Or so you’d think — Ill recount a short anecdotal story to the contrary at the end of this column.)

So I’m pushing our early first draft of a model toward characteristics of teams who win. To be honest, I’m not nearly as concerned about teams that lose;  we’re trying to mimic success, not failure. You will have failures along the way, so let’s put up a goal of what success would look like and shoot for that.

The First Month

Now with these numbers in place, we can create a simple spreadsheet to see how we do after a month of testing. [You can download this spreadsheet from the link to my site at the bottom of this column]. Here are Cato’s results in the first month:

Let’s step through this to make sure we’re all on the same page. We’re doing 10 tests per month. Each of those ten tests has fixed costs of $1,000. We get one winner of $25k out of every ten tests, and six losers out of ten which cost $10k each. Residuals I’ll cover in a moment, but there aren’t any in the first month so it’s zero here. And Net Gain/Loss is the sum of the Wins and the Residuals minus the Fixed Costs and the Losses, for the month. Add all this together, and we get a $45,000 loss for the month.

Ugh! Wow, if all we did was one month of testing we’d surely think that Testing was costing us more than what we could ever get from it. But it that really the whole story?

As you might guess, it’s the Residuals that make the difference. If your team proposes and implements a test and it scratches or even loses money, you end up going back to your original  version of whatever you tested. You incur the cost of the test but only during the time when the test is running. As soon as you determine the test was not a success you switch back to your original. You might do further testing in this same part of your site — or not — but that’s a different test.

But when you have a winner, you get to permanently run that version in place of your original, so even though the costs of the winner were paid up front, you get a “residual” impact, at no additional cost, to your efforts month after month into the future. I’ll repeat that: that is permanent, at least until you find an even better winner or underlying circumstances change. So Cato’s team is entitled to take credit for the improvement from the winner in month 1 across all future months when it considers the total impact of testing on the company going forward.

A Full Year of Testing

Let’s push our model out to a full year of testing:

As you can see, in month 2, the Residual from month 1 kicks in, dropping the expected loss in half. By month 3, it’s wiped out the deficit and started to show a profit, because now we have Residuals from both months 1 and 2. And look at the numbers after 6, 9, and 12 months.

Now, is every testing month a linear multiple of previous ones? No, there will be variations. Some months you will not do as well; but other months you will do better than expected. The point is to keep on working hard and remain focused on what the long-term impact of testing has had on your company.

Penny Wise, and Pound Foolish

We’re almost done for today, though I promised you a short anecdote. This issue of properly attributing success via Residuals may seem obvious while you read this article, but I warn you it’s often forgotten “in the wild.” This loss of memory isn’t restricted to small companies. I know one very large company who every reader would recognize by name (if I were free to say it), who started down the continuous testing path. After six months, the company was able to see direct, measurable improvement of several millions of dollars to their annual income based solely on the impact of their winning tests. Yet, they were “only” winning 1 out of 20 tests. And their costs of testing were running about $30k per month. What do you suppose was their next step? Get ready for it: they pulled the plug on their testing program. They couldn’t get past the fixed costs and the 19 out of 20 tests not being huge homeruns. So don’t ever, ever doubt the ability of humans to snatch defeat from the jaws of victory.

That’s it for this part of this extended column on Testing. Tune in next time when we’ll dive deeper into how to measure your testing efforts. We’ll tighten up the model a bit, and discuss how to calculate the “true costs” of continuous improvement testing. I’ll also introduce the concept of the “opportunity cost” and “hindsight cost” of running a test, which is how you get senior management to deglaze and perk up when you show them your optimization team’s ongoing efforts.

If you have any questions about this (or any) month’s column, or if you need more specific advice about how to implement a testing plan in the context of  your company’s conversion improvement strategy, please feel feel to contact me directly at marketingland -at- johnquarto.com

To download the spreadsheet for this month’s column, please visit:

http://johnquarto.com/documents/BenefitsOfContinuousTesting-version1.xls

It’s been 4 years since the book made the best-seller list, in some ways I’m insulted it took this long to get its PDF ripped by pirates.

Nevertheless if you don’t already own a copy of my book, I encourage you to download this. I’d rather have the valuable information (and stunning writing! ha!) in your hands than not, even if I miss out on my royalty.

The start of a new year is a popular time for resolutions. Many people resolve to eat better (read: ‘eat less’). Or exercise more. You know the drill.

And right about now, three weeks into a new year, is when the vast majority of those resolutions start to break down.

Of course, even among marketing cognoscenti new year’s resolutions rarely involve analytics and key performance indicators (KPIs) — so I’d like to take this opportunity to outline a plan you can follow to slim down and shape up in 2012, metrics-wise.

What Are Metrics For?

Now, in many ways there’s a direct correlation between the everyman’s resolutions (and  subsequent frequent failures) and those you can practice for marketing metrics. When we talk about “eating less” or “exercising more” what we really mean is practicing those activities in the overall context of “being more fit”. Let’s face it, isn’t that the real goal? If you actually end the year healthier than you started, would it matter nearly so much if you’d also eaten more? The purpose of a metric is to support incremental improvement toward a goal.

When companies want to get more out of their numbers they often suffer from what might best be described as “number glut”. They figure if they measure more, they will have more data (ok, true enough) and that more data means better insight (dubious, at best). In fact, I’m going to assert that the more metrics you add to your arsenal of KPIs, the more likely you are to be confused as to your true current state of business health, possibly even unconsciously. In such a case, you don’t have metrics, you have a fetish.

Let me suggest a plan for improvement that can be part of your professional new year’s resolutions.

Step 1: Identify The Real Goal

Back away from any metrics you currently look at and instead ask yourself what you want the end state to be. Make it concrete. Some common retail or lead gen goals might be: “This time next year I want to lift conversions on my site by 25%” or “I want us to double sales in 2012” or “I want to halve the number of unsubscribes from our email list”.  For simplicity, I’ll chose  one — “increase conversions by 25%” — for the remainder of this column.

While macro goals are different for each business, there’s nevertheless a rather finite number of them. Therefore, explicitly stating a primary goal might feel obvious — until you find yourself listing two, three or even more such goals, some of which might be mutually exclusive. Simple and “obvious” exercises like this help define which goals are more important than others.

Step 2: Lay It All Out

Look at all the metrics, KPIs, and other data streams bearing down on you every day, week, month.  For each one, describe how it should change if it’s moving you toward your goal. You might even choose to add some conditions just to keep yourself honest. Here are some examples:

• “Gross revenue” — Revenue can be a great measure of conversion improvement, and certainly both should rise at the same time. Just be sure to question yourself as to your goal: was it really “increase conversion” or was it “increase revenues”? If the latter go back to step 1.
• “Conversion rate” — This one is almost a no-brainer. If conversions are going up, then the conversion rate will go up, too. But the opposite isn’t necessarily true — you can improve conversion rate without necessarily improving total conversions (see the following)
• “Visitors per weeks” — Normally, you want visitors rising to get more conversions, but the caveat is that you have to be driving more of the right sort of traffic for this also to be reflected in increased conversions. If you sell red sweaters, and you drive more visitors from south Florida to your site, we can agree we’re unlikely to see a huge spike in conversions. Or you may drive too specific a niche market to the site — those red sweaters might be mighty popular to folks in Nome, Alaska, but are there enough such customers to keep your business alive?

When I said “lay it all out”, I mean it. I use 3×5 cards for this and lay it out on a table. Old school, yes, but fast and effective.

Step 3: Pick Your Favorite Date For Saturday Night

Now pick out the metrics you see as most important. I’ll let you decide what’s constitutes “important”, but personally, I ask myself “if I could have one (and only one) of these metrics, which one would I pick? What can’t I live without?” Then, “if I could only have two, which two would I pick?” (doesn’t have to include the single metric answer). This is when you’ll be glad you used the 3×5 cards. You can include as many of these as you’d like to list out, but I highly recommend you keep this to a manageable list, say 10-12 (max) of your most important metrics. If you only come up with 4, then so be it.

Step 4: Insist On Delivery

Once you’ve got your prioritized metrics listed — and remember you’ve already pre-described how each of these feeds into the goal from Step 1 — you now want to hold them to account for producing results. For the next 90 days, measure how each of these KPIs changed with respect to your goal. And make changes to your online efforts with techniques designed to move these metrics in the direction you’ve already outlined. In our example, driving better qualified traffic to the site is expected to lead to an improved conversion rate, so ask how you can drive better qualified traffic — more compelling and relevant copy to improve organic search? Goal-focused paid search ads? Paying more for specific search terms? Or, perhaps better use of negative match terms to weed out unwanted traffic? You already know a boatload of tactics for driving traffic, but which specific tactics can drive strategic improvement of each metric?

Step 5: Vote Someone Off The Island

At the end of the 90 days, list out the same metrics from Step 2, but this time prioritize them by the relative “bang for the buck” you got from each one.

Some of your KPIs will have done quite well, yielding you the most insight toward improvement of the goal. Keep those little nuggets.

But some of your KPIs are likely not to have performed as expected. If so, it’s time for someone to leave the island. Whoever is last on the performance list, well, it’s best that you part company and make room for someone new. Remove it from the list.

It may well be that a particular metric in such a sad state is one which you’ve always thought about as “important” or is perhaps considered sacrosanct in your industry — yet if this metric performs poorly in moving you toward your goal, then it is as dangerous to your new year’s resolution as a 24-hour Krispy Kreme donut shop. Drive on by!

Keep the other metrics; and, to replace the metric voted off, plug in one of your other metrics that didn’t quite make it onto your list in step 3. What we want to do is create a meritocracy — or better yet, a “metric-ocracy” — where only those KPIs who actually prove themselves effective at moving you towards your goal stay in the mix.

Now that you’ve gotten through the first 90 days, the exercise will become progressively easier. Repeat it again. And again. You’ll be amazed where you get a year from now when you’ve finished your fourth iteration.

Good luck!

P.S. Don’t combine Step 2 and Step 3 at the same time, because you think you’ll save time. I guarantee you’re doing yourself a disservice. Step 2 is about left brain specifics, whereas Step 3 is about your right-brain instincts. Good business people use both sides of their brain, but get the best outcomes when done separately.

P.P.S. Let me pre-answer the obvious question: What to do with a metric that has been voted off? Kill it forever?  I’d answer “absolutely not.”  I’d keep it available and every once in a while mix it back in to see if a second chance is merited.

The purpose of testing is not to find out what works, but rather to find out what does not work.

I often encounter clients who have what they consider a large percentage of “failed” tests. Yet these tests reveal a rather large amount of information and insight towards future testing.

I’ve been able to restore multiple years of blog posts, but will have to handle all the posts from late 2010 and all of 2011 by hand.

This applies unfortunately to my popular posts “Math for Marketers”. The good news i I’ll be posting those separately to a new blog dedicated solely to that topic (www.MathForMarketers.com) starting in January 2012

Takes 2 seconds to drop my laptop into the Hengedock when I want to go “full desktop”, and 2 seconds to easily pull the laptop out when I’m rushing out the door.

I love my Hengedock. Apparently, so do a lot of Mac owners as the new second-gen Hengedock has been recently released.  What I don’t love is the company’s new policy on upgrades for early adopters.

My understanding from their recent email to me, is that they are offering a full replacement to anyone who bought the first-gen product in the last 30 days. So far, so good.  In fact, so far so unexpectedly nice. Though, you do need to send it back before they send you the new one.

But if you were a super early adopter (like me!), you only get a 50% discount towards a new Hengedock. Wait, you’re going to offer the guy who just bought one in October a full replacement, but you’re going to offer me — who bought the product when no one knew or cared about www.hengedocks.com  – less of a discount?

It’s funny in some ways because their email seems to acknowledge this issue of Early Adopter Syndrome:

Has the news of our second generation docks got you feeling a bit of early adopter remorse? We know that feeling all too well, so we’re offering free upgrades and credits toward a new docking station to our existing customers

So they get the problem — but what they clearly don’t get is the solution. Why are companies so wacky about things like this and think their customers are trying to rip them off? It actually hurts the good will I feel towards the product and the company by penalizing me for having bought the dock when it first came out. How can that possibly be good, for me or for the Company? This is a product, not a software license. They’re not going to do anything different with the “fresh” piece of plastic that gets returned to them from the other guy than they would do for the “stale” piece of plastic I have.

Now if this were THIRD generation and Hengedocks.com were making me an offer to replace my first-gen item at a discount that would feel quite different — because something happened in between (the second gen product). But the item a fellow bought 30 days ago  is literally the same thing I bought 6 months ago **when I helped you build your company and your brand**.

The other fellow did nothing of the sort, probably learned about it from other early adopters like me who raved at its nice design and utility. I call that “sweat marketing”. I was the one who took a chance and the one who should be treated better, as a “founder”-customer, than Johnny-come-lately who saw an ad for the product paid for by the money I spent with you not so long ago.

At least, that’s how it feels on this end.  Just when I was really grooving on my hengedock, I’m starting to feel a bit … unhenged.

“What if we create a completely new site from scratch using the persuasion framework you developed and then A/B test it against our old site — What would happen?”

Let me answer the second part of the question first: The one thing I’m quite certain of is that humans are awful at predicting the future. If someone tells you they  know what’s going to happen in the future put it to the test. I’ve got one buddy who claims that he can “often” predict the future, albeit only about 10 seconds forward. “Great!” I told him. Let’s go to Vegas, and I’ll put up the money and we’ll play craps all weekend.  That’s surely a fast enough game that your 10 second limitation won’t stop us from getting rich!” We never did make it there — somehow it was never the “right time” to predict the future, I suppose.

But when it comes to optimizing your online efforts and you (or, more typically, the boss) presumes to know what will happen next, hold yourself and your team to a hard objective measure: TEST! It’s the best way to leave the subjective world of opinion and enter the objective world of reality.

But of course, that’s what our client really meant with his question.  To rephrase it, one might ask,  ”Is it more efficient to A/B test large scale changes by jumping into a vastly improved architecture first and then proceed with incremental improvement of that new architecture?”  As your instincts might indicate, this can be an ambitious way to jump-start optimization efforts, under the right circumstances.

Most sites have a legacy architecture that actively works against persuasion and conversion, and if your pockets (and your nerves) are big enough, the ultimate A/B test is to say, “Hey, what we have now is a Control benchmark. What we really want to to make sure that the major persuasive scenarios are aggressively planned to be ready for optimization and then go from there.”  You’ll note that I’m emphasizing the persuasive planning portion because just throwing a new design up isn’t likely to leave you in a position to learn from future optimization as efficiently as possible. Instead the purpose of the newly revamped site is to accept the old site as a benchmark to measure against, and to put in place the series of key performance indicators and measurement points which will be used in ongoing optimization quarter after quarter, using experience and insight.

There are some additional issues that should be thought through of course: First off, the technical implementations that will continue to send a portion of your traffic to the old site. Second, make sure you have your apples and oranges understood so that you compare the correct new analytics with their counterparts from the old site’s analytics.  And third, don’t forget that you’re not testing in a sterile academic environment where the goal is to understand all permutations completely, but rather in the living, breathing biodome of a company with live customers and a live income stream. As improvements are verified more and more traffic should be assigned to those improved points of conversion and persuasion — much like the medical profession where the adage “first, do no harm” is the rule.

Personally, I have to give this client props for the sheer nerve of bringing this topic up. It shows a confidence in his team’s ability to implement and live with change  — “the only certainty is change” — as well as a willingness to quickly say “ok, we were wrong in this aspect, so let’s analyze/test/optimize yet again”.

I think this is going to be a fascinating long-term experiment. What about you?

Or are you a lettuce picker? The sort of person that toils for long hours in the field and accomplishes an honest day’s back-breaking labor of work that most white collar business execs would consider a less-than-optimal career.

That fact is, when it comes time to harvest, virtually the entire crop must be worked on at the same time. You don’t have time to cherry pick, and anyway the average business isn’t expert enough in how and what to optimize to know which portion of the crop should be cherry picked. Instead, when that crop is ripe, it’s time to get out there in the field and put in a 14-hour day getting it harvested.

Often when we speak wiith prospective clients, they have the impression that there is some magical formula that leads to higher conversion rates and that it can be achieved without any hard work or commitment. The reality, however, is that our most successful clients who enjoy on-going regular improvements of 40-80% in their conversion rates year after year are the ones who are implementing change on a regular basis. They’re lettuce pickers, and not so proud as to let hard work get in the way of increased revenue.

Are you a hard-working lettuce picker when it comes to your website? Are you guessing at what changes will improve your site? Or do you work diligently every week, every month, and every quarter to effect continuous improvement?