Optimization & Testing

Are Your P-values Killing your AB Testing Efforts?

Sep 4th, 2014 | By
Are Your P-values Killing your AB Testing Efforts?

If you’re running AB tests, you’re surely running into concepts like “95% confidence intervals” and the “Null hypothesis”. And you’re already using “p-values”, too, though you may not know it.

Unfortunately there’s a great chance you’re using them wrong, and it’s having an impact on your AB testing program?

Here is a business person’s overview to help you avoid common problems and misunderstandings about p-values.



Thinking About Your Negative Metrics

Apr 16th, 2013 | By

“Negative” metrics — You might prefer the term “Deoptimization Metrics”. Either way they can be just as important  to your continuous optimization efforts as your positive ones. The purpose of a negative metric  is to isolate for you the deleterious effect you may be having on your positive metric. A negative metric is not necessarily
[continue reading…]



I Before E, Except After C

Mar 19th, 2013 | By
I Before E, Except After C

“I Before E, Except After C” — do you remember that from 4th grade spelling? Well, it works just as well when you’re concentrating on
[continue reading…]



Treating Your Conversion Symptoms But Not The Disease

Jan 21st, 2013 | By
Treating Your Conversion Symptoms But Not The Disease

Many organizations get fixated on metrics data, without thinking of where it comes from. But what if your metrics indicate a problem which originates somewhere else?



Polls and Testing: When Is Close No Cigar?

Nov 20th, 2012 | By
Polls and Testing: When Is Close No Cigar?

Now that the 2012 U.S. Presidential elections are over, there’s a bit of a buzz around why some folks thought the election would be close and others who, using some solid statistical techniques, predicted a definitive (and it turns out, accurate) outcome. I want to try to explain the latter, especially for marketers. Why? Because
[continue reading…]



Oh, What A Tangled Web We Weave …

Oct 1st, 2012 | By
Oh, What A Tangled Web We Weave …

“Oh, What A Tangled Web We Weave, When First We Practice To {Self} Deceive” With apologies to Sir Walter Scott and my mangling of his
[continue reading…]



Measuring The Impact of Conversion Lift

Jun 12th, 2012 | By
Measuring The Impact of Conversion Lift

How do you measure the impact of conversion improvement on your site? Most marketers know the basics of how to calculate this, but might get
[continue reading…]



If You’re Not a BRAT, You Might Be The Wurst

Apr 17th, 2012 | By
If You’re Not a BRAT, You Might Be The Wurst

As I’ve mentioned in earlier articles, the biggest threat to effective improvement on a regular continuous basis is looking at too much data. The more metrics
[continue reading…]



Making Millions From Losing Tests (Part 1)

Feb 17th, 2012 | By
Making Millions From Losing Tests (Part 1)

Over the course of this series of columns, we’re going to develop a spreadsheet that you can use to model your own company’s efforts towards testing, each month adding more and more features to the underlying model.

The bigger the company, it seems, the more disconnect there can be between the talented testing and optimization team and senior management. And that, as you might guess, ends up undermining large portions of the revenue and efficiency improvements that come from continuous testing.



Why Test? The Purpose of Testing

Jan 17th, 2012 | By
Why Test? The Purpose of Testing

Why Test? The purpose of testing is not to find out what works, but rather to find out what does not work. I often encounter clients who have what they consider a large percentage of “failed” tests. Yet these tests reveal a rather large amount of information and insight towards future testing. In fact, when a test
[continue reading…]