Our Blog

Every time I watch this video it reminds me that what most of us are doing isn’t working. We constantly look at data and try to assess patterns and understand cause and effect. This is actually pointless. The video shows a very simple osculating set of pendulums that can take on all kinds of different pattens that are not created by cause and effect relationships. The actual patterns in the video are nothing more than optical illusions.

I have been reading the book The Lean Startup by Eric Ries recently. I love this book and would recommend it to everyone. I have even purchased copies for my entire staff. The most important take away for me so far is that we need to stop looking backwards at data, reports, and metrics to try to understand what happened or to come up with a story to explain what happened.

Instead, we should be looking forward and conducting A-B testing or split testing. This method has been around since direct mail. A company would basically split a mailing list down the middle and randomly send one form of the mailing to one list and a different mailing to the other. The company would then measure response rates to determine which mailing was more effective. You could say with certainty that “approach A” is better than “approach B,” for example, because it yielded better results. With today’s tools, we can run these kinds of tests all the time - at i.c.stars we are getting on a weekly “build-measure-learn” cycle.

We still need big picture metrics to know overall if we are moving in the right direction. “Is the business performing better?” is the only real question that you can answer by looking back at data.

“Are revenues going up?” -Yes.
“Why are revenues going up?” -You cannot possibly tell from a revenue chart.

There are simply too many variables that affected any revenue result to be understood in terms of cause and effect. You can’t answer “why” by looking in the rear view mirror.

This insight has lead to a profound release of stress of and anxiety. Instead of big initiatives that roll out new approaches that may or may not impact the metrics, we are simply focused on learning validated, causal insights, every week.

But this has brought up an implementation question - what tools to use to run and manage these tests? There are several I’ve seen and heard of (Kissmetrics being the market leader). What tools are you using to conduct and report on these kinds of tests? Where does the ‘learning’ get stored so we don’t forget?