Summary

In this era of booming online competition, an average user experience isn’t going to cut it. The tiny percentage of content shown on the first page of Google gets the massive majority of clicks, and therefore, revenue. You need to dominate the competition. How do you do that?

Experimentation Works by Stefan Thomke has answers.

WHAT IT’S ABOUT: Innovate your organization by continuously running high-quantity, high-velocity experiments.

WHY YOU SHOULD READ IT: To remain competitive during this continuing digital revolution, businesses—yes, yours too—need to harness the power of consumer behavior data.

HOW YOU’LL USE IT: This is your bible of data-driven decision-making. It houses a pile of evidence arguing against deferring to HiPPOs (more on that later), and a guide for beginning your business’s experiment-focused innovation.

What is experiment-focused innovation?

To get the most clicks, you need to know what’s working on your site and make it better. To do that, says Thomke, you need experiment-focused innovation.

Experiment-focused innovation means regularly altering the customer experience to see if customers like a new alternative better. These tests should be plentiful and fast, says Thomke. Not a single element of the customer experience should be exempt, including online features like web appearance or aspects of physical stores like product placement.

So, Thomke says, you need to regularly change your design and content to weed out what doesn’t work and narrow in on what works best. This may be more nausea-inducing than it sounds—data shows most of your changes will have a negative effect or none at all.

Jeff Bezos says you should experiment anyway:

“Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right . . . We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs.

The difference between baseball and business, however, is that . . . no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs.”

—Jeff Bezos, 2016 Letter to Amazon Shareholders

Experimentation works because digitalization has made each “swing” cost near nothing. Better yet, each one has the potential for an exponential return. According to Thomke, Google swings so often that their results are useless 96.1% of the time.

Embrace failure, and your business can earn the innovative prowess of Google.

Next Article

Sustainable Business

Leading Sustainably Should Be Required Reading for Businesspeople

Book review: Leading Sustainably: The Path to Sustainable Business and How the SDGs Changed Everything by Trista Bridges and Donald Eubank.

Team Management

5 Reasons Remote Work Is the Ideal Creativity Incubator

If you’re worried about keeping your remote teams productive and efficient, you may be missing a big remote work advantage: creativity.

Starting a Business

The Golden Rules for Pitching Sustainability to Angel Investors

Your business plan is solid—now you just need capital. So how do you get the attention of sustainability-focused investors?

How do I start experimenting in my organization?

According to Thomke, you need two things to get started with experiment-focused innovation. The first is the right process. The second is the right culture.

What’s the right process for experiment-focused innovation?

Here are five basic steps you should keep in mind while building your company’s experimentation infrastructure:

1. Define Success Metrics: Finding a short-term metric that predicts long-term outcomes is difficult. Instead, consider using an overall evaluation criterion (OEC). This is a number, like a KPI, that you will measure all of your experiments against. Booking.com, for example, uses conversion rate as their OEC.
   
2. Organize for Experimentation: Choose an organizational model that works for you, whether it’s centralized, decentralized, or something in between. The important thing is having a standardized system and easy access to experimentation specialists.
   
3. Ensure Your Experimentation System Scales: Don’t let the rate of failure make you pause—only about 10–20% of experiments generate results with a clear outcome, even for companies built on experimentation, like Google and Microsoft. To get more impactful results, run more experiments, not fewer.
   
4. Build Trust in the System: It doesn’t matter how well-designed your tests are if people don’t trust the results. Build faith in the system through employee training, ensuring experiment integrity, and demonstrating the benefits.
   
5. Keep It Simple: Including too many variables makes it hard to pin down the source of an effect. Complex designs also make experiments more vulnerable to mistakes. Rather than testing multiple changes with a single experiment, choose one. Save the other changes for separate experiments.

What kind of culture do I need for experiment-focused innovation?

You can prepare your organization for innovation by ensuring your culture has Thomke’s seven recommended attributes:

1. A Learning Mindset: Be ready and excited to learn from failure! A low failure rate is a sign that you’re not taking enough risks. But don’t worry, it’s worth it: Studies have shown that short-term failures facilitate innovation and performance in the long run.
   
2. Consistent Rewards, Values, and Objectives: Avoid mixed messages by ensuring employees aren’t punished for failure. Studies show that employees are more willing to experiment when the company emphasizes the value of learning from failures and not penalizing for failure.
   
3. Intellectual Humility: Large-scale experimentation can yank your beliefs out from right under you. This is stressful for everyone involved and can even push organizations past their breaking point. All parties must be willing to be wrong to prevent meltdowns.
   
4. Moral Scrutiny—But Not Too Much: Too much internal scrutiny may slow experimentation to a trickle, but too little scrutiny and you’ll have a PR crisis on your hands. Create a clear policy about what kind of experiments are not okay and stick to that. 
   
5. Tool Trust: This is both a culture and process concern. Tools are only effective if people and organizations trust them, so take the time to address any questions or concerns your employees have.
   
6. The Exploration-Exploitation Balance: There’s only so much time in the day, and organizations need to balance innovating new value (exploring) and capturing value through known processes (exploiting). An all-or-nothing approach is unsustainable.
   
7. Equal Opportunity: The most creative ideas often come from the bottom and have difficulty reaching senior management. As a result, teams tend to accept the opinion of the highest-paid person in the room (colloquially known as the highest-paid person’s opinion, or HiPPO).  

Rather than following HiPPOs, says Thomke, ideas should be equalized. Put all proposals to the same three questions: What experiment did you run? What were your leap-of-faith assumptions? And if no test has been conducted yet, how quickly can you run an experiment so we can decide based on those results?

Thomke is clear: the key to success in a data-driven era is creating systems and cultures where we make decisions by high-velocity experimentation instead of “by PowerPoint, politics, and position in the hierarchy.”

What keeps management from adopting experimentation?

According to Thomke, nearly all arguments against experiment-focused innovation can be boiled down to three theses: the perversity thesis, the futility thesis, and the jeopardy thesis.

The Perversity Thesis

“Any action to improve some aspect of a system will backfire, so don’t bother.”

Objections like “online experiments will lead to incremental innovation, but not big breakthroughs” fall in this camp, says Thomke. The natural rebuttal is that breakthroughs are rarely the result of large, singular changes—they come from minor changes that accumulate.

The Futility Thesis

“Efforts to transform an organization aren’t worth it because they don’t address structural challenges.”

Many organizations discard experimentation when they don’t immediately yield earth-shattering results. But managers need to hold the line: experimenting should be as frequent and ingrained as running the numbers with an experiment-based strategy. “When asked ‘What’s the ROI on experimentation,’” says Thomke, “an appropriate answer is ‘What’s the ROI on breathing?’”

The Jeopardy Thesis

“A proposed action, though beneficial, involves unacceptable risk and costs.”

Thomke identifies the jeopardy thesis as the most dangerous of the three. While it’s easy to identify risks and costs up front, the benefits of an action are more elusive, especially if you haven’t taken action yet.

A common objection of this type is “running experiments on customers without advance consent is always unethical.” Though this objection raises legitimate concerns, the solution is not to avoid experimentation altogether. The solution, says Thomke, is to set clear boundaries about what experiments you will or will not do and make that information public.

For example, LinkedIn, another extensive experimenter, does not run experiments “intended to deliver a negative member experience, have a goal of altering members’ moods or emotions, or override existing members’ settings or choices.”

In Conclusion

It’s easier than ever to create products and content, and harder than ever to stand out. As digitalization expands, businesses will need to continually rediscover what customers like best to stay in the black.

Experimentation Works will help you do that. It has valuable insights for all businesspeople regardless of management level or experimentation expertise. Pick it up and begin your business’s journey toward digital domination today!