Why You Should Run More Experiments, Not Fewer Even If Most Fail

At Microsoft, 67% of experiments fail. At Google, Bing, and Netflix, that number climbs above 80%.

But that doesn’t mean experiment less. It means experiment smarter.”

This surprising insight, shared by experimentation expert Ron Kohavi, flips conventional wisdom on its head.

Most of us are conditioned to think failure is bad. But when it comes to product development, failure is often the most powerful way to learn. Not because it’s fun, but because it’s honest.

Let’s break down why experimentation matters — and why failure isn’t something to fear, but a tool to embrace.

The Hidden Gold in the Backlog

At Bing, there was a small, almost laughably simple idea sitting in the backlog: bring the first line of the ad description into the title.

It sat untouched for months. No one prioritized it. It didn’t “feel” like a big win.

But when they finally tested it?

It generated over $100 million in annual revenue.

That’s the thing about experimentation: the results often defy intuition. Without testing, that insight would’ve stayed buried — a line item in a JIRA board that no one ever noticed again.

This story isn’t unique. It happens at every product-driven company. Small changes that seem insignificant on the surface turn out to be massive drivers of value. But you only discover them if you’re systematically testing, learning, and iterating.

The Problem with “Gut Feel”

Your gut feeling is the lowest form of evidence.
It’s just dressed-up instinct.

Let’s look at how we actually make decisions in product and strategy. Ron Kohavi outlines a hierarchy of evidence that should guide our thinking:

1. Meta-analyses of experiments
2. Randomized Controlled Experiments (A/B tests)
3. Non-randomized controlled tests
4. Observational studies
5. Case studies, anecdotes, HiPPOs (Highest Paid Person’s Opinion)

Now think honestly: where does most roadmap prioritization happen?

Often at level 5. Gut feeling. Executive pressure. Loud opinions in the room.

But if you want to reduce risk, uncover real user insights, and build a better product — you need to start moving up that pyramid.

And that starts by testing more, not less.

The Real Point of Experimentation: Learning

When teams run experiments, the goal isn’t to prove you’re right.

It’s to find out what’s true.

Even when an experiment “fails,” it adds to your understanding. You now know what doesn’t work. That’s a win. Because every failed test sharpens your ability to ask better questions, design tighter hypotheses, and move faster next time.

Experimentation builds organizational memory. It’s how you move from guessing to knowing.

If you’re not running experiments, you’re not learning.
You’re just hoping.

It’s okay to rely on baselines or north-star metrics for reporting. But when you’re trying to understand if a product change is driving behavior, only an experiment can give you confidence.

And over time, as your experiment volume grows, you can start building meta-analyses — layered insights that give you compounding learning across your org.

Building a Culture of Experimentation

If your team doesn’t have a habit of testing, here’s how to start:

• Begin with small bets. Frame them as learning exercises.
• Track not just “wins,” but what every experiment taught you.
• Make it safe to be wrong. Reward clear hypotheses, not just results.
• Share experiment results widely. Normalize failure as discovery.
• Systematize it. Use templates, tools, and reviews to make experimentation repeatable.

Remember: failure isn’t the opposite of success in product. It’s the path to it.

If you want to build a system where every test teaches you something meaningful, and every failure is part of the roadmap to success, I highly recommend diving deeper here: Full breakdown

The teams that win aren’t the ones who guess best.
They’re the ones who learn fastest.