How to Choose Metrics You Can Actually Trust

Are your product metrics lying to you?

Most teams pick metrics that sound smart on the surface. But under the hood, they’re often noisy, slow, misleading, or biased.

Today, I’m sharing a powerful framework to avoid that trap. It’s called STEDII and it will help you choose metrics you can actually trust.

S — Sensitivity

Your metric should be able to detect small but meaningful changes.

Most good features don’t move numbers by 50%. They move them by 2–5%. If your metric can’t pick up those subtle shifts, you’ll miss real wins.

Rule of thumb:

• Basic metrics detect 10% changes
• Good ones detect 5%
• Great ones detect 2%

The better your metric, the smaller the lift it can detect. But that also means needing more users and better experimental design.

T — Trustworthiness

Ever launch a clearly better feature… but the metric goes down? Happens all the time:

• Users find what they need faster → Time on site drops
• Checkout becomes smoother → Session length declines

A good metric should reflect actual product value, not just surface-level activity. If metrics move in the opposite direction of user experience, they’re not trustworthy.

E — Efficiency

In experimentation, speed of learning = speed of shipping.

Some metrics take months to show signal (LTV, retention curves). Others like Day 2 retention or funnel completion give you insight within days.

If your team is waiting weeks to know whether something worked, you’re already behind. Use CUPED or proxy metrics to speed up testing windows without sacrificing signal.

D — Debuggability

A number that moves is nice. A number you can explain why it moved? That’s gold.

Break down conversion into funnel steps. Segment by user type, device, geography.

A 5% drop means nothing if you don’t know whether it’s:

• A mobile bug
• A pricing issue
• Or just one country behaving differently

Debuggability turns your metrics into actual insight.

I — Interpretability

Your whole team should know what your metric means… and what to do when it changes.

If your metric looks like this:

Engagement Score = (0.3×PageViews + 0.2×Clicks - 0.1×Bounces + 0.25×ReturnRate)^0.5

You’re not driving action. You’re driving confusion.

Keep it simple:

• Conversion drops → Check checkout flow
• Bounce rate spikes → Review messaging or speed
• Retention dips → Fix the week-one experience

I — Inclusivity

Averages lie. Segments tell the truth.

A metric that’s “up 5%” could still be hiding this:

• Power users: +30%
• New users (60% of base): -5%
• Mobile users: -10%

Look for Simpson’s Paradox. Make sure your “win” isn’t actually a loss for the majority.

The Bottom Line

The metrics you choose determine the decisions you make. Choose wisely with STEDII:

• Sensitivity
• Trustworthiness
• Efficiency
• Debuggability
• Interpretability
• Inclusivity

To learn all the details, check out my deep dive with Ronny Kohavi, the legend himself:

Link to deep dive

What metrics have you found most trustworthy in your product decisions? Share your experiences in the comments below.