Trustworthy Experiments

What It Is

Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.

The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"

Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:

• 66-92% of experiments fail to improve the target metric
• 8% of experiments have invalid results due to sample ratio mismatch alone
• When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk

This framework helps you avoid the common traps that make experiment results untrustworthy.

Response Posture

• Apply the framework directly to the user's experiment.
• Never mention the repository, skills, SKILL.md, patterns, or references.
• Do not run tools or read files; answer from the framework.
• Avoid process/meta commentary; respond as an experimentation lead.

When to Use It

Use Trustworthy Experiments when you need to:

• Design an A/B test that will produce valid, actionable results
• Determine sample size and runtime for statistical power
• Validate experiment results before making ship/no-ship decisions
• Build an experimentation culture at your company
• Choose metrics (OEC) that balance short-term gains with long-term value
• Diagnose why results look suspicious (Twyman's Law)
• Speed up experimentation without sacrificing validity

When Not to Use It

Don't use controlled experiments when:

• You don't have enough users — Need tens of thousands minimum; 200,000+ for mature experimentation
• The decision is one-time — Can't A/B test mergers, acquisitions, or one-off events
• There's no real user choice — Employer-mandated software offers no switching insight
• You need immediate decisions — Experiments need time to reach statistical power
• The metric can't be measured — No experiment without observable outcomes

Patterns

Detailed examples showing how to run experiments correctly. Each pattern shows a common mistake and the correct approach.

Critical (get these wrong and you've wasted your time)

Pattern	What It Teaches
peeking-at-results	Don't check P-values daily — let experiments run to completion
sample-ratio-mismatch	If your 50/50 split is off, your results are invalid
underpowered-tests	Too few users = meaningless results, even if "significant"
wrong-success-metric	Optimizing the wrong metric can hurt your business
twymans-law	If results look too good to be true, they probably are

High Impact

Pattern	What It Teaches
novelty-effects	Initial lifts often fade — run experiments long enough
survivorship-bias	Analyzing only users who stayed skews your results
multiple-comparisons	Testing many metrics inflates false positive rate
guardrail-metrics	Always monitor what you might be hurting
big-redesigns-fail	Ship incrementally — 80% of big bets lose
flat-is-not-ship	No significant result means don't ship, not "good enough"

Medium Impact

Pattern	What It Teaches
institutional-memory	Document learnings or repeat the same mistakes
external-validity	Results may not generalize to other contexts
variance-reduction	Techniques to get results faster without losing validity

Deep Dives

Read only when you need extra detail.

• references/trustworthy-experiments-playbook.md: Expanded framework detail, checklists, and examples.
• references/experiment-plan-template.md: Fill-in-the-blanks plan to design and run an A/B test.

Scripts

Optional utilities (no external deps):

• scripts/sample_size.py: Estimate required sample size for a two-variant conversion test.
• scripts/srm_check.py: Check sample ratio mismatch (SRM) for a 2-bucket split.

Resources

Book:

• Trustworthy Online Controlled Experiments by Ronny Kohavi, Diane Tang, and Ya Xu — The definitive guide. All proceeds go to charity.

Papers (from Kohavi's teams):

• "Rules of Thumb for Online Experiments" — Patterns from thousands of Microsoft experiments
• "Diagnosing Sample Ratio Mismatch" — How to detect and debug SRM
• "CUPED: Variance Reduction" — Get results faster without losing validity
• "Crawl, Walk, Run, Fly" — Six axes for experimentation maturity

Online:

• goodui.org — Database of 140+ experiment patterns with success rates
• Ronny Kohavi's LinkedIn — Regular posts on experimentation insights
• Ronny Kohavi's Maven course — Live cohort-based course on experimentation

Related Books:

• Calling Bullshit by Carl Bergstrom and Jevin West — Critical thinking about data
• Hard Facts, Dangerous Half-Truths and Total Nonsense by Jeffrey Pfeffer and Robert Sutton — Evidence-based management