Minimum Detectable Effect (MDE)
The minimum detectable effect is the smallest difference between control and variant that a test can reliably detect, given its sample size, baseline rate, significance level, and statistical power. It is set at design time, before the test runs: MDE = (Z_alpha/2 + Z_beta) * sqrt(2 * p * (1 - p) / n). If your MDE is 10% and the real improvement is 3%, the test will return "not significant" even though the improvement is real.
Also known as: MDE, minimum effect size, sensitivity
Why minimum detectable effect (MDE) matters
The MDE is the bridge between experiment design and business value. It answers: "Given my traffic volume and test duration, what is the smallest real improvement this test can find?" If the MDE is 10% but you expect a 3% improvement, your test will almost certainly return "not significant" even if the improvement is real.
Setting the right MDE requires balancing ambition with reality. Required sample size scales with the inverse square of the MDE, so halving the effect you want to detect quadruples the traffic you need. Going from a 10% relative lift to a 1% relative lift needs roughly 100 times the sample, which is why sensitive tests are only practical on high-traffic pages.
The MDE should be driven by business relevance, not arbitrary convention. Calculate the lift needed to justify the cost of the change: if a variant requires a week of engineering to build and maintain, what revenue lift makes that investment positive? That is your minimum meaningful effect, and your MDE should be at or below it.
The newsletter
Join our KISS newsletter
One short read a week on what actually moves revenue, in a free email. Read by 10,000+ operators and founders.
No spam. Unsubscribe in one click.
How to calculate minimum detectable effect (MDE)
MDE is determined by your sample size, baseline conversion rate, significance level (alpha), and power (1 - beta). For a given sample size, MDE = (Z_alpha/2 + Z_beta) * sqrt(2 * p * (1 - p) / n), where p is the baseline rate and n is the sample per group. At the standard settings (95% significance, 80% power) the two Z terms are 1.96 and 0.84, so the leading constant is 2.80.
Worked example. Baseline conversion rate 5%, 10,000 visitors per variant. sqrt(2 * 0.05 * 0.95 / 10,000) = 0.00308. MDE = 2.80 * 0.00308 = 0.0086, or 0.86 percentage points absolute. Against a 5% baseline that is a 17% relative lift, so this test can only reliably detect improvements of roughly 17% or larger. To detect a 5% relative lift instead (0.25 percentage points) you would need about 119,000 visitors per variant.
Run the calculation in the direction that matches your constraint. If traffic is fixed, solve for MDE and use it to filter your experiment backlog. If the effect size is fixed by the business case, solve for n and use it to set the test duration.
Minimum Detectable Effect (MDE) examples
A niche retailer with 10,000 monthly visitors and a 2% conversion rate calculates their MDE at 30% relative lift for a 4-week test. This means they can only detect large improvements, so they focus on radical redesigns rather than incremental tweaks.
Benchmark: High-traffic sites can detect 2-5% MDE; low-traffic sites may need 20%+ MDE
A product-led growth company with 3,000 weekly signups calculates they can detect a 7% relative lift in activation within 2 weeks. They use this MDE to filter their experiment backlog, only running tests expected to produce at least a 7% improvement.
How to Track in KISSmetrics
Before launching any experiment in KISSmetrics, calculate your MDE from the traffic actually available to the page or flow you are testing, not from sitewide totals. If the MDE is larger than the effect you expect, either run longer, route more traffic to the test, or test a bolder variant. The Funnel and Product Usage reports give you the per-step volumes that go into the calculation, and the A/B Test report shows the observed lift and significance once the test is running.
Common Mistakes
- -Not calculating MDE before launching, then being surprised when the test is inconclusive
- -Setting MDE unrealistically low for the available traffic, resulting in tests that run for months
- -Confusing MDE with the expected effect - MDE is a design parameter, expected effect is a prediction
- -Ignoring that MDE increases as baseline rates move toward 0% or 100% (extreme base rates need more traffic)
Pro Tips
- +Create an MDE reference table for your key pages: "With our homepage traffic, a 2-week test can detect a 5% relative lift in signups"
- +If your MDE is too large for subtle tests, use composite metrics (like revenue per visitor) that capture multiple improvements simultaneously
- +Recalculate MDEs when traffic patterns change (seasonal shifts, marketing spend changes) to keep test planning accurate
- +Use MDE to prioritize your testing roadmap - test bold ideas on low-traffic pages and subtle refinements on high-traffic ones
Related Terms
Effect Size
A quantitative measure of the magnitude of a difference between groups in an experiment, independent of sample size. It answers the question "how big is the improvement?" rather than "is there an improvement?"
Statistical Power
The probability that a test will correctly detect a real effect when one exists, typically set at 80% as a minimum standard. Higher power means a lower chance of missing genuine improvements.
Type II Error
A false negative in hypothesis testing - failing to reject the null hypothesis and concluding that a change had no effect when it actually did produce a real improvement.
Confidence Level
The percentage probability that a confidence interval calculated from a given experiment will contain the true population parameter, commonly set at 90%, 95%, or 99% in A/B testing.
Hypothesis Testing
A statistical method used to determine whether observed differences in data - such as a higher conversion rate in a test variant - are likely real or could have occurred by random chance.
Further Reading
A/B Testing Statistical Significance: When to Call a Winner
Understand statistical significance in A/B testing. Learn sample size calculation, confidence intervals, and why stopping tests early gives false results.
Introduction to A/B Testing: How to Run Experiments That Actually Work
A beginner-friendly guide to A/B testing covering hypothesis formation, test design, statistical significance, and common mistakes to avoid.
A/B Test Sample Size: How to Calculate It and Why Most Teams Get It Wrong
How to calculate A/B test sample size using MDE, baseline conversion rate, and statistical power, plus common mistakes like stopping early and peeking at results.
Build your business intelligence layer for free.
KISSmetrics records the variant as a property on the person, so every other report splits by variant without extra setup.