Effect Size
A quantitative measure of the magnitude of a difference between groups in an experiment, independent of sample size. It answers the question "how big is the improvement?" rather than "is there an improvement?"
Also known as: treatment effect, lift
Why effect size matters
Statistical significance tells you whether an effect exists. Effect size tells you whether it matters. A test can be highly significant (p < 0.001) while detecting an improvement so small (0.01% conversion lift) that it has no practical business impact.
Effect size is what drives business decisions. When deciding whether to invest engineering resources in shipping a variant, you need to know the expected magnitude of improvement, not just whether the improvement is nonzero. A 0.5% lift might not justify the maintenance cost, while a 5% lift clearly does.
Effect sizes also enable comparison across experiments. If you want to know which of your 10 past experiments produced the most impact, comparing p-values is meaningless (they depend on sample size). Comparing effect sizes gives you a genuine ranking of what moved the needle most.
The newsletter
Join our KISS newsletter
One short read a week on what actually moves revenue, in a free email. Read by 10,000+ operators and founders.
No spam. Unsubscribe in one click.
How to calculate effect size
For conversion rate experiments, effect size is simply the absolute or relative difference between variant and control conversion rates. Absolute effect: variant rate minus control rate. Relative effect (lift): (variant rate - control rate) / control rate. For standardized comparisons across different metrics, use Cohen's d: (mean difference) / pooled standard deviation.
Effect Size examples
A beauty brand runs a test that shows a statistically significant improvement in add-to-cart rate. However, the effect size is only 0.3% absolute (from 4.2% to 4.5%). Given the engineering cost to maintain the variant, they decide the effect size is too small to justify shipping.
A collaboration tool tests a new trial onboarding sequence and measures a 15% relative lift in 7-day activation (from 20% to 23%). This effect size translates to 300 additional activated users per month, generating an estimated $45,000 in incremental annual revenue - clearly worth shipping.
How to Track in KISSmetrics
KISSmetrics reports effect sizes as both absolute and relative differences in experiment results. When planning experiments, set a minimum effect size that represents a practically meaningful improvement for your business. Use this as the basis for power analysis to determine required sample sizes.
Common Mistakes
- -Focusing on statistical significance while ignoring whether the effect size is large enough to matter
- -Comparing raw effect sizes across metrics with different scales without standardizing
- -Expecting large effect sizes from minor changes - most UI tweaks produce 1-5% relative improvements
- -Not distinguishing between absolute and relative effect sizes, which can be very different for low-baseline metrics
Pro Tips
- +Establish minimum meaningful effect sizes for your key metrics before running experiments - what lift actually changes your business trajectory?
- +Track historical effect sizes by experiment type to calibrate future expectations and power analyses
- +Use relative lift for communicating results to stakeholders (it is more intuitive) but absolute differences for sample size calculations
- +When an effect size is smaller than expected, consider whether the test variant was bold enough or whether the hypothesis was wrong
Related Terms
Minimum Detectable Effect (MDE)
The minimum detectable effect is the smallest difference between control and variant that a test can reliably detect, given its sample size, baseline rate, significance level, and statistical power. It is set at design time, before the test runs: MDE = (Z_alpha/2 + Z_beta) * sqrt(2 * p * (1 - p) / n). If your MDE is 10% and the real improvement is 3%, the test will return "not significant" even though the improvement is real.
Statistical Power
The probability that a test will correctly detect a real effect when one exists, typically set at 80% as a minimum standard. Higher power means a lower chance of missing genuine improvements.
Confidence Interval
A range of values that likely contains the true effect of a change, calculated from experiment data. A 95% confidence interval means that if the experiment were repeated many times, 95% of the calculated intervals would contain the true value.
P-Value
A p-value is the probability of seeing a result at least as extreme as the one you measured, assuming the null hypothesis of no real difference is true. You calculate it by converting the observed difference into a test statistic such as a z-score, then reading the tail area beyond it. A p-value below 0.05 is the conventional threshold for calling a result statistically significant.
Hypothesis Testing
A statistical method used to determine whether observed differences in data - such as a higher conversion rate in a test variant - are likely real or could have occurred by random chance.
Build your business intelligence layer for free.
KISSmetrics records the variant as a property on the person, so every other report splits by variant without extra setup.