Frequentist Testing
The traditional statistical approach to A/B testing that evaluates results by calculating how likely the observed data would be if there were no real difference between variants, using p-values and confidence intervals.
Also known as: classical hypothesis testing, Neyman-Pearson testing
Why frequentist testing matters
Frequentist testing is the standard framework for most A/B testing platforms and scientific research. Its strength is rigor: when properly implemented with predetermined sample sizes, significance levels, and test durations, it provides well-calibrated error rates that you can rely on.
The framework forces discipline in experiment design. You must decide your sample size, significance level, and power before the test starts. You must not peek at results and stop early. You must not change your success metric mid-test. These constraints, while sometimes frustrating, prevent the most common ways that experiments produce misleading results.
Frequentist methods are well-understood, widely taught, and supported by every major experimentation platform. When you report a result at "95% significance," most data-literate stakeholders understand what that means (or at least have a reasonable approximation). This shared understanding makes communication easier.
The newsletter
Join our KISS newsletter
One short read a week on what actually moves revenue, in a free email. Read by 10,000+ operators and founders.
No spam. Unsubscribe in one click.
Frequentist Testing examples
A home improvement retailer follows strict frequentist methodology for checkout flow experiments. They pre-register all tests, use fixed sample sizes calculated via power analysis, and only evaluate results after the predetermined sample is collected. Their experimentation program has a 90% post-launch validation rate, meaning 9 out of 10 "winners" actually improve metrics after full rollout.
A B2B analytics platform uses frequentist sequential testing (with alpha spending) to monitor pricing experiments in real time while maintaining valid error rates. This lets them stop a test early if a pricing variant is clearly hurting revenue, without inflating false positive risk.
How to Track in KISSmetrics
KISSmetrics uses frequentist methods for experiment analysis by default. Follow the standard protocol: define your hypothesis and sample size before the test, run the test to completion without peeking, and interpret results using p-values and confidence intervals. The experiment dashboard shows statistical significance and estimated effect size.
Common Mistakes
- -Peeking at results before the predetermined sample size is reached, which inflates false positive rates
- -Not pre-registering the primary success metric and test duration, enabling post-hoc rationalization
- -Misinterpreting p-values as the probability the variant is better (they are not)
- -Running tests without power analysis, resulting in underpowered experiments that waste traffic
- -Ignoring multiple comparison corrections when testing several variants or metrics simultaneously
Pro Tips
- +Pre-register your experiment with hypothesis, primary metric, sample size, and significance level before launching
- +If you absolutely must monitor results during the test, use sequential testing methods (alpha spending) that adjust for repeated looks
- +Complement frequentist results with effect size estimates and confidence intervals for richer decision-making
- +Keep an experiment log that records the pre-registered plan and actual results to build organizational learning
Related Terms
Bayesian Testing
An experimentation approach that uses Bayes' theorem to calculate the probability that one variant is better than another, incorporating prior knowledge and updating beliefs as data accumulates.
Hypothesis Testing
A statistical method used to determine whether observed differences in data - such as a higher conversion rate in a test variant - are likely real or could have occurred by random chance.
P-Value
A p-value is the probability of seeing a result at least as extreme as the one you measured, assuming the null hypothesis of no real difference is true. You calculate it by converting the observed difference into a test statistic such as a z-score, then reading the tail area beyond it. A p-value below 0.05 is the conventional threshold for calling a result statistically significant.
Confidence Interval
A range of values that likely contains the true effect of a change, calculated from experiment data. A 95% confidence interval means that if the experiment were repeated many times, 95% of the calculated intervals would contain the true value.
Type I Error
A false positive in hypothesis testing - incorrectly rejecting the null hypothesis and concluding that a change had a real effect when the observed difference was actually due to random chance.
Build your business intelligence layer for free.
KISSmetrics records the variant as a property on the person, so every other report splits by variant without extra setup.