P-Value
A p-value is the probability of seeing a result at least as extreme as the one you measured, assuming the null hypothesis of no real difference is true. You calculate it by converting the observed difference into a test statistic such as a z-score, then reading the tail area beyond it. A p-value below 0.05 is the conventional threshold for calling a result statistically significant.
Also known as: p value, probability value, significance level result
Formula
Z = (p1 - p2) / sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
Why p-value matters
The p-value is the most commonly used (and misunderstood) statistic in experimentation. It answers a specific question: "If there were truly no difference between control and variant, how likely would I be to see a result this extreme?" A p-value of 0.03 means there is a 3% chance of seeing this result by random chance alone.
P-values help you make go/no-go decisions about test results. The conventional threshold of 0.05 (5%) means you accept a 1-in-20 chance of a false positive. For high-stakes decisions, you might use 0.01 (1%). For exploratory tests, 0.10 (10%) might be acceptable.
However, p-values do not tell you the magnitude of the effect, the probability that your hypothesis is true, or whether the result is practically meaningful. A p-value of 0.001 for a 0.01% conversion rate improvement is statistically significant but practically worthless. Always pair p-values with effect size and confidence intervals.
The newsletter
Join our KISS newsletter
One short read a week on what actually moves revenue, in a free email. Read by 10,000+ operators and founders.
No spam. Unsubscribe in one click.
How to calculate p-value
Calculating a p-value for an A/B test takes three steps. First, turn the observed difference into a test statistic. For two conversion rates that is a z-score: pool the groups so p_pool = (conversions1 + conversions2) / (n1 + n2), then Z = (p1 - p2) / sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2)). Second, find the tail area of the standard normal distribution beyond that Z. Third, double it, because you almost always want a two-tailed test.
Worked example. Control converts 500 of 10,000 visitors (5.0%). The variant converts 570 of 10,000 (5.7%). p_pool = 1,070 / 20,000 = 0.0535. The standard error is sqrt(0.0535 * 0.9465 * (1/10,000 + 1/10,000)) = 0.00318. Z = 0.007 / 0.00318 = 2.20. The one-tailed area beyond Z = 2.20 is 0.0139, so the two-tailed p-value is about 0.028. That is below 0.05, so the result is significant at the conventional threshold.
In a spreadsheet the last two steps are one function: =2*(1-NORM.S.DIST(ABS(Z),TRUE)). For raw counts you can skip the z-score entirely and run CHISQ.TEST on a 2x2 table of conversions and non-conversions, which returns the p-value directly. Continuous metrics such as revenue per visitor use a t-test instead of a z-test, because the variance has to be estimated from the sample.
Z-Test P-Value Approximation Calculator
Z = (p1 - p2) / sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
P-Value examples
A DTC brand tests two pricing strategies. The test shows p = 0.02 for revenue per visitor, indicating strong evidence that the new pricing outperforms the old. Combined with a 12% effect size and positive confidence interval, they confidently roll out the change.
A B2B tool tests three different onboarding emails. After applying Bonferroni correction for multiple comparisons (threshold of 0.017 instead of 0.05), only one variant achieves significance with p = 0.008, avoiding a false positive from testing multiple variants.
How to Track in KISSmetrics
KISSmetrics reports statistical significance in the A/B Test report alongside the observed lift and the sample size behind it, so you are reading the p-value and the effect size together rather than in isolation. Monitor it as the experiment accumulates data, but resist the urge to stop the test the moment it crosses 0.05. Pre-commit to a sample size and run the test to completion, or switch to a sequential testing method that is designed for continuous monitoring. If the metric the test is measured on does not exist yet, you can describe it in the AI chat and have it built before the experiment starts, which keeps the experiment and the business metric on one definition.
Common Mistakes
- -Interpreting the p-value as the probability that the variant is better - it is not; it is the probability of the observed data given no real difference
- -Peeking at p-values during a test and stopping early when they cross the threshold, which inflates false positive rates
- -Treating p = 0.05 as a hard boundary where 0.049 means "significant" and 0.051 means "not significant" - the difference is trivial
- -Ignoring effect size and focusing solely on whether the p-value crossed the threshold
- -Running multiple comparisons without adjusting the p-value threshold (Bonferroni correction or similar)
Pro Tips
- +Always report p-values alongside confidence intervals and effect sizes for a complete picture
- +Use sequential testing methods if you need to monitor results during the test without inflating false positive rates
- +Adjust your significance threshold based on the cost of a false positive - higher stakes warrant lower p-value thresholds
- +Remember that a high p-value does not prove no effect; it may indicate an underpowered test
- +Consider using Bayesian methods for a more intuitive interpretation of experiment results
Related Terms
Null Hypothesis
The default assumption in a statistical test that there is no meaningful difference between the control and test groups - any observed difference is due to random chance rather than a real effect.
Confidence Interval
A range of values that likely contains the true effect of a change, calculated from experiment data. A 95% confidence interval means that if the experiment were repeated many times, 95% of the calculated intervals would contain the true value.
Confidence Level
The percentage probability that a confidence interval calculated from a given experiment will contain the true population parameter, commonly set at 90%, 95%, or 99% in A/B testing.
Type I Error
A false positive in hypothesis testing - incorrectly rejecting the null hypothesis and concluding that a change had a real effect when the observed difference was actually due to random chance.
Hypothesis Testing
A statistical method used to determine whether observed differences in data - such as a higher conversion rate in a test variant - are likely real or could have occurred by random chance.
Further Reading
A/B Testing Statistical Significance: When to Call a Winner
Understand statistical significance in A/B testing. Learn sample size calculation, confidence intervals, and why stopping tests early gives false results.
Introduction to A/B Testing: How to Run Experiments That Actually Work
A beginner-friendly guide to A/B testing covering hypothesis formation, test design, statistical significance, and common mistakes to avoid.
Build your business intelligence layer for free.
KISSmetrics records the variant as a property on the person, so every other report splits by variant without extra setup.