Introduction to A/B Testing: How to Run Experiments That Actually Work

“Every change you make to your website, product, or marketing is a bet. A/B testing replaces those bets with evidence.”

You are betting that the new headline will convert better than the old one, that the redesigned checkout flow will reduce abandonment, or that the updated pricing page will increase revenue. Without a controlled experiment, you are flying blind.

At its core, A/B testing is simple: show two versions of something to two similar groups of people and measure which version performs better. But doing it correctly - in a way that produces reliable, actionable results - requires understanding a handful of important principles that many teams overlook.

This guide covers everything you need to run your first A/B test and avoid the mistakes that lead to false conclusions. Whether you are testing landing pages, email subject lines, or in-product experiences, the fundamentals are the same.

What Is A/B Testing?

A/B testing (also called split testing) is a controlled experiment where you divide your audience into two or more groups and show each group a different version of a page, feature, or experience. Group A sees the original (the “control”). Group B sees the variation (the “treatment”). You then compare the performance of each version against a predefined success metric.

The key word is controlled. Unlike before-and-after comparisons, where you change something and then check whether metrics improved, an A/B test runs both versions simultaneously. This eliminates confounding variables like seasonality, marketing campaigns, or external events that might influence results independently of your change.

For example, suppose you redesign your sign-up form and then observe a 15% increase in registrations the following week. Was it the redesign? Or was it the product launch article that drove extra traffic? Or the holiday weekend that changed user behavior? Without a control group, you cannot know. An A/B test eliminates this ambiguity by comparing the new form against the old form at the same time, with the same traffic mix.

A/B testing applies to virtually anything your users see or interact with: web pages, emails, in-app messages, onboarding flows, pricing tiers, checkout processes, and even backend algorithms that affect the user experience. If it can be varied and measured, it can be tested.

Forming a Hypothesis

The most important step in A/B testing happens before you write a single line of code. A good test starts with a clear hypothesis - a specific, falsifiable prediction about what will happen and why.

The If/Then/Because Framework

Structure your hypothesis using the if/then/because format:

• If we [make this specific change],
• then [this metric] will [increase/decrease] by [estimated amount],
• because [this is the reason we believe it will work].

For example: “If we replace the generic hero image on our landing page with a screenshot of the product dashboard, then our sign-up conversion rate will increase by 10% to 20%, because user research shows that visitors want to see the product before committing to a trial.”

The “because” clause is the most important part. It forces you to articulate your reasoning, which serves two purposes. First, it helps you evaluate whether the test is worth running. If you cannot explain why a change should work, you are guessing rather than testing. Second, it helps you learn regardless of the outcome. If the test fails, the “because” tells you which assumption was wrong.

Where Hypotheses Come From

Strong hypotheses are grounded in data, not opinions. The best sources include: analytics data showing where users drop off or struggle, user feedback from surveys and interviews, support tickets highlighting recurring confusion, session recordings revealing UI problems, and competitive analysis showing what alternatives do differently. Your analytics reports are usually the best starting point for identifying high-impact areas to test.

Avoid the HiPPO trap (Highest Paid Person’s Opinion). Tests motivated by executive preferences rather than evidence fail at the same rate as random changes. The entire point of A/B testing is to let data override opinion.

Designing Your Test

Good test design ensures that your results are valid and interpretable. Three principles matter most: isolation, randomization, and measurement.

Isolation: Test One Thing at a Time

Each test should change a single variable. If you simultaneously change the headline, the hero image, and the CTA button color, and the variation wins, you cannot know which change drove the improvement. Maybe the new headline was great but the new image was terrible, and the net result was a modest positive.

This does not mean you can only change one element on the page. It means you should change one conceptual variable. Rewriting the entire value proposition of a page (headline, subheading, and supporting copy) is one variable: the value proposition. Changing the value proposition and the layout and the pricing display is three variables.

If you need to test multiple variables simultaneously, use multivariate testing (MVT), which is designed for this purpose but requires significantly more traffic to reach conclusions.

Randomization: Split Traffic Fairly

Users must be randomly assigned to the control or treatment group. This means each visitor has an equal chance of seeing either version, regardless of when they visit, where they come from, or what device they use. Most A/B testing tools handle randomization automatically, but verify that your tool assigns users consistently (the same user always sees the same version) rather than re-randomizing on each page load.

Measurement: Define Success Before You Start

Choose your primary metric before launching the test, and commit to evaluating results based on that metric. Common primary metrics include conversion rate, revenue per visitor, average order value, sign-up rate, or engagement metrics like time on page.

You can (and should) track secondary metrics as well. If your primary metric is sign-up conversion, also monitor downstream metrics like activation rate and 30-day retention. A change that increases sign-ups but decreases activation is not a win - it is attracting lower-quality users. This is why connecting your testing data to downstream metrics is critical for making sound decisions.

Sample Size and Duration

Sample size is the most misunderstood aspect of A/B testing. Running a test for “a few days” and then checking which version is ahead is not a valid approach. You need a statistically adequate sample size to distinguish a real effect from random noise.

Calculating Sample Size

Before launching any test, calculate the required sample size based on three inputs: your current conversion rate (the baseline), the minimum improvement you want to detect (the minimum detectable effect, or MDE), and the level of statistical confidence you require (typically 95%).

As a rough guide: if your current conversion rate is 5% and you want to detect a 10% relative improvement (from 5.0% to 5.5%), you will need approximately 30,000 visitors per variation. If you want to detect a 20% relative improvement (from 5.0% to 6.0%), you need approximately 8,000 per variation.

Smaller effects require larger samples. This is why testing micro-optimizations (button color, font size) is impractical for low-traffic sites. Focus on changes that could plausibly produce large effects: headline rewrites, layout changes, pricing structure modifications, or entirely new features.

How Long to Run a Test

Calculate the required duration by dividing your needed sample size by your daily traffic. If you need 30,000 visitors per variation and your page gets 2,000 visitors per day, the test must run for at least 30 days (15,000 per variation with a 50/50 split is only two weeks, but you need 30,000 per variation).

Always run tests for a minimum of one full business cycle - typically seven days - even if you reach sample size sooner. Behavior on Monday differs from behavior on Saturday. A test that runs from Monday to Thursday may capture a biased sample that does not represent your full audience.

Measuring Results

When your test reaches the required sample size and has run for at least one full cycle, it is time to analyze the results. Focus on three things: statistical significance, effect size, and practical significance.

Statistical Significance

Statistical significance tells you the probability that the observed difference between your control and treatment is real, not due to random chance. The industry standard is a 95% confidence level, meaning there is no more than a 5% chance the result is a false positive.

Most A/B testing tools report a p-value. If the p-value is below 0.05, the result is statistically significant at the 95% confidence level. If it is above 0.05, you cannot conclude that the treatment performed differently from the control.

Effect Size

Effect size is the magnitude of the difference. A statistically significant result that improves conversion from 5.00% to 5.02% is real but probably not worth the engineering effort to implement. Look at both relative improvement (percentage lift) and absolute improvement (percentage point change). Calculate the expected annual revenue impact to determine whether the result justifies implementation.

Practical Significance

Practical significance is a judgment call. Even if a result is statistically significant and the effect size is meaningful, consider the cost of implementation, the maintenance burden of the change, and whether the improvement aligns with your broader product strategy. A 5% lift on a page that affects 1% of your revenue may not be worth prioritizing over other work.

What to Do with Inconclusive Results

Not every test produces a clear winner. If your test finishes without reaching statistical significance, it means the difference between the versions is too small for your sample to detect. This is still useful information: it tells you that the variable you tested does not have a large effect on the metric you measured. Document the result and move on to higher-impact tests.

Common Mistakes That Invalidate Tests

A/B testing is conceptually simple but operationally demanding. The following mistakes are common even among experienced teams, and each one can lead to false conclusions that waste resources or hurt performance.

Stopping Tests Too Early

This is the single most common mistake. You launch a test, check it after two days, see that the variation is winning with “92% confidence,” and declare victory. The problem is that early results are extremely volatile. Statistical significance calculated on small samples fluctuates wildly. A test that shows 92% confidence on day two may show 40% confidence on day five as the sample normalizes.

The fix is simple: calculate your required sample size before launching, commit to it, and do not check results before the test is complete. If you must monitor for catastrophic failures, set up automated alerts for large negative effects (conversion dropping by more than 50%) rather than manually watching the confidence meter.

Testing Too Many Variables

When you change five things at once, you cannot attribute the result to any specific change. Even worse, opposing effects may cancel each other out, producing a null result that hides two important insights: one change that would have been a big win and another that would have been a big loss.

Resist the temptation to bundle multiple changes into a single test “to save time.” You are not saving time; you are destroying information. Run focused tests and build a learning velocity that compounds over months.

Ignoring Segments

Aggregate results can hide segment-level effects. A test that shows no overall difference may be producing a 20% lift for mobile users and a 20% decline for desktop users, netting to zero. Always check results by device type, traffic source, new vs. returning visitors, and any other meaningful segments. Understanding your user populations makes segmented analysis far easier.

However, be cautious about declaring segment-level winners without adequate sample sizes within each segment. If your test had 10,000 total visitors but only 500 were mobile, the mobile segment result is unreliable. Pre-plan your segments and ensure your total sample size is large enough to support the segmentation you want to do.

Running Multiple Tests on the Same Page

If two tests are running simultaneously on the same page, their effects can interact in unpredictable ways. A headline test and a CTA test running together may produce results where the winning headline only wins with the original CTA, and vice versa. Most tools offer mutual exclusion features that prevent users from being enrolled in multiple overlapping tests. Use them.

Not Tracking Downstream Impact

Optimizing a single step in isolation can hurt the overall journey. A more aggressive pop-up might increase email sign-ups by 30% but annoy visitors so much that product conversions drop by 10%. Always measure the impact on your end-to-end funnel, not just the step you are testing.

Beyond the Basics

Once you are comfortable with standard A/B testing, several advanced techniques can increase the speed and depth of your experimentation program.

A/B/n Testing

Instead of testing one variation against the control, test multiple variations simultaneously. This is useful when you have several hypotheses about what might work and want to explore them in parallel. The trade-off is that each additional variation increases the required sample size, since you need adequate traffic in each group.

Sequential Testing

Sequential testing methods (like group sequential designs or always-valid p-values) allow you to analyze results at multiple checkpoints without inflating your false positive rate. This is a rigorous alternative to the “peek and stop” approach that addresses the legitimate desire to end tests early when results are clear.

Personalization Testing

Instead of finding the single best version for everyone, personalization testing identifies the best version for each user segment. This requires more sophisticated tooling and analysis but can produce significantly larger gains than one-size-fits-all optimization. Understanding your user populations is essential for this approach.

Getting Started with Your First Test

If you have never run an A/B test before, start small. Pick a high-traffic page with a clear conversion goal. Formulate a hypothesis using the if/then/because framework. Design a single, focused variation. Calculate the required sample size. Launch the test, wait for it to finish, and analyze the results objectively.

Your first test might not produce a statistically significant result. That is fine. The goal of the first test is to build the muscle: the process of hypothesizing, designing, measuring, and learning. The wins will come as your hypotheses improve through accumulated insight.

The teams that get the most value from A/B testing are the ones that test consistently, learn from every result (including losses and inconclusive outcomes), and use each test to inform the next one. Testing is not a one-time tactic. It is a discipline that compounds over time, turning your product and marketing decisions into evidence-based bets with increasingly favorable odds.

Key Takeaways

A/B testing is the single most reliable way to make product and marketing decisions. Here is what to keep in mind: