How do you run a proper A/B test on ads and landing pages?

To run a proper A/B test on ads or landing pages, write a hypothesis, pick one success metric, calculate how many visitors or clicks each version needs, split traffic randomly and evenly, and run the test for whole weeks until that sample is reached. Then read the result once, against the plan you wrote before launch.

Most bad tests fail on the last step. Someone checks the results every morning, sees version B ahead with "95% significance" on day four, and ships it. Evan Miller's widely cited essay "How Not To Run an A/B Test" explains why that is a trap: significance figures assume you decided the sample size in advance, and repeatedly peeking and stopping on the first significant reading inflates the false positive rate well beyond what the dashboard reports. Everything below is designed to avoid fooling yourself.

What is an A/B test?

An A/B test is a controlled experiment that shows two versions of something, the control (A) and a variant (B), to randomly assigned groups of people at the same time, and compares an outcome such as conversion rate. Random assignment is what makes it a test: because both groups are alike on average, a difference in outcome can be credited to the change rather than to timing, audience or luck, within a known margin of error.

A before and after comparison is not an A/B test. If you change a landing page on Monday and compare this week with last week, seasonality, budget changes, competitor activity and algorithm updates are all mixed into the result.

How do you write a good A/B test hypothesis?

A useful hypothesis names the change, the metric, the expected size and the reason. A simple template:

Because [evidence], we believe that [change] for [audience] will [increase or decrease] [primary metric] by at least [minimum effect]. We will know after [sample size] visitors per variant.

For example: "Because session recordings show most mobile visitors never scroll to the form, we believe moving the form above the fold on the demo page will raise mobile form completion from 3.0% to at least 3.6%." The evidence part keeps you testing ideas with a reason behind them, and the minimum effect feeds straight into the sample size calculation.

What should you test first on ads and landing pages?

Asset	High impact tests	Usually low impact
Search ads	Offer or value proposition in headlines, price or proof points, matching the landing page promise	Swapping synonyms, punctuation
Social ads	Creative concept, first three seconds of video, audience pain point	Small color changes
Landing pages	Headline and offer, form length, proof near the call to action, page that matches ad intent	Button color, minor copy edits
Campaign settings	Bidding strategy, match types, audience expansion	Several settings changed at once

Bigger, bolder changes produce bigger effects, and bigger effects need much smaller samples to detect. On low traffic, testing a new offer is realistic; testing a button color is not. Our guide to landing pages that convert paid traffic lists the page elements that usually matter most.

How big does the sample need to be?

Sample size depends on three inputs: your baseline conversion rate, the smallest effect you care about detecting (the minimum detectable effect) and how much risk of a wrong answer you accept. The usual defaults are a 5% significance level, meaning a 5% chance of declaring a difference that is not real, and 80% power, meaning an 80% chance of detecting a real effect of the size you specified.

A common rule of thumb for those defaults is that each variant needs roughly 16 × p × (1 − p) ÷ d² visitors, where p is the baseline conversion rate and d is the absolute difference you want to detect. The table applies it to a few scenarios:

Baseline conversion rate	Lift to detect	Absolute difference	Visitors per variant (approx.)
2%	+50% (to 3%)	1.0 point	3,100
2%	+20% (to 2.4%)	0.4 points	19,600
5%	+20% (to 6%)	1.0 point	7,600
5%	+10% (to 5.5%)	0.5 points	30,400
10%	+10% (to 11%)	1.0 point	14,400

Klepha calculation using the rule of thumb above for a two-sided test at 5% significance and 80% power. Dedicated calculators give slightly different figures; use one for real tests.

The pattern matters more than the exact numbers. Halving the effect you want to detect roughly quadruples the sample you need. A page with 2% conversion and 1,000 visitors a week cannot reliably detect a 20% lift in any reasonable time, so either test a bolder change or measure a metric that happens more often, such as clicks to the form, and confirm the downstream effect later.

How long should an A/B test run?

Divide the total sample by your weekly traffic to get the number of weeks, then round up to whole weeks. Running in full weeks matters because weekday and weekend visitors often behave differently, and a test that covers four weekdays and no weekend measures a skewed audience. As a working rule:

• At least one full week, and ideally two, even if the sample is reached sooner.
• Long enough to cover your sales cycle if the metric is a lead or purchase that happens days after the click.
• Not much longer than about four to six weeks, because cookies expire, people switch devices and the outside world changes, which all add noise.
• Avoid unusual periods, such as a major sale or a Google core update rollout, when traffic mix shifts for reasons unrelated to your change.

Write the stop date down before you launch. If the calculated length is many months, the test is not feasible as designed.

How do you set up ad and landing page tests?

Google Ads

Google Ads custom experiments create a trial version of a campaign that "shares your original campaign's traffic (and budget)," so both versions run over the same dates. Google notes that custom experiments are available for Search, Display, Video and Hotel campaigns, not App or Shopping, and that while you can schedule up to five experiments, "you can only run one experiment at a time." For Search campaigns you can choose a cookie-based split, which keeps each user in one version, or a search-based split. Google also warns that editing the original campaign during the experiment makes results harder to interpret, so freeze other changes.

For copy tests, Google's ad variations feature tests a text change across many ads at once. For bigger structural changes such as match types or bidding, use a custom experiment. If you are deciding whether to widen match types to reach multiturn ads in AI Mode, an experiment is the cleanest way to see the cost.

Landing pages

For page tests, use a testing tool or your site platform to split visitors randomly and keep each visitor in the same version on return visits. Two practical checks: make sure both versions load equally fast, since a slower variant loses for reasons unrelated to its content, and make sure your analytics records which version each visitor saw. If you send paid traffic to two different URLs, tag both with consistent UTM parameters so the results do not split across reports.

Social and other ad platforms

Most major ad platforms offer a built-in split test that divides the audience so the same person does not see both versions. Use it rather than running two ad sets side by side, which compete in the same auction and do not produce a random split.

How do you read A/B test results honestly?

1. Read once, at the planned end. Look at interim numbers only to catch broken tracking or a variant that is clearly harming the business.
2. Look at the confidence interval, not only the p-value. "B converted 12% better, with a plausible range of 2% to 22%" is more useful than "B won."
3. Check the sample split. If you planned 50/50 and got 54/46, something in the setup is wrong; investigate before trusting the result.
4. Check guardrail metrics. A variant that raises leads but lowers lead quality, average order value or retention is not a win. Our guide to CAC and LTV shows how to weigh those.
5. Treat segments as new hypotheses. If B won on mobile but not desktop, that is a candidate for the next test, not a finding, because slicing results many ways will always turn up some "significant" segment by chance.
6. Accept inconclusive results. No significant difference means the effect, if any, is smaller than your test could detect. Keep the simpler version and move on.

Keep a log of every test: hypothesis, dates, sample, result and decision. Over a year, that log is worth more than any single winner, because it tells you which kinds of changes actually move your numbers.

Which traps create false winners?

Trap	Why it misleads	Prevention
Peeking and stopping early	Repeated checks inflate false positives beyond the stated significance level	Fix the sample size in advance, or use a sequential or Bayesian method designed for monitoring
Testing many variants at once	Each extra comparison adds another chance of a fluke winner	Limit variants, or adjust for multiple comparisons
Changing several things in one variant	You cannot tell which change caused the effect	Test one idea at a time, or accept you are testing a whole concept
Novelty effect	Returning visitors react to anything new, then revert	Run at least two weeks and compare new versus returning visitors
Editing during the test	The test measures a moving target	Freeze budgets, bids and pages for the test period
Optimizing a proxy metric	Click-through rate rises while sales do not	Choose a primary metric close to revenue

Testing is one part of a wider growth system that connects paid, organic and AI search visibility. Our solutions, expertise and case studies pages explain how we run that work, and our guide to Google Ads on a small budget covers what to do when traffic is too low for formal tests.

Sources

• Google Ads Help, "Set up a custom experiment", accessed November 11, 2026. support.google.com
• Google Ads Help, "Set up ad variations", accessed November 11, 2026. support.google.com
• Evan Miller, "How Not To Run an A/B Test", on repeated significance testing and fixed sample sizes, accessed November 11, 2026. evanmiller.org
• Klepha, sample size table calculated November 11, 2026 with the approximation n = 16p(1 − p)/d² per variant (two-sided, 5% significance, 80% power).

Frequently asked questions