A/B testing (also called split testing, bucket testing, or split-run testing) compares two versions of a webpage, email, or app against each other to determine which performs better against a specific goal. It eliminates guesswork by showing different variants to randomized user groups and measuring actual behavior rather than opinions. For SEO practitioners and marketers, it provides the data to defend changes to stakeholders and avoid risky site-wide rollouts based on hunches.
What is A/B testing?
A/B testing is a randomized controlled experiment that applies statistical hypothesis testing to compare multiple versions of a single variable. One group sees the original version (the control or A), while another sees the modified version (the variation or B). Traffic splits randomly between the two to ensure unbiased results.
The method extends beyond two variants (A/B/n testing) and differs from observational or quasi-experimental approaches because it requires random assignment. Simple A/B tests isolate one element at a time, such as a headline color or call-to-action text, while keeping all other page elements identical.
Why A/B testing matters
- Reduces financial risk. Testing a blue button against a yellow button with a 50/50 traffic split cuts the risk of exposing your entire audience to the less effective experience by half.
- Drives revenue quickly. A Microsoft employee testing advertising headline formats on Bing produced a revenue increase of 12% within hours without negatively impacting user-experience metrics (Harvard Business Review).
- Enables data-driven culture. Organizations like Google and Microsoft each conduct over 10,000 A/B tests annually (Wikipedia), treating every feature launch as a testable hypothesis rather than a subjective decision.
- Supports segmentation. While one variant may win overall, another might perform better for specific segments. Proper segmentation strategies can yield significant uplifts; for example, targeting specific variants by gender increased expected response rates by 30% in documented cases (Wikipedia).
- Validates at scale. Google conducted 17,523 live traffic experiments resulting in 3,620 launches in 2019 alone (Salesforce).
How A/B testing works
Follow this framework to run valid experiments:
- Identify the problem. Use analytics to find high-traffic pages with high drop-off rates or underperforming conversion points.
- Set a SMART goal. Define a Specific, Measurable, Achievable, Relevant, and Time-based objective with a baseline metric (for example, reducing bounce rate from 40% to 20%).
- Form a hypothesis. State clearly why you believe the change will improve results. For example: "Changing the CTA button to high-contrast orange will increase click-throughs because it draws more attention."
- Create variations. Build the control (A) and variation (B). Change only one isolated element for standard A/B tests to ensure clean attribution.
- Split traffic randomly. Assign users to control or variation using software that ensures random selection to avoid bias. For high-risk changes, start with a small percentage (for example, 90/10) rather than 50/50.
- Run the experiment. Set a predetermined duration based on required sample size. Do not stop early to peek at results, as this introduces false positives.
- Analyze for statistical significance. Use appropriate statistical tests (such as Welch's t-test for means or Fisher's exact test for click-through rates) to determine if differences are real or due to chance.
- Implement or iterate. If the variation wins, deploy it widely. If not, document learnings and test the next hypothesis.
Types of A/B testing
Choose the approach that matches your traffic volume and question complexity.
| Type | What it tests | When to use | Traffic requirements |
|---|---|---|---|
| Standard A/B | One element change (for example, button color) | Validating single hypotheses | Standard site traffic |
| A/B/n | Multiple variants (A vs. B vs. C vs. D) | Testing several headline or image options | Higher traffic (splits among more groups) |
| Split URL | Different URLs entirely (for example, /freebies vs. /resources) | Major redesigns or different user flows | Sufficient to measure per URL |
| Multivariate (MVT) | Multiple variables simultaneously to find combinations | Refining layouts after A/B testing identifies winning elements | Very high traffic (combinations multiply quickly) |
| Multi-page | Consistent changes across a funnel (for example, checkout flow) | Testing user journey consistency or sitewide banners | Substantial traffic across multiple pages |
Best practices
Test one element at a time. Isolating variables ensures you know exactly what caused the change in performance. Testing a completely different page design against the original makes attribution impossible.
Define statistical significance before launching. Determine your confidence threshold (commonly 90% or 95%) upfront. This prevents rationalizing random noise as wins.
Use 302 redirects for URL tests. If your test redirects users from the original URL to a variation, use a 302 (temporary) redirect rather than a 301 (permanent). This signals search engines to keep the original URL indexed (Optimizely).
Add rel="canonical" to variations. When running split URL tests, place the rel="canonical" attribute on variation pages pointing back to the original to prevent search engines from indexing test URLs as duplicate content.
Segment only with sufficient traffic. Breaking results down by device, geography, or user type requires large sample sizes for each segment. Start with common splits like new versus returning visitors rather than creating numerous small audiences that generate false positives.
Document test plans comprehensively. Record the problem, hypothesis, primary metric, audience definition, lever (what changes), and duration before building. This prevents team collisions and ensures alignment.
Common mistakes
Peeking at early data. Stopping a test as soon as results look favorable produces false positives. Set your sample size and duration based on power calculations, then let the test run to completion.
Testing without randomization. Assigning users based on time of day, geography, or other non-random factors introduces selection bias that invalidates results. Use software that ensures true random assignment.
Insufficient sample size. A/B tests are sensitive to variance. Low-traffic sites may need to run tests for weeks or use variance reduction techniques like CUPED (Controlled Experiment Using Pre-Experiment Data), a method developed by Microsoft to reduce required sample sizes (Wikipedia).
Cloaking to search engines. Showing Googlebot different content than users see to manipulate rankings violates search guidelines and risks demotion or removal from search results. Test tools must not segment by user-agent or IP to show different content to crawlers.
Using 301 redirects for temporary tests. Permanent redirects tell search engines to transfer ranking signals to the test URL, which can cause indexation problems when the test ends.
Looking at too many metrics. Focusing on secondary metrics (time on page, pages per session) instead of your primary success metric (conversions, revenue per visitor) leads to confusion and false discoveries.
Examples
Bing revenue optimization. A Microsoft employee created an experiment to test different methods of displaying advertising headlines on Bing. The alternative format produced a revenue increase of 12% within hours while maintaining user-experience metrics (Harvard Business Review).
Optimizely homepage engagement. The digital team tested adding an interactive "pet the dog" element on their homepage. Visitors who saw the dog consumed content at 3x the rate of those who did not see the element (Optimizely).
Obama 2008 campaign optimization. The presidential campaign used A/B testing on their website to optimize newsletter signups. They tested four distinct button styles and six different accompanying images to determine which combinations drove the highest conversion rates (Wikipedia).
A/B testing and SEO
Google permits and encourages A/B testing, stating that proper experimentation poses no inherent risk to search rankings (Optimizely). However, violations of testing protocols can harm SEO:
- Never cloak. Do not segment traffic to show Googlebot different content than human users based on user-agent or IP address.
- Use rel="canonical" tags. Point all variation URLs back to the original page using the canonical attribute to consolidate indexing signals.
- Use 302 temporary redirects. When redirecting the original URL to a variation, use 302 status codes to indicate the redirect is temporary.
FAQ
How long should an A/B test run?
Run tests for a predetermined duration based on required sample size calculations, typically at least one to two weeks to account for day-of-week effects. Do not stop early when results appear significant.
What is statistical significance?
Statistical significance measures the probability that observed differences between your control and variation are real and not due to random chance. A 95% confidence level means you can be 95% certain the result is repeatable.
Can A/B testing hurt my search rankings?
Proper A/B testing does not hurt SEO. Google explicitly allows testing. However, cloaking (showing different content to search engines than users) or using 301 permanent redirects for temporary tests can cause indexing issues.
What is the difference between A/B testing and multivariate testing?
A/B testing changes one element at a time to isolate its impact. Multivariate testing changes multiple elements simultaneously to analyze interaction effects between variables. MVT requires substantially more traffic to achieve statistical significance for each combination.
How many variations should I test at once?
For standard A/B testing, start with one variation against the control. If traffic supports it, A/B/n testing allows multiple challengers, but remember that each additional variant dilutes your sample size and extends required test duration.
What sample size do I need?
Required sample size depends on your baseline conversion rate, the minimum detectable effect you want to measure, and your desired statistical power. High-variance metrics or small expected improvements require larger samples. Microsoft's CUPED technique can reduce required sample sizes by accounting for pre-experiment variance.
