A/B Testing: The Complete Guide to Tests That Hold Up

A/B testing: the short answer

A/B testing is a research method in which you show two versions of the same page, email or advert to randomly split visitors at the same time and measure which version produces more conversions. Version A is your current page, version B your change. Only when the difference is statistically significant do you know it was not chance.

This guide is about the method itself: how to design a test that holds up. The wider discipline around it lives in conversion rate optimization, and what a single page can win for you sits in landing page design. Here, everything is about the experiment.

What is a/b testing, exactly?

An a/b test is a randomised controlled experiment. You split your traffic randomly into two groups: group A sees the existing version, group B sees the variant with one change. Both groups visit your site in the same period, under the same conditions, and you measure how many visitors in each group complete the action you care about.

That randomisation is the entire point. Without a control group you are comparing this month with last month, which means you are also measuring the weather, a campaign that ended and a competitor who cut prices. An a/b test isolates exactly one variable.

You will see the method written as a/b testing, as ab testing without the slash, and as split testing. Oracle and Optimizely also use the older term bucket testing. It is all the same experiment.

What should you test, and what wastes your traffic?

Test things where the outcome changes your behaviour. The rule of thumb: if you would do the same thing either way, you do not need to run the test.

What wastes your traffic: button colours, corner radii, background shades and other cosmetic details. They rarely shift behaviour enough to rise above the noise, and you burn weeks of visitors finding that out. The famous story about testing forty shades of blue at Google involves a company with billions of sessions a day. You do not have that traffic.

From data to hypothesis: the only good starting point

Every test starts with an observation in your data, not with an idea in a meeting. Look at where visitors drop off, which page has high traffic and low conversion, and which step in your form loses the most people.

Then translate that observation into a hypothesis with a fixed shape:

An example. Because we observe that 68 percent of visitors open the quote form but never submit it, we expect that removing the company name and job title fields will increase completed submissions, because every extra field adds friction on mobile. We will measure completed requests per visitor who reaches the form page.

That last sentence does more work than it looks. You are committing in advance to the metric that counts. Without it you pick the winning metric after the fact, and that is no longer research.

Statistical significance without the maths anxiety

Statistical significance answers one narrow question: how likely is it that you would see this difference if there were genuinely no difference? The standard threshold is 95 percent confidence, which means you accept a 5 percent chance of declaring a winner that does not exist.

Three numbers get fixed before you start.

That third number is where most people go wrong. You are not deciding how big the effect will be. You are deciding how small an effect you are willing to miss. The smaller that number, the more visitors you need.

How many visitors does an a/b test need?

There is a usable rule of thumb, known in the literature as the rule of 16 and set out by Ron Kohavi, Diane Tang and Ya Xu in Trustworthy Online Controlled Experiments (2020). At 95 percent confidence and 80 percent power, the visitors needed per variant is roughly 16 times the variance of your metric, divided by the square of the absolute difference you want to prove.

For a conversion rate that variance is simply the conversion rate times one minus the conversion rate. Filled in for a 20 percent relative lift, it looks like this:

The relationship is quadratic, and that is the most underestimated fact in this whole field. If you want to detect a 10 percent lift instead of a 20 percent lift, you do not need twice as many visitors, you need four times as many. At a 2 percent baseline that works out at roughly 78,400 visitors per variant.

The honest warning: most small sites never reach significance

Run that table against your own site. A B2B firm with 3,000 relevant monthly visitors on the tested page and a 2 percent conversion rate needs roughly 39,200 visitors. That is thirteen months for one test. In thirteen months your offer, your traffic and your market all change, so the result is worthless before it arrives.

Here is the sentence almost no agency writes down: below roughly ten thousand relevant visitors a month, classic a/b testing is rarely the right instrument. That is not a reason to stop improving. It is a reason to pick different instruments.

How long should you run an a/b test?

Calculate your sample size in advance, divide it by your daily traffic, and round up to whole weeks. Those three steps solve most of the problems.

• Always run whole weeks. Tuesday behaviour differs from Sunday behaviour. Stop on a Wednesday and you have one and a half weekday cycles in your data, so you are partly measuring the day of the week.
• Two full weeks is a practical minimum, even when your sample fills sooner. That way you capture at least one complete purchase cycle.
• Beyond four to six weeks it gets dangerous. Visitors clear cookies and return as new participants, seasonality creeps in, and your site changes underneath the test.
• With long decision cycles, wait until the conversions that started in week one have genuinely had their chance to complete.

The mistakes that make a/b testing worthless

The first one deserves an explanation, because it is genuinely treacherous. It is called peeking: you check your dashboard every morning and stop the moment the tool turns green. In his classic piece How Not to Run an A/B Test, Evan Miller showed that monitoring continuously and stopping at the first significant moment pushes the real false positive rate to roughly 26 percent, instead of the 5 percent you think you are accepting. More than one in four winners found that way does not exist.

The fix is boring and effective: fix your sample size in advance, agree the end date, and look on that date. Tools built on sequential or Bayesian analysis, which most modern platforms now offer, are designed to allow monitoring along the way. If you are not using one, the boring rule applies.

A/B test, multivariate test or split-URL test?

Split-URL tests run into SEO. Google covers this explicitly in its website testing documentation: put a rel=canonical on your test variants pointing at the original URL, use a temporary 302 rather than a permanent 301 for redirects, and never serve Googlebot a different version than your visitors, because that is cloaking. Also, do not keep the test running longer than you need.

A/B testing tools in 2026

Google Optimize was switched off on 30 September 2023, along with Optimize 360 and every integration. That removed the free entry point a great many small and mid-sized sites relied on, which is why you still stumble across outdated tutorials in 2026.

What is left falls into roughly three groups.

• Classic experimentation platforms: VWO, Optimizely, AB Tasty, Kameleoon and Convert Experiences. Visual editor, built-in statistics, aimed at marketing teams.
• Feature-flag platforms with experiments built in: Statsig, GrowthBook, PostHog and LaunchDarkly. Developer-first, server-side testing, and GrowthBook and PostHog both offer an open-source edition you can host yourself.
• Testing inside the channel itself: Google Ads experiments, A/B tests in Meta Ads, and the built-in split testing in email platforms such as Mailchimp and Klaviyo. You do not need a separate platform to test a subject line.

Pricing runs from free and self-hosted to enterprise contracts, and it changes every year. Always ask for a current quote based on your monthly visitor volume rather than trusting a blog post. What you can settle now: choose client-side for fast marketing tests, and server-side when the test touches pricing, stock or logged-in functionality.

Why a/b testing is pointless on a slow or badly built site

This is where we most often part company with the rest of the market. A/B testing assumes your foundation is sound. If it is not, you are measuring noise with expensive software.

Three conditions come before your first test. First, speed: if your largest element only appears after four seconds, you lose visitors before they ever see your variant, and you are testing two versions of the same problem. Why that hits so hard is covered in why speed decides and in Core Web Vitals. Second, measurability: if your conversion goals do not record reliably, every result is fiction. Third, a page that already does the basic job, with a clear offer and a visible action. That is design, not experimentation, and it sits in web design trends.

At Viralistic we build sites that are fast and measurable before a single test runs, with findability, design and engineering under one lead. For SilverDrive, an Amsterdam chauffeur service that grew internationally on organic findability, that order of operations returned more than any button-colour test could. Joshua Faerber of SilverDrive put it this way (translated from the Dutch original): “Viralistic guided SilverDrive towards becoming an internationally focused company that runs entirely on organic findability. The website and all the systems save us an enormous amount of time.”

Where a/b testing fits in the bigger picture

An a/b test is the last step in a cycle, not the first. You measure where the leak is, you write a hypothesis, you test it if you have the traffic, and you ship what wins. Then you start again at the next biggest opportunity.

What you learn from tests is also more portable than the single page you found it on. If a more concrete headline wins on your quote page, that is a signal about how your audience reads, and you apply it to your adverts, your emails and the rest of your site. The structural side of that lives in conversion rate optimization and getting more from your website.

Frequently asked questions about a/b testing

What is a/b testing?

A/B testing is a randomised experiment in which you show two versions of a page, email or advert to randomly split visitors at the same time and measure which one produces more conversions. It is also called split testing or, without the slash, ab testing. The question what is a/b testing usually gets an answer that is too simple: it is not just about having two versions, it is about random assignment and a result that survives statistical scrutiny.

What is an a/b test used for?

For improving conversion on websites, landing pages, emails, adverts and apps in a way you can defend. You replace an opinion about what works with a measurement. Companies such as Netflix and Booking.com publish openly about their experimentation culture, but the method works at any scale where the traffic exists.

How long should you run an a/b test?

Calculate the sample size you need in advance, divide it by your daily traffic and round up to whole weeks. In practice that means a minimum of two full weeks and preferably no more than four to six, because cookie loss and seasonality start to contaminate your data after that.

Is a/b testing the same as hypothesis testing?

At its core, yes. An a/b test applies classic hypothesis testing to web traffic: your null hypothesis is that there is no difference between A and B, and you only reject it when the observed outcome is unlikely enough under that assumption. The difference is the context, not the statistics.

When should you not use ab testing?

When you have too little traffic, when your site is too slow to measure reliably, when the decision has already been made, and when the change has to happen anyway, such as a legal requirement. With rare conversions too, think a handful of signed contracts a month, interviews and session recordings get you further faster.

What are the limitations of a/b testing?

It costs traffic and time, it only measures what you thought of in advance, and it tells you nothing about why something works. It also captures short-term behaviour: a variant that wins clicks today can damage your brand over a year. Always pair quantitative testing with qualitative research.

Which tool should you use for a/b testing?

Since Google Optimize was retired on 30 September 2023, teams choose between classic platforms such as VWO, Optimizely, AB Tasty, Kameleoon and Convert, and developer-first alternatives such as Statsig, GrowthBook and PostHog. For email and adverts, simply use the testing feature built into the channel.