Sample Size Calculator for Conversion A/B Testing

Use it when you are going to compare variants and need to know whether the difference between them is real:

You have a baseline conversion rate measured on the control variant (not estimated from memory)

You can name the minimum effect that would justify the change (MDE): how much it has to improve for shipping it to be worth it

Traffic accumulates over time: there is no closed population capping how many people can enter the test

The metric is a proportion (conversion, clicks, activation). If you are comparing times or SUS scores, use the calculator's other A/B testing variants.

If instead you want to estimate a value in a population — how many people prefer X — the question is a survey, not an experiment.

The standard formula for comparing two proportions (Kohavi et al., 2009; Cohen, 1988) is:

n = 2 * (Za + Zb)^2 * p * (1-p) / D^2

n = Sample per variant | Za = Z of the confidence level | Zb = Z of the power

p = Baseline conversion as a proportion | D = Minimum detectable effect (MDE) as a proportion

The result is per variant. An A/B test needs 2n people in total; a three-variant test, 3n. It is the most expensive misreading of this formula.

The relationship with MDE is quadratic: detecting half the effect costs four times the sample. That is why dropping the MDE from 10% to 5% does not double the test's cost but quadruples it.

A notable absence: population size does not appear. There is no finite population correction because you are not sampling a closed universe.

Baseline conversion (p): measure it, do not estimate it. With a low baseline (1-2%) the sample explodes, and that figure is what decides whether the test is viable before you design it.

MDE: it is a business decision, not a statistical one. The right question is not 'what improvement do I expect' but 'what improvement would be so small that I would not ship the change'. That threshold is your MDE.

Relative vs absolute MDE: 10% relative on a 5% baseline is 0.5 points; 10% absolute is 10 points. The difference between the two is two orders of magnitude in required sample. The calculator lets you pick which one you are using.

Confidence (95% by default): controls the false positive — declaring a winner that is not one.

Power (80% by default): controls the false negative — failing to detect an improvement that is real. Raising it to 90% is defensible when the cost of missing the improvement is high, but it makes the sample more expensive.

Peeking at the test and stopping when you see significance: the most common error and the most expensive. Checking daily and stopping when the p-value crosses 0.05 inflates the false positive rate far above the declared 5%. The size this calculator gives you is a commitment made up front, not a target to beat early.

Cutting mid-week: a Tuesday does not behave like a Sunday. Round the duration up to full weeks even when the sample is already covered.

Evaluating many metrics and reporting the one that came out well: each additional metric is another chance at a false positive. Define the primary metric before launching.

Confusing significance with relevance: with a large enough sample, an irrelevant difference comes out significant. That is why the MDE is set beforehand: it is your relevance threshold.

Extending the test until it wins: if the result is inconclusive at the planned sample, the finding is that the effect is smaller than the MDE you defined. That is information too.

Most of the products we work with in LatAm do not have the traffic to detect small improvements. With a 3% baseline conversion and 500 daily visits, detecting a 5% relative improvement takes more than a year. That is not a limitation of the method: it is the method telling you that test cannot be run.

The way out is not lowering power or confidence so the number fits the calendar. That does not shorten the test, it only raises the odds of making a wrong decision with the same air of certainty.

Test bigger changes. If you can only detect effects of 20% or more, test full flow redesigns, not copy variations on a button.

Move up the funnel. A metric more frequent than purchase — add to cart, start signup — has a higher baseline and needs less sample, as long as you can argue it moves together with final conversion.

Change methods. When the experiment is not viable, five well-run usability sessions answer more than an underpowered A/B test. The choice of method comes before the choice of size.