Problem Discovery Sample Size Calculator

Use it when the goal is to find and fix, not to measure:

You are running a usability test to detect what breaks the experience, before launch or before a redesign

You have to justify to someone why 5, 8 or 15 sessions rather than a number pulled from the air

You are iterating: test, fix, test again

When not to use it: if you need to say how many people it happens to, how long they take, or how much things improved over the previous version, this calculation will not help. That is a different question with a different calculator — quantitative usability, which starts at 40 participants (Budiu & Moran, 2021).

The difference is not one of rigour but of purpose. A formative test with 8 people is as legitimate as a quantitative one with 40: they answer different things.

The famous number comes from Nielsen and Landauer (1993), who modelled problem discovery as 1 - (1 - p)^n. Across the studies they analysed, the average problem affected 31% of users. Put p = 31% into the formula and 5 sessions give an 84.4% chance of seeing that problem: hence “5 users find 85% of the problems” (Nielsen, 2000).

The assumption is the part that breaks. That 31% is an average across the projects they studied, in the early 1990s. In a mature product, the problems still standing are precisely the ones that do not hit one in three users.

That is where 5 stops working. If the problem affects 10% of people, 5 sessions give a 41% chance of seeing it: you are more likely to miss it than to find it. To reach 85% at that frequency you need 19 sessions.

So p is the decision, not the output. The formula is arithmetic; what moves the number is which kind of problem you declare you are hunting. With no estimate of your own, 20% is the sensible working assumption: it gives 9 sessions for 85% detection.

One note on rounding: the figures in circulation (5, 7, 13) round down and land below the target they claim. This calculator rounds up and shows you the detection you actually get, so you can see the gap instead of inheriting it.

p — problem frequency. The share of users it happens to. You do not know it up front (if you did, you would have fixed it), so it is an explicit bet: below which frequency you accept that a problem may escape this test.

C — detection probability. How sure you want to be of seeing that problem at least once. 85% is the usual benchmark because it is the one the “five users” literature uses; 95% is defensible when missing the problem is expensive, and costs roughly 50% more sessions.

The relationship is not linear. Lowering the frequency you chase raises the cost far faster than raising your certainty: going from p = 20% to p = 10% doubles the sample, whereas going from 85% to 95% detection raises it by roughly half.

Rare problems are not hunted with sessions. If you care about something affecting 2% or 5% of users, the lab is the expensive tool: analytics, support tickets and session recordings find those sooner and cheaper.

The number is not run in one go. Both Nielsen (2012) and Sauro recommend splitting it: 5 sessions, fix whatever surfaces, another 5 on the corrected design. The total ends up similar, but each round tests something better than the last.

A formative session is expensive in time, not in people. Moderated, each participant is 45 to 90 minutes plus analysis. That cost, not the statistical one, is usually what sets the real ceiling on sessions.

Recruiting quality outweighs sample size. Eight participants from the right profile find more than fifteen who would never use your product. The model assumes everyone has the same chance of hitting the problem, and that only holds if the profile is homogeneous.

If you have distinct segments, the calculation is per segment. A new user and an expert do not trip over the same problems: that is two studies, not one with the sample split in half.

In LatAm, budget for no-shows. A 10% to 20% drop-off rate is normal for moderated sessions: over-recruit relative to the number the calculator returns.