Sample size and margin of error calculator

The calculator solves the classic problem of estimating a proportion: what part of a population has a given trait — satisfied with the product, using a feature, willing to pay. For a simple random sample from a large population, the size follows the formula given by W. G. Cochran in “Sampling Techniques”:

Example: 95%, ±5 pp, p = 0.5. n₀ = 1.96² · 0.25 / 0.0025 = 384.2 — round up to 385. That is the famous “385 respondents” quoted in guides: it only holds for a large population and random selection.

Finite population correction

When the population is small — a company’s employees, customers on one plan, webinar attendees — each respondent noticeably reduces uncertainty about those not surveyed. The finite population correction (FPC) accounts for this:

For N = 1,000, 385 becomes 278; for N = 10,000 it is 370; for N = 100,000, 383. The correction matters when the sample exceeds 5% of the population; for an audience of hundreds of thousands it changes almost nothing.

Margin of error of a completed survey

The reverse problem is the “Margin of error” tab. For an observed share p̂ in a sample of n responses:

The second factor is the same finite population correction; without N it equals one. This normal interval behaves poorly for small n and shares near 0 or 100%: it can cross the bounds and covers the true share less often than claimed. So the calculator also shows the Wilson interval, which Brown, Cai and DasGupta recommend as a replacement for the Wald interval:

The product p · (1 − p) peaks at p = 0.5, where it equals 0.25. At p = 0.2 it is 0.16; at p = 0.1, 0.09. So if you know in advance that a trait is rare, you need a smaller sample: at p = 20%, 95% confidence and ±5 pp, 246 people suffice instead of 385.

But a questionnaire almost always has more than one question, with different shares. A sample sized for p = 0.2 gives a margin of about ±6.2 pp on a “50/50” question. That is why surveys are planned with p = 50%: it guarantees the stated margin for any two-option question. Use a smaller share only when the survey is built around one metric whose level is known from past measurements.

Remember subgroups too. If you want to compare new and long-time users, you need ±5 pp in each subgroup — 385 responses in each, not in total.

Sample size only controls random error. Representativeness is a different property: a sample resembles the population if every member had a known, non-zero chance of being surveyed. The formulas above assume simple random selection, and no number of responses fixes systematic bias.

Self-selection in in-app surveys

A survey in a pop-up or an email is answered by those who choose to. Active users and people with strong feelings — either way — respond more often. Those who have already left the product never see the invitation. Such a sample is not random, and a ±3 pp margin is a formality: the real error can be much larger and cannot be estimated from the responses themselves. The American Association for Public Opinion Research (AAPOR) specifically warns that the classic margin of error loses its meaning for non-probability online samples.

What helps in practice:

• Randomize who sees the invitation. For example, every tenth user who reaches a given step, not everyone.
• Ask at a meaningful moment. Asking about product value on first login is pointless: the person has not done anything yet. After completing a key onboarding step is better.
• Compare responders with non-responders on what analytics tells you: plan, account age, activity. If the groups differ a lot, weight the results or at least state the caveat.
• Track the response rate. Response rate = responses / invitations. The lower it is, the higher the risk that responders differ from everyone else. Norms depend on channel and audience, so compare with your own past surveys.

How many invitations to send

If you need 385 responses and one in five usually replies, show the invitation to 385 / 0.2 = 1,925 users. The calculator does this automatically. If you need more invitations than there are people in the population, change the method, not the sample: shorten the survey, add a reminder, pick a different moment.

• Confusing percent and percentage points. A ±5% margin in sample size calculators means percentage points: a share of 30% means “25 to 35%”, not “28.5 to 31.5%”.
• Sizing for everyone but drawing conclusions by segment. Comparing groups needs an adequate sample in each; testing the difference between them is a separate calculation, as in an A/B test.
• Trusting the margin with self-selection. The formula covers random error only. Bias from who responds is not in it.
• Rounding down. Sample size always rounds up: 384.2 is 385.
• Forgetting response rate and junk answers. Some responses will be incomplete or straight-lined. Add a buffer to invitations.
• Confusing confidence with precision. 99% does not make the result more precise — the interval gets wider, and the sample for the same margin grows about 1.7 times.

If the survey is meant to compare two onboarding versions, calculate the sample size for a hypothesis test instead of a margin — in the A/B test calculator.

1. Cochran W. G. Sampling Techniques. 3rd ed. Wiley, 1977 — sample size for a proportion and the finite population correction (ch. 4).
2. Wilson E. B. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association, 22(158), 1927 — the Wilson interval.
3. Brown L. D., Cai T. T., DasGupta A. Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 2001 — comparison of intervals for a proportion.
4. Groves R. M. et al. Survey Methodology. 2nd ed. Wiley, 2009 — coverage error, nonresponse, self-selection.
5. AAPOR. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys — response rate definitions; AAPOR Report on Online Panels, 2010 — on the margin of error of non-probability samples.