Sample Size Matters in Marketing Research

“Do I really need a sample size of 400? I was told 200 is enough.”

Raise your hand if you’ve heard this one before.

The Simple answer in marketing research sample size

Yes, you do in fact need a sample size of 400 (or possibly more). Two situations where that may not apply are when you don’t have sufficient budget, or if sample is simply unavailable. Or, of course, your target population is so small that 400 will be a big proportion of it (say, a quarter or more) – something that occurs infrequently.

Longer (traditional) answer

The sample size of 400 has a sampling error of about +/- 5%, while 200 has an error of +/- 7%. It doesn’t seem that big a difference, right? But don’t neglect the +/- sign. The spread is 10 versus 14 percentage points. Which, of course, means that two numbers need to be further apart to have a statistically significant difference for the smaller sample size. It gets worse for subgroups with even smaller sample sizes.

Now, let’s say as an experiment, you are running exactly the same study with these two sample sizes. When the first study (with n=400) finds say, ten significant differences, the second one (with n=200) will find fewer (say, 6-7) significant differences.  In a business environment where executives are looking everywhere for insights and edges, do you really want to constrain yourself by sacrificing sample size?

But what about the cost difference? That’s a great question – back when telephone interviewing was the norm, and every additional interview was expensive. In the online world, the addition of even a few hundred interviews often boils down to a few thousand dollars. Is it worth it to blind ourselves to potential insights by “saving” that money, especially when significantly more is already invested in the study?

Longer (more troubling) answer

The fact that smaller sample sizes will make it harder to reveal insights has been known for almost a century. But what is less well known is another issue with small sample sizes. A true relationship (say, a positive relationship) between two variables can show up as the opposite (i.e., a negative relationship) in a statistically significant manner. And, there would be no way to tell that was happening.

The article in the reference section provides a longer, more technical explanation of this phenomenon. But a simple way for practical researchers to think about it is in the framework of qualitative and quantitative research. Let’s say you do a handful of IDIs, or a focus group. Is it likely that you wouldn’t unearth all the insights you could have because your sample was so small? Of course – and that’s similar to running a small sample quantitative study and not finding some differences that actually exist.

Is the opposite possible? That is, finding differences that don’t generalize to the population? Of course. Taking that even further, is it possible to find something that is the opposite of what truly exists in the population? For example, your focus group says it loves your product, while a proper quantitative study would have shown the opposite. Yes, that’s possible too, and that’s why experienced researchers are careful about over-interpreting results from qualitative research.

As described in the referenced article, what is less well known is that such a thing is possible with quantitative studies as well – when sample size is small. Just calling a study “quantitative” does not make the risk disappear. It just makes it invisible.

So what should a market researcher do?

Generally speaking, the smaller the effect size one is looking for, the larger the sample size needs to be. For example, in a pre-post ad awareness study, if the general belief is that the ad campaign had been only somewhat successful, then a larger sample size would be needed, compared to one where external metrics indicated a very successful campaign.

But in practical market research, information on such effect sizes is usually not reliably available. So you should use as big a sample size as possible contingent on sample availability. With no other guidance, a sample size of 400 is generally recommended – but not because it eliminates sampling error, (nothing does, other than a census). It’s because (in a bow to practical considerations) there are diminishing returns from sample size beyond that. But it really doesn’t become small until about n=1000 (when it goes down to 3%). So, if you can afford it, go big.

This applies to “agile” research as well, which is usually done with small sample sizes. It’s a great first step to get quick feedback, but not something on which business decisions should be based. Insights from that first step can usefully inform the design of a robust succeeding step – where proper sample considerations can validate the insights. Just calling research agile doesn’t reduce the risk. It just makes it fragile.

A related issue – and one well known to quantitative researchers – is sample representativeness. A big sample size (say n=1000) is not particularly useful if the sample ends up (unwittingly) representing a sliver of the market. Small sample sizes can be more prone to issues of representativeness, and are harder to adjust with weighting solutions. That can call the results into question in a hurry.

In Conclusion

Yes, we live in a world where insights are demanded as fast and as inexpensively as possible. But there’s a right way to do it and a wrong way. The next time someone suggests that a quantitative study can be run cheaper by compromising sample size, think about what we discussed here and decide for yourself – is it really worth it?

References

Andrew Gelman and John Carlin (2014), Beyond Power Calculations; Assessing Type S (Significance) and Type M (Magnitude) Errors, Perspectives on Psychological Science, Vol 9.