Non-probability sampling: what it is and how to do it right

When surveying a population or conducting experiments, you want results you can use. For example, if you're conducting market research on a toddler's toy in one state, collecting feedback from seniors in another will not give you the results you need. 

The process of choosing individuals to participate in a survey or an experiment is known as sampling. Getting the right sample requires careful thought and planning, as there are lots of ways to design, distribute, and collect data from surveys and experiments in ways that make extrapolating useful insights difficult, if not impossible.

Sampling can be categorized as either probability or non-probability sampling. 

In probability sampling, you randomly select participants from your population, with every participant having an equal chance of being selected. In non-probability sampling, you choose non-random criteria upon which to base your sampling choices from a larger population—not everybody gets a chance at being selected. 

There are four main subtypes of non-probability sampling (and variations of these subtypes) that researchers can commonly use in business and academic settings.

Convenience sampling

Convenience sampling involves engaging participants that are most convenient for you to access. For example, suppose you're looking to survey people's political opinions about a topic and decide to go door-to-door in your neighborhood to ask questions. In that case, you'd be creating a convenience sample. 

One type of convenience subtype is consecutive sampling, in which a researcher first gathers their convenience sample and engages in research. When they complete it, they continue to recruit and engage other respondents who fit the study's screening criteria, forming secondary and tertiary convenience samples and studying them consecutively.

Purposive sampling

Also known as judgmental sampling, purposive sampling is a method by which a sample is selected based purely on the researcher’s knowledge and credibility. Several types of purposive sampling exist, including:

Critical case sampling

In critical case sampling, you're making a judgment call about which small group of participants or cases are most important to the study of your subject.

Deviant case sampling

Here, you're looking for the most extreme case representing a particular subject you're trying to study.

Expert sampling

When you engage in expert sampling, you gather a sample of those individuals with the greatest expertise relevant to your subject.

Homogenous sampling

If you're investigating an attribute that a group has in common, you may wish to assemble a group that strongly resembles each other in one or more aspects.

Maximum variation sampling

To look at a subject from all possible perspectives, you build a sample that’s as diverse as possible.

Typical case sampling

In a typical case sample, you're looking for participants who exemplify what the average subject would look like when it comes to a particular subject or phenomenon.

Quota sampling

Quota sampling involves selecting a sample representative of the population from which you're trying to collect feedback. For example, suppose you're surveying an audience of sports fans, a third of whom like teams A, B, and C, respectively. No matter how many people you choose to sample, you'd draw participants equally from the three different groups of sports fans. 

However, it's important to note that in quota sampling, you're not randomly drawing participants from different subgroups. You're using some non-random attribute(s), such as proximity to you, to determine who participates.

Snowball sampling

Snowball sampling involves members of hard-to-reach populations. In such a sample, you start by engaging one member of this population willing to engage in your survey or experiment and ask them to introduce you to others in their group. Typically, researchers who've studied indigenous populations with little outside contact with the developed world must use this sampling method.

Before determining which non-probability sampling method to use, it's important to understand what the difference between probability and non-probability sampling means for your research. As stated above, in probability sampling, you're randomly drawing participants from a population. When you do so, you're eliminating many forms of bias that may be found in the results.

Probability sampling doesn’t remove all forms of bias from a research project. For example, you could inadvertently exclude members from your research if the list of individuals you sample (your sampling frame) differs from the population. But there are far fewer potential biases when you use a probability sample than when you use a non-probability one. 

However, probability sampling takes more effort and time than most non-probability samples. For example, say you're surveying a population and your sample includes 1,000 individuals, but only ten percent respond to your initial inquiry. To complete the research, you would need to spend time and money tracking down and encouraging the remaining participants to respond.

When you use a non-probability sample, you may find it easier to recruit willing participants. If you offer $5 coupons in a high-traffic area to participate in a survey, your results may not greatly reflect local area attitudes. But chances are you'll have a high participation rate. 

With non-probability sampling, there are many forms of bias you may introduce to your study, including:

Healthy user bias

In studies regarding health and health interventions, healthy users are more likely to opt-in to these studies. This overrepresentation will skew results if the sampling frame isn’t weighted appropriately.

Non-response bias

If many respondents fail to participate and you form your conclusions based on those who do, the absence of those participants may skew the results.

Pre-screening bias

You may also introduce bias into a study based on how you pre-screen participants. If you advertise a study about weight loss, you may attract more people who are motivated to lose weight than the general population.

Self-selection bias

Respondents opting into particular studies may share characteristics that skew the data. For example, marijuana enthusiasts may volunteer to take a survey about attitudes toward marijuana at higher rates than members of the population at large, which may skew the results.

Undercoverage bias

Some population members are less likely to participate due to logistical issues. You might have difficulty recruiting participants in rural areas with inconsistent Internet access, resulting in under coverage of certain population segments.

Despite the risks of introducing biases in your research, there are many instances when using non-probability sampling rather than probability samples makes sense.

The best way to determine which sampling method to use is to examine your study and determine your desired outcomes. For example, if you're looking to study participants who typically don't respond to studies, you may have to resort to snowball sampling by necessity. Or, say you need to obtain feedback from a population, but only those with a specific attribute provide detailed feedback. You may want to oversample from that group to get the practical insights you need.

If you choose to use non-probability samples, you'll want to minimize the biases you introduce to the study to the greatest extent possible. Make sure that your screening process, research description, and questions don't create biases and skew results. 

Oversample certain subgroups to avoid under-coverage. And make sure you spend appropriate time and resources recruiting participants to ensure that you attract the number of participants and level of engagement you need for your study.

Many common examples of non-probability sampling can be found in our day-to-day lives. Whenever you receive a customer feedback survey on a receipt, a company uses non-probability sampling. Political organizations that go door-to-door soliciting opinions are engaging in non-probability sampling. An employee survey that excludes managers from participation is another example. 

Many businesses also use non-probability sampling when beta testing, conducting focus groups, or sending surveys to their entire customer base.

In probability sampling, every member of a population has an equal and non-zero chance of being selected for a study. In a non-probability sample, certain population members have a zero chance of being selected.

Stratified sampling is an example of probability sampling. In a stratified sample, a population is subdivided into different non-overlapping subpopulations known as strata. When sampling, a researcher randomly selects each element (aka member) of strata. If the populations overlap or elements aren’t chosen randomly, the researcher uses non-probability sampling.

In random sampling, each population element has an equal chance of being selected in the final. By contrast, certain elements are more likely to be selected than others in a non-random sample.

Simple sampling (also known as simple random sampling) is an example of probability sampling, not non-probability sampling. In simple random sampling, a researcher chooses random elements from the sampling frame.

Statistical significance doesn’t depend on the type of sampling selected. Rather it depends on the effect size and the sample size. The effect size is the size of the difference in outcomes between two samples. The sample size or the number of participants in a study determines the amount of collected information, which affects the precision or level of confidence in the sample estimates. 

The bigger the sample size, the more likely it is to find a statistically significant difference between the study groups. However, researchers should always perform a sample size calculation in advance to avoid wasting resources in over-recruiting, which may also unnecessarily inflate the study results.