Written by Jen Sosin, Chief Research Strategist
Market researchers are often challenged by the choice between conducting quantitative surveys online, usually by tapping pre-recruited panels of respondents, or conducting surveys by telephone. The choice is often framed as a choice between high-cost, high-reliability telephone surveys, and low-cost, low-reliability on-line surveys. There is some truth to this frame, but it is not the entire truth. For example, telephone surveys are not always reliable, and on-line surveys are not always low cost. How do we choose?
The Argument for Telephone Surveys
Probability theory is the foundation of statistics. In probability theory, for a sample survey to be reliable, every individual in the population that interests us must have an equal probability of being interviewed. That is what “random sampling” means. With random-digit dial telephone surveys, where telephone numbers are generated and phoned at random, and where good research methods involve specific techniques to reach people who are less likely to answer the phone and to persuade those who initially refuse to participate, survey researchers can come reasonably close to random sampling.
By definition, on-line surveys do not rest on probability-based sampling methods. First, respondents are limited to people who are on-line. This means that people who spend little or no time on-line have little chance of being interviewed. Second, respondents are entirely self-selected.
They must have voluntarily chosen to opt into a survey panel, and they must have voluntarily chosen to respond to that particular survey. This means that people disinclined to participate in surveys, or people who simply haven’t taken the time to opt into a panel, or people who are not interested in a survey’s topic, also have little chance of being interviewed.
This creates all sorts of unknown biases: How might people who spend more time on-line and who are interested in participating in surveys differ from others? It also creates known biases. For example, on-line panels almost always under-represent older and lower income people.
These biases raise legitimate doubts about the reliability of on-line surveys. Indeed, academic researchers remain extremely skeptical of on-line research, and a study by Jon Krosnick of Stanford made a strong case that on-line samples can generate results that are quite different from probability sampling by telephone. Thus, most government and academic agencies continue to conduct their surveys by telephone, the New York Times has a policy against publishing the results of research conducted on line, and most U.S. news-sponsored research organizations will not use on-line sampling in their own research.
The Argument for On-Line Surveys
The case for on-line surveys rests on two basic points: First, the reliability of telephone surveys is declining, while on-line panels are improving. This means that the reliability gap between telephone and on-line sampling is shrinking. Second, conducting research on-line can be much less costly and is often the only practical choice.
First, as the proportion of households that rely solely on mobile phones grows, now topping 40% in the U.S., and as refusal rates have increased, telephone samples are becoming less and less reliable. While we balance the migration to mobile phones by supplementing random-digit dial samples with mobile phone samples, it is much more expensive to interview people via mobile phone than by land-line (in part, because by U.S. law, mobile phones must be dialed by hand).
In addition, the larger the proportion of respondents reached via a mobile-phone sample, the greater the distance from pure probability sampling methods, and the greater the opportunity for bias.
Meanwhile, as the major on-line panel providers compete for market share, panels are growing in size and diversity, new techniques are being deployed to validate that respondents are who they say they are, and new methods of maximizing response rate are being developed. The result: It is more and more possible to recruit on-line samples that, at least on the surface, look like the underlying populations. While elderly and less affluent people are underrepresented in panels, they do exist in panels, and even the Krosnick study suggests that careful use of sample stratification and weighting can increase the reliability of on-line surveys.
In the case of business research, when we are trying to sample and interview business professionals, the reliability gap is even smaller. In most cases, probability sampling of business professionals is simply not possible. To interview professionals by telephone, we must first develop or acquire a list of possible research participants, and commercially available lists vary widely in their accuracy and completeness. Then we must persuade potential respondents (or, more frequently, their gatekeepers) to participate in the research. In general, refusal rates with B2B research are quite high, adding all kinds of additional non-response bias, and moving us further away from true random sampling, which requires that every professional in the category that interests us has an equal probability of being interviewed. In the end, for B2B audiences, sometimes on-line methods are the only practical choice.
Second, conducting research on-line can be much less costly. Depending on the number of interviews we want and how narrowly we are screening respondents (are we happy to interview any adult or are we looking specifically for mothers of teenage boys?), the cost advantage of going on-line can be quite large.
Other Advantages of On-Line Surveys
We can add to the narrowing reliability gap and the cost differences some further advantages of on-line surveys. One of these is survey content: On-line surveys allow a wider range of question-and-answer techniques. We can ask respondents to put things in rank order, sort items into categories, or absorb and react to more information—including visual images. When we ask unaided questions, we can gather exactly what people say, rather than relying on interviewers’ ability to type quickly. Thus, on-line research can be a more powerful tool for understanding the reasons for people’s opinions. [1]
An additional advantage is speed. Particularly when we are looking for very specific types of respondents, or trying to interview business professionals, on-line surveys can be much faster than telephone surveys, because they avoid the need to make hundreds of phone calls looking for qualified respondents, and because they avoid the need to navigate business gatekeepers to get an interview appointment on the schedule of a busy executive.
The result of all these factors is that, in the commercial world, on-line research is generally the norm, not the exception. By 2011, for example, Unilever had moved 80% of its market research on-line.
The further result is that, editorial policy notwithstanding, most U.S. media will report on on-line surveys if convinced that there is a compelling and methodologically sound rationale for going on-line. For most media, editorial discretion prevails, and can supersede stated policies.
Moreover, in a number of markets outside the U.S., on-line has become the norm for the media as well. The U.K. is a case in point, where some major newspapers now rely entirely on on-line surveys.
Making the Decision
How, then, do we decide between telephone and on-line methods? In general, we weigh goals and priorities as illustrated below:
| When the research will be used for decision-making and not shared publicly: | When the priority is obtaining the most reliable measures and estimates, we recommend telephone methods.In most other cases, we recommend on-line methods.We are particularly likely to recommend on-line methods when we are testing visuals or generating ideas, or when we care more about understanding the reasons for opinions than accurately measuring how many have an opinion. |
| When the research will be shared widely or publicly: | When it is important that the results have credibility among government, academia, and elite media, we recommend telephone methods.When credibility among elite audiences is not a priority and when we don’t need to describe the results as accurately representing the public [2],we recommend on-line methods. |
Then, in instances where the decision is not obvious, we also consider the following:
What is the incidence of the population we are trying to interview? In other words, if we dial households randomly, what proportion of households will have someone qualified to participate in the survey? If we are trying to reach adults, age 18+, incidence is almost 100%. If we are trying to reach parents, incidence is roughly 45% (that is, about 45% of American households include at least one child). If we are trying to reach women who live alone, incidence is about 15%. The lower the incidence, the more expensive telephone sampling will be, and the longer it will take. With consumer audiences with less than 20% incidence, random telephone sampling usually becomes impractical.
How relevant are the major biases of on-line research? If we are doing a survey of seniors, on-line research would not be a wise choice, since such high proportions of seniors do not spend time on the internet. Conversely, if we are doing a survey of on-line gamers, doing the research on-line makes greater intuitive sense.
How important is it for the research to deliver accurate measurement of population parameters? If we are trying to determine what proportion of Americans (or any other population) behave a certain way, or have certain characteristics, probability sampling is very important. For example, when we wanted to learn what proportion of American households include someone with diabetes [3], probability sampling was a must. Often, though, our objectives are less about absolute measurement and more about understanding the reasons for opinions and the relationships between opinions. The more important it is to generate reliable population estimates, the more important it is to come as close to random sampling as possible. The more we are interested in reasons and relationships (which we sometimes call the “internals”), the more reasonable it becomes to conduct the survey on-line.
What do we want to ask? If, for example, exposing respondents to visual images is an important part of the research, it is a strong argument for on-line research.
What are the budget and schedule constraints? For low-incidence consumer audiences, probability sampling by telephone can significantly more than on-line sampling. It can also take days or even weeks longer. In some such cases, on-line research is the only practical choice, and we deploy all the tools at our disposal to maximize its reliability and to consider potential biases as we interpret results.
For business audiences, one question dominates all others: What strategies are available for reaching who we want to reach? In other words, can we create or acquire a good list—meaning a list that is both comprehensive and accurate? What information will be included on that list—just company names? Or individual names and contact details? And will those contact details include email addresses? How do available lists compare to on-line panels of business professionals who have already opted into participating in surveys? In short, with business audiences, often, the choice between telephone and on-line will be driven entirely by the availability of appropriate lists and panels.
In the end, however, the choice between telephone and on-line surveys increasingly resembles the choice between a hammer or wrench or saw. No method is perfect, all methods have appropriate uses. The goal is choosing the right tool for each project, with full awareness of its advantages and disadvantages.
