650.823.3042
Good, fast, cheap. Pick two.” I first heard that phrase when a friend of mineˡ used it to describe the dilemma that faces every researcher when designing or purchasing a marketing research project. But Web-based marketing research studies are fundamentally and permanently changing the role marketing research will play in future businesses. Good, fast, cheap. Pick two? Not any more. Now you can pick all three.
The Web is altering the landscape of business in many ways. One of those ways is in marketing research. Marketing research is an essential tool for businesses to gain vital information on which to base sometimes critical business decisions. One complaint with traditional marketing research is that the value of the information often isn’t worth what the research cost. Another complaint, particularly in industries with short product life cycles, such as high tech industries, is that the research process takes too long. Oftentimes, research results are not available until it is too late for them to be useful. Obsolete before they exist.
The Old Days
If you wanted a project to be high quality, that is to say, good, and you wanted it to be done fast, no problem. But it was expensive. If you wanted a project that was high quality and cheap, no problem there, either. But it would take a while, sometimes a long while, to complete it. And if you wanted a project to be completed quickly, i.e., fast, and not cost much, i.e., cheap, that could be accomplished, too. But it wouldn’t be very good (usually, it was awful). In research, in the old days, you could want it all and you could even want it all now, but you couldn’t have it. That is, you couldn’t have it all until now.
Good
Web-based surveys have many advantages over traditional methods. For starters, there are no interviewers involved. Have you ever observed interviewing from a real survey? It can be frightening. The first survey I ever designed was on the subject of home heating systems. When I went to observe first day of field I was surprised and horrified to realize that most of my interviewers could not pronounce many of the technical terms I used in the survey and virtually none of them knew what those terms meant! Not the best way to collect data.
Further, if you have ever seen a raw data set before cleaning and editing, you know that interviewers make numerous mistakes, some of which are irreparable. Those interviews are simply thrown out (or should be), which adds to the cost and time of the survey. Web-based interviews eliminate interviewer errors.
Web-based surveys eliminate interviewer bias, too. If the interviewer is in a bad mood or doesn’t like certain types of people or dislikes the subject matter being discussed, etc., the data can be affected. In Web-based surveys, every respondent has exactly the same interviewer; an interviewer that is never tired, moody, prejudiced, impatient or opinionated.
And telephone interviews suffer from the problem of being limited to audio. No pictures. Web-surveys can be truly multi-media. We recently did a new product study for a PC games developer. We showed respondents a variety of screen shots as well as a video clip of sample game play. If the survey is in -person, research exhibits can be constructed to give a more multi-media feel to the interview. But exhibits are not only costly and time-consuming to build, they can be damaged during interviewing, resulting in biased results.
Web-based surveys give the researcher so much more control over data quality, it is staggering. Logic checks can be built into the survey so that contradictory or non-sensical answers are not allowed. This eliminates the need for data cleaning and editing. Further, brands and attributes can be randomized to eliminate order bias. Literally, every respondent can have his or her own uniquely customized survey, with the end result being higher quality data than ever before possible.
Fast
There are many steps required to conduct a traditional market research survey. Once the questionnaire is written, it must be photocopied and sent to all the locations conducting the survey (if in-person), the field facilities must be briefed by the research agency, the field facilities must brief and train their interviewers, the interviews must be conducted during certain hours (you can’t call at 3 am, for example, and malls aren’t open then, either), the collected data must be edited by the field supervisor and shipped on to the tab house, which further cleans and edits the data, keypunches it into the computer and then runs tables. Analysis is done at this point and then the results are reported to the client, either in the form of an oral presentation or a paper-based report. From written questionnaire to results typically takes six weeks.
With a Web-based survey, the questionnaire is posted on a secure Web-site. Respondents are generated either from banner ads on other Web sites or personal email invitation. Respondents drop by the survey site whenever they want to (even at 3 am) and send in their surveys. Often, a sufficiently large sample, say 300 or 470 respondents, can be completed over a weekend. There is no interviewer briefing (no interviewer error, either), no data cleaning, no keypunch. Results can be automatically posted to a separate, secure Web site for instantaneous client viewing as soon as any respondent has completed the interview. We have done studies in four or five days on the Web that would have taken six or seven weeks using paper and pencil.
Cheap
There are two major cost components to any research project: data collection and analysis. Oftentimes, research studies are conducted just to find simple facts that require little or no analysis, such as determining advertising awareness or brand preference. But data collection costs are almost always there. In a typical study, respondents are screened to be the target customer for the client and then they are asked a variety of carefully crafted questions, either on the telephone or in a shopping mall. We’ve already discussed some of the problems with the people who ask these questions. But these people still have to get paid. There are field agencies all over this country (and the world) and their job is simply to collect data for someone else to analyze and report on. Generally speaking, this data collection cost is the most expensive part of any study.
With Web-based marketing research, this cost is oftentimes reduced to zero. For example, we have a software client that wanted to test a new package before committing to a product relaunch. They were able to post invitation banners on Web-sites that received a large amount of traffic from their target customers. These web-site visitors would click on the banner and be immediately sent to a secure web-site to conduct a very brief interview. At the completion of the interview, respondents were thanked and automatically returned to their point of departure. No incentives were needed. Our experience with Web-based surveys indicates that a respondent will typically complete a Web-based survey in about half the time it would take an interviewer to conduct the same survey by telephone or in person.
Another advantage is that low incidence samples, say people who raise orchids as a hobby, are very expensive to reach the traditional way. One would need to make a lot of telephone calls to generate 300 completed surveys from a group of people that represent only 1% of the total population. On the Web, you don’t find these hard-to-find people, they find you. How? By visiting Web-sites relevant to their interests, such as orchid growing, genealogy, ice hockey goaltending, ballroom dancing, etc. Oftentimes, research simply is not practical the old way. Only the Web will do.
And Caveats
The bad news is that we are not all the way there. Not just yet. The central weakness right now with Web-based surveys is sample. Is the sample that is generated for your Web-based survey representative of your target market? If not, your results could be misleading. We recently completed a study for a consumer electronics firm where we had two samples of registered owners of the client product. One sample was of owners who had Web access and the other sample was of owners who did not. The first sample was surveyed on the Web. The second sample was surveyed by telephone. Although somewhat disappointing, but not altogether surprising, we found that not only were the demographic profiles of the two samples quite different, but their product preferences and perceptions were, as well.
However, for many products, Web surfers are the target and this sampling issue is moot. For the others, it is estimated that currently 38% of Americans 16 or older, approximately 78 million, are on the Web at home and 11,000 new users come online everyday. It has also been estimated that Americans already spend as much time surfing the net as they do watching rented videos. With the introduction of Microsoft’s WebTV, Oracles Network Computer, DSL connections, home satellites and two-way digital cable all here or on the immediate horizon, as well as the plummeting cost of desktop PC’s and modems, Web access will certainly skyrocket over the next few years. It is easily imaginable that within five years, Web-based surveys will be viable for virtually any consumer product category in America.
Another issue to recognize is the two-dimensional limitation to computer monitors. Web-based survey respondents can’t currently hold a new product in their hands, feel its weight and texture, taste it or smell it. But with advances in holography and a variety of innovative peripherals being developed, even this limitation may be only temporary.
Summary
Web-based surveys offer tremendous potential to the marketing research industry. Web-based surveys are potentially faster to conduct, generate more accurate information and cheaper by several magnitudes. Used properly, Web-based market research will in the near future vastly increase the amount of customer feedback on which managers in all industries base critical business decisions.
Good, fast, cheap. Pick three.
The author wishes to thank John Luck for his keen wit in applying “Good, fast, cheap. Pick two” to the market research industry.
Prior to founding MACRO, Mr. McCullough was the senior vice president for a leading international strategic consulting and design research firm. Before that, he was market research manager for a $300 million division of Levi Strauss & Company.
Mr. McCullough has spoken before the American Marketing Association, the Marketing Research Association and various universities on a wide range of marketing topics. He is a past board member of the American Marketing Association, San Francisco chapter, and has been quoted in Canadian Packaging Magazine. He has chaired and lectured at applied multivariate statistics seminars for several years and is a member of numerous professional associations.