Pilot Test Market Research Survey Questionnaires
Pilot Test With the Usability Technique of Cognitive Interviewing
Why Pilot Test Survey Questionnaires?
Regardless of how diligently market researchers approach the design and construction of questionnaires, mistakes are inevitable. Because the market researchers are often too close to the survey work, it becomes difficult to spot an ambiguous question, a statement that people don't really understand, or wording that suggests bias.
Pre-testing or pilot testing the questionnaire is the surest way to reduce the probability of errors.
For do-it-yourself (DIY) market researchers conducting surveys research, it is not necessary to run the pilot test with a representative sample of people. However, it is helpful to select people for the pilot test who will have a fundamental understanding of the topics covered in the questionnaire and who will find the questionnaire relevant.
It is very difficult to see where misunderstandings originate when reviewing the language of a survey that one has crafted. In addition, a respondent may completely misunderstand a closed-ended question without any overt indication that this is so. It is essential for market researchers (and DIY market researchers) to learn about the cognition that is behind the responses that are provided on the survey questionnaire.
Cognitive Interviewing Approach to Pilot Testing of Questionnaires
Avoid reading the questionnaire aloud to people as this will not give an accurate test of the questionnaire instrument.
The people in the pilot test sample should read through the questionnaire and respond to the questions and statements just as though they were taking the survey. It may sound unlikely, but it is not uncommon for a question to seem fine during a quick scan or rapid reading, but later turn out to be impossible to answer when actually taking the survey and performing a deeper read.
A recommended method for approaching pilot testing by do-it-yourself (DIY) market researchers is cognitive interviewing (Beatty, 2007; Willis, 2005). Usability testers use this or similar approaches in which they tap into the thinking of the person who is testing a product. The purpose of cognitive interviewing is collect data that reflects the respondents' thoughts as they complete the questionnaire. That is to say that the respondents say aloud what they are thinking as they answer the questions and process the meaning of the statements.
Part of this process entail having the respondents tell what they think the questions are asking or what the statements on the questionnaires mean. Valuable information is shared by the respondents with the market researchers in the process of cognitive interviewing. The market researchers learn which questions communicate clearly and effectively. If the question or statement items miss the mark, it will be apparent to the market researcher during the cognitive interviewing process.
What Role Do Focus Groups Play in Cognitive Interviewing?
Focus groups are the group version of cognitive interviewing. It is important to recognize that the process of participating in the focus group can alter the comments that respondents give with regard to the survey questions they are answering.
A focus group provides respondents an opportunity to elaborate about their perceptions and thinking about the survey, and a chance to discuss the survey or topics covered with others.
Focus groups can provide market researchers with information that is useful before survey questionnaires are developed, such as what topics are salient and how the topic is understood by the survey target population. One of the strongest advantages of using focus groups to pilot test a survey questionnaire is the understanding that can be gained about how people interpret the question items, and how framing a particular question, statement, or topic influences the responses given by participants.
Beatty, P.C., Willis, G.B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287-311.
DeWalt, D.A., Rothrock, N., Yount S., Stone, A., (2007). Evaluation of Item Candidates –The PROMIS Qualitative Item Review. Med Care 2007; 45, S12–S21. PROMIS Cooperative Group.
Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (1994). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68(1) 109-130. DOI: 10.1093/poq/nfh008
Willis, G.B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications.