Recently, L&E Research hosted a webinar with guest speaker David F. Harris, Founder of Insight & Measurement. David is the author of, The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions.
(Click here to view the recording of David’s Webinar.)
David wrote his book to address the following concerns about market research and questionnaires:
- Research is rarely organized to support decision making
- Often, no qualitative research is conducted prior to questionnaire development
- Questions are often unclear, biased, or not answerable by the respondent
- Questionnaire pretesting is not practiced widely enough
Questionnaire writing is serious business and is as much art as science. Most researchers have struggled with questionnaire design, recognizing that understanding how consumers think about a topic, as well as science, logic, reasoning and sound research practice, are all necessary to producing a good questionnaire.
Qualitative research is often skipped in the questionnaire development process, to the detriment of question quality. David recommends a research framework that includes two points of qualitative research to inform and improve questionnaire development:
Before You Plan the Questionnaire
First, qualitative research should be conducted before the questionnaire planning begins. Conducting a focus group or several in-depth interviews will let you hear the words and phrases the consumer uses in discussing the product or the purchase experience, understand the logical sequence of topics for the consumer, and identify the correct metrics to use in your questions. Let’s say you are conducting a survey about children’s studying habits. You think children study outside of the classroom between two and five hours per day, six days a week. In your qualitative research you find the norm is actually 30 minutes to two-and-a-half hours, five days a week. If you had based your question on your assumption, you might have used the wrong scale in your quantitative research, leading respondents to overstate their studying habits and for bad decisions to be made based on this information.
Hopefully, it goes without saying you must do the qualitative research with the same pool of respondents who will be completing the quantitative research. You can’t just ask your coworkers to help you out with this!
Pretest (and Revise!) the Questionnaire
Even if you conduct qualitative research before you plan your questionnaire, David recommends conducting qualitative research a second time, just before the quantitative data collection begins. The only way to truly test whether a question works is to have a real respondent try to answer it. David suggests a two part pretest – First, ask internal resources to review your questions, find problems, and make improvements. Second, pretest the questionnaire with real respondents, ensuring that you administer the questionnaire pretest to a sample of each segment of respondents. Bring these respondents – and again, real respondents – into a focus group facility and ask them to complete the survey using the phone, online, or whatever methodology you will use in the actual research.
After each respondent has completed the survey, go through it with them question by question. Did they have any uncertainty about what was being asked or how to answer? Was any wording vague or unfamiliar? When they were unsure about what was being asked, how did they decide upon an answer? For example, if your question asks about cars owned by the respondent, does that include trucks? Would “vehicles” be a better word choice? What if you lease your vehicle – does that count? Qualitative research can help you make sure the information you get from your questionnaire is the information you need to have.
As you consider implementing these suggestions into quantitative research project, you may encounter objections that it will take too long, cost too much, or that you already know about your customers. In spite of these objections, researchers must take responsibility for asking the best questions possible. Think of qualitative research as an insurance policy on your results; it may be very cost effective in the long run. After all how long will it take and how much will it cost to recover from a poor business decision?
Want more advice on writing better questionnaires? Check out David’s website –www.davidfharris.com or watch one of his previous workshop videos on our webinars page.