Recently, L&E Research hosted a webinar with guest speaker David F. Harris, Founder of Insight & Measurement. David is the author of, The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions.

(Click here to view the recording of David’s Webinar.)

David wrote his book to address the following concerns about market research and questionnaires:

• Research is rarely organized to support decision making
• Often, no qualitative research is conducted prior to questionnaire development
• Questions are often unclear, biased, or not answerable by the respondent
• Questionnaire pretesting is not practiced widely enough

Questionnaire writing is serious business and is as much art as science. Most researchers have struggled with questionnaire design, recognizing that understanding how consumers think about a topic, as well as science, logic, reasoning and sound research practice, are all necessary to producing a good questionnaire.

Qualitative research is often skipped in the questionnaire development process, to the detriment of question quality. David recommends a research framework that includes two points of qualitative research to inform and improve questionnaire development:

Before You Plan the Questionnaire

First, qualitative research should be conducted before the questionnaire planning begins. Conducting a focus group or several in-depth interviews will let you hear the words and phrases the consumer uses in discussing the product or the purchase experience, understand the logical sequence of topics for the consumer, and identify the correct metrics to use in your questions. Let’s say you are conducting a survey about children’s studying habits. You think children study outside of the classroom between two and five hours per day, six days a week. In your qualitative research you find the norm is actually 30 minutes to two-and-a-half hours, five days a week. If you had based your question on your assumption, you might have used the wrong scale in your quantitative research, leading respondents to overstate their studying habits and for bad decisions to be made based on this information.

Hopefully, it goes without saying you must do the qualitative research with the same pool of respondents who will be completing the quantitative research. You can’t just ask your coworkers to help you out with this!

Pretest (and Revise!) the Questionnaire

Even if you conduct qualitative research before you plan your questionnaire, David recommends conducting qualitative research a second time, just before the quantitative data collection begins. The only way to truly test whether a question works is to have a real respondent try to answer it. David suggests a two part pretest – First, ask internal resources to review your questions, find problems, and make improvements. Second, pretest the questionnaire with real respondents, ensuring that you administer the questionnaire pretest to a sample of each segment of respondents. Bring these respondents – and again, real respondents – into a focus group facility and ask them to complete the survey using the phone, online, or whatever methodology you will use in the actual research.

After each respondent has completed the survey, go through it with them question by question. Did they have any uncertainty about what was being asked or how to answer? Was any wording vague or unfamiliar? When they were unsure about what was being asked, how did they decide upon an answer? For example, if your question asks about cars owned by the respondent, does that include trucks? Would “vehicles” be a better word choice? What if you lease your vehicle – does that count? Qualitative research can help you make sure the information you get from your questionnaire is the information you need to have.

As you consider implementing these suggestions into quantitative research project, you may encounter objections that it will take too long, cost too much, or that you already know about your customers. In spite of these objections, researchers must take responsibility for asking the best questions possible. Think of qualitative research as an insurance policy on your results; it may be very cost effective in the long run. After all how long will it take and how much will it cost to recover from a poor business decision?

Want more advice on writing better questionnaires? Check out David’s website –www.davidfharris.com or watch one of his previous workshop videos on our webinars page.

Take the time and struggle out of writing screeners and start getting the right respondents for your qualitative research.

Screeners are ultimately short questionnaires. But unless they are written extremely well, the quality of respondents will suffer. On May 13, 2016, L&E hosted a webinar with David F. Harris, author of, The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions. David provided detailed examples of how to improve your screeners to get better respondents for your qualitative research.

Webinar Recording

Webinar Presentation

To download a PDF of David’s presentation from the webinar, click here.

About David F. Harris

David F. Harris conducts qualitative and quantitative research for companies in a variety of industries. As founder of Insight & Measurement, he also conducts training and consulting on questionnaire design and screener development. He is author of, The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions. He received his B.A. from Reed College, in Portland, Oregon, and his M.A. in Quantitative Psychology from the L. L. Thurstone Psychometric Laboratory at the University of North Carolina at Chapel Hill.

“Businesses are drowning in data but starving for insights. Worse, they have no systematic way to consistently turn data into action.”

-“Digital Insights Are the New Currency Of Business”, Forrester Research, April 2015

“Big Data” is the term used to describe how advancing trends in technology will change the way information is delivered to businesses. A lot more data now exists. In fact, industry pundits note that 90% of the data that exists today was created in the last two years. And, with the sheer volume of social media and mobile data streaming in daily, businesses expect to use Big Data to aggregate all of this data, extract information from it, and identify value to clients and consumers.

The potential value for brands using these data assets is huge, but the opportunity of Big Data is also its challenge. We are drowning in data. And tweets, posts and video are not the structured data that fits well into relational databases for traditional querying. As a result, Big Data requires new ways of thinking about how to store and analyze information to extract insights, and turn those insights into actionable decisions.

The market research industry, especially qualitative research, has a key role to play in unlocking the value of Big Data, because when it comes to understanding consumers, market researchers are hard to beat. This is not about market researchers simply adding the ‘why’ to the ‘what’ of Big Data. There are much bigger roles for market researchers to play when it comes to Big Data.

Read more about the role market researches can play in in downloading our new eBook, Why Qual is Neccessary to Add Meaning to Big Data.

“There is nothing like looking if you want to find something. You certainly usually find something, if you look, but it is not always quite the something you were after.” – J.R.R. Tolkien

All marketers struggle with scarce resources to address business challenges. Perhaps in no other marketing area are resources quite so limited as in market research. Management does not want to invest in high-quality research. Respondents don’t want to participate, and if they do participate, it is challenging to get them to give valuable information. Sometimes it seems as if insight is as rare as hen’s teeth.

Therefore, it is critical that market research be planned specifically to lead to decision-making. Recently, L&E Research hosted a workshop in our Raleigh facility led by David Harris, author of The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions. (Click here to view the video of David’s talk.)

According to David, research is rarely organized to support decisions, and he wrote his book to propose a solution. His market research framework lays out a process for making the most of the scarce resources available to market researchers. Without cutting corners, David’s framework helps marketers focus their thinking to generate research projects that successfully lead to decisions and actions based on those decisions.

David’s Framework starts with three steps:

• Plan research to support decision making
• Conduct qualitative research
• Plan the questionnaire

Note that the third step is to plan the questionnaire, not write it!

Planning for decision making before you even start to write the questionnaire is challenging, because marketers are born question-writers. However, it ensures that the research that you end up doing is specifically designed to support the decisions that need to be made.

Most decisions are made with a limited set of information, often as few as three to seven facts. And yet, research typically goes far beyond the information needed for the decision. In order to focus research for decision making, David recommends you write a research plan that identifies three things:

• The decisions the research will support
• The information needed to make those decisions
• The design of the study that will collect that information

Answering these questions ensures that you use the best methodology (i.e., the most cost effective methodology that still delivers the information needed) to get the right and most accurate data. A bonus of this kind of research planning is that the questionnaires are shorter, enhancing participation rates among respondents.

Once you have focused on the information to seek, qualitative research can help you know what questions to ask, how to ask them, and who to ask. David advises, “In order to understand a topic, you need the insights that only qualitative research can provide.” Qualitative research allows you to hear the words and phrases respondents use when they speak about the topic, how they understand the metrics you need to collect, as well as what criteria they use in evaluating the topic.

Finally, you are ready to write the questionnaire plan. David recommends a matrix that identifies, for each decision:

• What information is needed
• How to ask the question
• How the data will be analyzed

All of these factors, of course, influence the specific questions to be included in the research instrument, but note that you are still distinguishing between the information you need for the decision and the questions themselves. Maintain the discipline not to write any questions until the questionnaire plan is complete.

Without this considerate and thoughtful approach, you run the risk of completing research that goes astray, and is less than optimal in addressing the business challenge. Using David’s framework, you will find not only “something, if you look” (per Tolkien) but certainly the “something you were after.” Moreover, that “something”, will enable you to make successful decisions for your business.

Want to learn more about David’s market research framework? We’ve created a handy infographic that summarizes the first series of steps in planning your research and your questionnaire.

We are wrong and we know it.  Marketing research has always been plagued by error and assumptions; random sampling almost never is, all kinds of bias – observer, response, fatigue, scale.  The list of issues goes on and on.  Just as importantly, marketing research has been wrong by omission; that is to say that we could only ask questions – and evaluate the answers. At the end of the day, it is a calculated and reasonable “wrong”.

But we are getting better… and that shows no sign of slowing.  Some of the improvements over the past several years are about the process (better sampling, improved questions); many others are about killing the error of omission – and to L&E, this is the exciting part.

New sources of data about consumers allow for vectors of knowledge that did not exist even 3 or 4 years ago.  Behaviors can be established from any number of sources: mobile tracking, receipt scanning, in-store video – the list goes on and on.  Social Media allows a sneak peek into peoples’ lives that lets us see how they wish to be perceived – and not just through their words, but also the images they associate with their life.  Emotions and the emotional triggers can be identified through non-conscious methods from facial coding, to neuro methods and wearables.

As costs continue to fall from competitive pressure and technological advances (I just saw a Virtual Reality viewer for $20), it will be easier to put more of these pieces together to get a clearer picture of the “whole truth” – which has long been the goal in our industry.  Let’s take a look at two examples.

Eye tracking technology lets us identify the area of a document or image that is getting attention.  Facial coding technology allows us to understand the emotion a person is feeling at a particular point in time.  Independently, each of those technologies answers an important question.  Together they answer the question of the specific driver of that emotion.  One particular client takes it a step further and integrates in-depth interviews based on laddering to understand the “why” of the emotional triggers.  In this example, there is little error of omission, as multiple approaches within a single respondent engagement have answered the what, the how, and the why.

Behavioral research has generally been based on observation (expensive) or diaries (questionable).  The mobile phone and, in particular, mobile panels change all this.  Now, behavioral research can be conducted in connection with shopper journey, use tests, day in the life, etc., with reasonable completeness and accuracy.  Scanner technology (the same kind used in Expensify) allows people to scan their store receipts instead of using a diary for their purchases.  Each of these methods provides valuable information about a person’s life.  The combination of these two methods provides a more holistic perspective of a person’s consumption life – as it lays out multiple places, multiple experiences, and multiple purchases – in the context of all their purchases (not just a category).  The error of omission is still there, but smaller and easier to forgive.

Understanding everything that motivates a person is hard.  Psychologists can spend years trying to understand someone and still not be comfortable that they have it right.  But we can get better at understanding – as we should – for our businesses, for our clients, and for consumers.

Market research is now particularly well positioned to do this using a holistic approach that combines qualitative, quantitative, behavioral, non-conscious and observational data; shifting the conversation from “we could be wrong for all of these reasons” to “we think we’re right for all of the data points we have to use”. That’s a big (and exciting) difference – and the GRIT data tells us it’s now a reality.

“If you could do only one type of research, would it be qualitative or quantitative?”

I was once asked this question in an interview for a corporate job. I paused at this question for two reasons. First, because it was a bit ludicrous.  Second, it made me think about the role of each of these and how they’re applied.

Human interaction counts

Every few years we hear predictions that hard data (quantitative) is going to replace qualitative. In fact, we’ve heard those predictions for many, many years now. The 2014 ESOMAR Global Market Research Industry report, and prior waves of the GRIT report suggest that these predictions are a bit exaggerated. Focus groups, in-depth interviews, and online qualitative are all either flat or growing slightly, depending on which numbers you choose. It’s easy to understand why these predictions are made, but it’s more than a bit short-sighted because human interaction counts for more than it’s often credited for.

Digging into the “why” of behavior

Qualitative research provides the nuance to the “how” of consumer behaviors, and much more importantly, the power to the “why.” While things like Big Data, passive measurement, geo-location, and data synthesis are grabbing many of the headlines in our industry, qualitative research goes on doing what it has always done: getting to the deeper understanding of the consumer.

Connects with people

The interactive conversation about the “why” of behavior is the essential benefit of qualitative and has yet to be surpassed for its depth, sensitivity, and flexibility. Topics that are difficult to cover otherwise, such as finances, health, relationships, sex, etc., can all be handled with a certain connection that is made between people instead of a survey.

And these qualities of qualitative research go a long way to meet the basic needs of a business’s deep understanding of consumers. As one respondent in this publication of the GRIT report said, the biggest challenge of marketing research is to “understand and deepen knowledge of consumer behavior. The consumer [should be] viewed in all mindsets, perspectives and through his or her side.”  And much of this is the purview of qualitative research.

The effect of mobile technology

Today, one area of qualitative is growing substantially. This is the area facilitated by mobile technology. Currently, about 20-25% of corporate researchers and about 50% of research suppliers are using mobile for at least some of their qualitative/ethnography research. And another 35% of corporate researchers say it’s under consideration.

Mobile qualitative has several unique benefits. First, it is real-time “in the moment” and consequentially allows an even deeper understanding of consumers because of the timing of an event and the understanding. Second, it can be non-intrusive as the tool is both available and comfortable for consumers. Lastly, it can be pretty fast, which is always good.  In many cases, the mobile component of qualitative is simply that: one component. The mobile tool is often integrated with more traditional tools, such as in-depth interviews, for an even more substantive outcome.

There’s a human gap in big data

No commentary on research would be complete without at least mentioning Big Data. We all have our favorite statistics about the proliferation of information. “It doubles every two years” is mine. However, there is long distance between the implication of data and the human understanding that brings that data to a level where most of us can understand, empathize, and act.

Respondents to the current GRIT report talk about this as well. One respondent noted, “Market Research needs to remain relevant by providing consultative resources and also helping to answer ‘why’ when big data seems to be revealing hard to understand insights.” You simply can’t (and shouldn’t) get away from the importance of “why.”

Brett Watkins
President
L&E Research

Leave your comments below or connect with me on LinkedIn.