Written by Roger Jackson, CEO of SenseCheck
In the marketing world, traditional research methods like surveys and focus groups often come under fire. Critics argue that these tools offer little more than false guidance, with surveys being particularly vulnerable to scrutiny by some.
As someone with decades of experience in moderating focus groups, I could discuss that point very happily. Here’s a clue: they have to be done well!. But today, let’s focus on the humble survey — an easy thing to be rude about to get overly simplistic nods, yet I will argue remains an invaluable tool in the marketer’s arsenal.
The critical role of questionnaire design
Survey sceptics are quick to highlight the flaws inherent in asking people to complete questionnaires. And they’re not entirely wrong. A poorly designed and executed survey can indeed yield misleading results. The adage “garbage in, garbage out” holds true—research and questionnaire construction is crucial, but it’s important to recognise that a well-crafted survey, designed with precision and thoughtfulness and executed with care, can offer useful insights.
Let’s think about the basics. The first rule of survey design is to ask questions that your respondents can answer truthfully and easily. These questions should tap into their opinions, perceptions, concerns, motivations, and needs—information that resides close to the surface of their consciousness and things that don’t require a feat of memory or imagination! Tools like choice models, which ask respondents to express preferences between two options, can also be effective in revealing deeper insights. They are easy to answer but rich in depth.
However, questionnaire design requires more than just asking the right questions. It’s also about how you ask them. Here are a few best practices:
- Clarify the purpose: Make sure respondents understand the objective of the survey. This motivates them to pay attention.
- Simplicity is key: Questions should be easy to understand and interpret, avoiding jargon or ambiguity. What might seem clear to you could be misinterpreted by your audience.
- Offer comprehensive options: Provide response choices that cover the main spectrum of potential answers, including an “other” option and a “don’t know” choice. Never force a response, but also don’t feel you have to include every imaginable option!
- Keep it brief: Both the overall survey and individual questions should be concise. Attention spans are short, especially in today’s TikTok-driven world, so aim for surveys that can be completed in under 10 minutes.
- Prioritise important questions: Place the most critical questions early in the survey when respondents are most focused.
- Mind the order: Ask about opinions and perceptions before seeking a conclusion or decision to avoid order effects that could skew the results.
Sampling: the silent influencer
Even the best-designed survey can falter if it’s not administered to the right audience.
Sampling bias is a significant risk, particularly in how respondents are recruited. Firstly, getting the “right” people costs money, and if the sample seems very cheap, there is probably a reason for that.
As way as paying for quality (i.e. getting these people) you then need to recruit the right balance, the right profile of sample. Of course, this must represent your target market well, but there are other things to consider, including purchase patterns. For instance, asking customers of Retailer A about their opinions on Retailer B will likely yield different results than asking Retailer B’s customers about their own shopping experiences. Balanced and representative samples are critical to mitigating these biases.
Today, the rise of bots and click farms presents an additional challenge on sample. Your survey providers must therefore employ advanced screening techniques and incorporate test questions to ensure data integrity, including getting rid of obvious ‘outliers’.
Even with these precautions, you can’t get rid of all “noise”. So, sample size itself remains the crucial precautionary factor—the larger the sample, the more reliable the results. Put pragmatically, a few percentage of dodgy answers won’t ruin the learnings. Don’t cut corners on sample, it’s always a false economy.
The human factor: navigating bias in interpretation
Just because they generate “facts”, surveys are not immune to interpretation error.
For example, confirmation bias—the tendency to favour information that confirms pre-existing beliefs—can skew the conclusions you may draw from survey data. This bias can manifest in both the commissioner of the research and the agency conducting it.
To counteract this, it’s essential to contextualise data points through benchmarks and norms. A seemingly impressive result might only be average when compared to industry standards. For example, in product testing, an average score of 4.2 on a 5-point scale might indicate market success, whereas a 3.9, though seemingly still positive, could signal mediocrity.
Understanding behaviour: The power of correlation
One common criticism of the survey idea is the claim that “people don’t do what they say”. While there is, of course, some truth to this (one reason we must be careful to ask the right questions), proven methodologies exist to bridge the gap between stated intentions and actual behaviour.
In many cases, by correlating a large body of history of survey responses with real-life outcomes, marketers can gain a more accurate understanding of implied consumer behaviour. This can be very useful, for example for advertising or product testing.
But in general, we must recognise that surveys are not meant to only replicate reality. Rather, they aim to provide insights that inform marketing strategies. They’re a tool for understanding the “why” behind consumer actions, which is just as important as tracking the actions themselves.
Effective surveys contribute to a more comprehensive decision-making process, serving as one of many inputs alongside other data sources, A/B testing, and even informed intuition.
A balanced approach
Surveys are not a magic bullet, but they are a powerful ingredient in the recipe for marketing success—when used wisely.
The key lies in understanding their limitations and executing them with precision. Research should be judged against the alternative—guesswork—not against a mythical standard of perfect understanding.
If surveys could provide 100 per cent accurate answers on actual behaviour, then by implication the role of savvy marketers would be redundant. Instead, in our real-world work surveys should be seen as a valuable tool that, when combined with other methods and your expert judgment, helps guide better decisions.
Surveys offer more than just data—they offer insights. And, in a world awash with information, insights are the true currency of effective marketing.
About the author
Roger Jackson is founder and CEO of SenseCheck