Poor survey design is one of the biggest problems in market research because it leads to data that you just can’t trust. I’m Troy and I’ve spent the last two decades designing hundreds of surveys for clients all around the world. Along the way I’ve made my fair share of mistakes, so I thought I’d provide some tips to help you avoid the most common problems and ensure you get high quality, actionable data from your surveys. Read on for my 8 tips for great survey design:
Ensuring that your surveys are well designed, intuitive, easy to complete, and dare I say fun, is the difference between a good survey experience and one that leaves your audience frustrated and unhappy (and will likely give you bad data!).
But first off, you need to consider if a survey is the right tool to deliver the insight you need. If you want to explore the range of opinions or motivations around a topic to identify variables of interest, you should use qualitative methods like one-to-one interviews, ethnography, or focus groups. Once you’ve done that, quantitative methods like surveys can then help you measure the outcomes to answer questions like how many, how often, or how much. Good qualitative work informs great quant because it ensures you are measuring the right things.
Once you’ve identified that you need quantifiable data from a specific target audience, you need to decide what questions you want to ask and who you want to answer those questions. But how do you get started?
When you are writing your survey, your primary goal is to create something that tells you what you need to know but in a way that is easy for your audience to understand and answer.
Your survey needs to consider the way your audience is going to think about the questions and follow this logic.
To do this, there are a few basic principles that will help you. Following these design tenets will help to ensure that you get the insight you want while providing a good experience to your audience.
1. Set clear objectives
Your objective is your north star. It will tell you the information you need and thereby determine which questions you ask. It will also help you structure the flow of your questions and the language you use. Your objective lets you focus on what the survey needs to deliver – data that can be analyzed for actionable use. If your data doesn’t let you draw actionable insights, then your survey has failed.
2. Keep it short
A short survey is a good survey. Researchers (myself included) have been trained to learn as much as possible. Resist this urge! One of the biggest survey design mistakes you can make is creating overly long and complex questionnaires. Respondents lose interest or drop out, leading to incomplete or biased data.
Keep your survey concise and focused on your research objective. Only ask questions that are necessary and directly related to your goals. Consider how you are going to use the responses from every question and how the data will meet your objective. Cut anything that does not clearly address that goal.
A good rule of thumb is 3-4 short questions per minute, so a 5 min survey will be 15-20 questions long, but the type of question and the format will impact this estimate.
3. Mix-up the formats
You want your survey to be simple, but you also want it to be interesting for your respondents. Your questions need to ensure that your audience considers their response each time rather than answering the same type of question robotically time after time. Using a mix of question types will help you to achieve this. Single and multi-select, ratings and rankings, grids and carousels, dropdowns, open-ended – the list goes on. Remember to limit open-ended questions because these will cause respondent fatigue. Allowing 2 or 3 open-ended questions that are direct and precise is a good guideline. It is also important to limit large grid/matrix questions; opt for a carousel as an alternative.
You can experience what different types of questions look like on the Glow platform by clicking here.
4. Screen for quality
Always include a screening section. You want to ensure you are surveying the right audience, whether consumers or business leaders. A screening section ensures you get the right people to answer your questions. You want questions that help you to quickly and accurately identify your audience.
Equally, you don’t want to feed the audience the answers. Avoid simple Yes/No questions (e.g., Have you purchased X in the last 3 months?) in favor of a short code frame of options that includes what you are looking for as well as options that you are not interested in. This lets your respondent qualify by picking the option you need. Yes/No answers are more easily manipulated by experienced respondents and bots. A word of caution about screening questions: only include demographic questions that you need for screening purposes. Any other questions you need for analysis should be later in your survey. Try to find who you want within the first 5-8 questions; a screener should never be more than 10 questions.
5. Leverage logic
Sound survey logic can improve the respondent experience immensely by ensuring respondents only see questions and/or answer choices that are relevant to them based on how they answered previous questions. Being forced to answer questions that aren’t relevant is extremely frustrating. If you need a respondent to remember a previous statement, remind them using survey logic. Human memory is fallible, so a well-placed reminder can reduce the most frustrating survey outcome: inconsistent responses. Good survey platforms have comprehensive logic built in.
6. Avoid ambiguity
You need to ensure that there is no ambiguity about how to answer a question. Define all answer options as clearly as possible and ensure that you provide a comprehensive code frame ensuring there is always a relevant answer choice. Where needed and of value, you can include an “Other” option where the respondent can add their own response, but if you are not going to use that extra data, don’t ask the respondent to specify.
7. Don’t lead
Leading questions can bias responses and affect the accuracy of your data. Ensure that your questions are neutral and free from hints, suggestions, or assumptions that might influence respondents. You also need to ensure that response options cover a spectrum of thoughts, feelings, or opinions.
For example, don’t ask “Do you think XYZ is good?” Yes or No.
Instead, do ask “What do you think about XYZ?” and provide something like a 5-point Likert scale ranging from Very Bad to Great.
This lets your respondents tell you more clearly how they feel about XYZ and does not make any initial assumptions that could lead to a biased result.
8. Speak their language
Speak the language of your audience and keep it simple. Sometimes that might mean using casual language, at other times, quite technical terms. The key is ensuring the language is crystal clear to your audience because questions that are ambiguous, hard to understand, or easily misinterpreted will give you bad data and/or result in respondents dropping out of your survey.
If you follow these 8 design considerations, your surveys should deliver high-quality data that you can use to create actionable results.
Written by:
Troy Kohut
Troy heads up Glow’s global operations team. A research Swiss army knife, he has over two decades of experience across all areas of the research business, from consulting to sales to operations. He’s designed hundreds of surveys for clients in tens of categories in markets around the world.
Need survey support?
Have a survey you need to run? Our team of experts are ready to help you with question design, scripting, quality assurance, and more. If you are already using the platform, then simply speak to your account manager. If not, book a meeting to see how you can quickly build and deploy surveys that produce actionable results.
Further reading
