The following post is from Forrest Anderson’s blog, Reputation, Research, Relationships and Messages. Additional comments and tips are strongly encouraged.
Surveys, when done properly, are a great way to gather the kind of information you can use to create communications campaigns that achieve business objectives. They are based on real information and insight into the target audiences surveyed rather than guesses and assumptions about them. Real information on what they care about, how best to reach them and so forth.
Online surveys are so easy to use that more and more folks are doing surveys themselves. Unfortunately, the insights users get from these surveys frequently are less than they had hoped. They sometimes are even be misleading because of methodological issues I will discuss.
So here are my top tips for all surveys. These are the ones that have come to my mind, and I’m sure many of my research colleagues will have others. I invite them to add their comments.
- Understand the business context for the research. Know what business decision will be affected by the research and how.
- Get the research question right. Know what you really want to know and how you will use the information in decision support.
- Design the research correctly. People frequently think they should do a survey when they should be doing focus groups, one-on-one interviews, searching data bases or even going to the library.
- Select and sample the right population. Online surveys depend on e-mail addresses, so people doing this kind of research tend to survey the people they have e-mails addresses for. However, they should be thinking about the actual target stakeholder universe and how they can get a representative sample of that universe. Consider that presidential election polls are not done online.
- Write questions that do not lead respondents to answer in a specific way (bias the question).
- Write questions that offer the answer choices respondents would select, so they can answer honestly and appropriately. For example, I have seen rating scales used that offered only positive responses (“Did you like this class, or did you really like this class?”). In such a case you get a positive rating, but you’ve learned nothing and your results may be misleading.
- Develop questionnaires with a logical flow. Flow problems can be jarring to respondents or inadvertently reveal information that biases answers to later questions.
- Use question formats that are easy for respondents to manage. For example, do not ask them to rank order the importance of 20 elements.
- Keep your questionnaires to two to three minutes, if possible. Try to focus on one topic.
- Use graphs and charts to present the data in a form that is easy to grasp. Online survey services and Excel offer ways to easily visualize data. But, for your audience to grasp the data, it is important that you use the right chart or graph for the situation and format it to be easily understood. For example, if you ask respondents which topic they would most like to hear about, your chart should be rank ordered to show the most popular topic at one end and the least popular at the other. This way, your audience will quickly grasp what is important and what is not.
- Make the leap from data to insight. This goes back to our first point about understanding the business purpose behind the research. When you have the answers to your survey questions, you need to make a case for what kind of decision the data supports. But business decisions are not just about data. They are about what is best for the organization. So, making insightful recommendations requires integrating your understanding of the data with your understanding of the business context. And the broader your contextual understanding is (that is, if it goes beyond communications to finance, accounting, HR, research and development, marketing and sales, etc.) the more valuable it will be to your organization.
I will go into more detail on these bullets in future posts. A number of my upcoming blogs will address how to do better online surveys, such as those we do with SurveyMonkey or Qualtrics. I use and like both.
If you’d like help developing, executing or interpreting surveys, please contact me. If you would like to share any additional tips, please do so.
Forrest Anderson is an independent Planning & Evaluation Consultant with 30-plus-years of experience.
Thanks Doug and Fraser for your comments. They are dead on, and, unfortunately, I agree on the length issue.
Some surveys can be pretty short, such as event evaluations, etc., but it is almost impossible to keep most business intelligence surveys to that. Still the goal should be to keep them as short as possible and not let our business partners add to them because “It would be nice to know.”
One of the less obvious issues with longer surveys I believe is respondent burnout. Not only do longer surveys frequently not get completed, but I believe they discourage respondents from taking other surveys. And those of us who do surveys know that it is getting harder and harder to get non-panel respondents for surveys (and of course you have to pay for panel respondents).
So, yes, two to three minutes is not always realistic, but shorter is an aspiration to be valiantly strived for.
Great points, Forest! I too would comment on the length of the survey. There’s always the tendency, if not out right need, to get as much context for the two, three key questions/variables, in the survey. Without the context, it’s hard to move forward with your communication planning.
Forrest, thank you for a great list of recommendations about doing quality survey research. I appreciate your many points that focus on developing questions, using appropriate (and accepted) scales, and survey length. In addition, your recommendations about understanding a study’s target universe and disciplined sampling are crucial. These two key elements – the questionnaire and sampling – often receive little attention by those conducting their own research.
I do have a couple comments. First, I totally agree with your first point that an understanding of the business context for the research is critcial, including such things as what strategies are to be developed or decisions made based upon the results, I go a step further and get each of my clients to sign-off on a clear goal and set of measurable objectives for the research. I use the objectives to drive questionnaire content and help clients understand that each question must address at least one of the study objectives. If it doesn’t, it’s not needed.
Finally, while I applaud your target of a two- to three-minute survey, I find that difficult to achieve. However, I am often disappointed by the number of surveys I see – even from well-respected international firms – that are well over 10 or 15 minutes in length. These excessive survey lengths is a key reason why participation rates among respondents continues to decline.