Question: What’s the difference between a survey and a questionnaire?
Answer: A survey is a process; a questionnaire is the series of questions and answers in the middle of that process.
It’s easy to focus on the questionnaire to the exclusion of other important aspects of the survey. A couple of examples in my in-box recently reminded me about the importance of the survey invitation and the reminder.
The untrustworthy invitation
For purposes of research, I tend to respond to every survey invitation I get if I possibly can, but a recent one shouted ‘spam’ at me. Why?
- It wasn’t addressed to me. It was addressed to #fullname#.
- The incentive seemed rather high for a survey: 25 euros (approx US $36) guaranteed? For 10 minutes?
- And a bit further down, there was an address of a US corporation but the contact email address was in the Netherlands.
I was hesitant.
Response relies on trust, perceived effort and perceived reward
Let’s look at it from the point of view of Dillman’s Social Exchange Theory. This says that people will respond to your survey if the perceived reward, which doesn’t have to be monetary, is in balance with the perceived effort – but only if they trust you. And if you’re exceptionally trustworthy, then being nice to you is itself a reward. Think about a mother, patiently answering a series of ‘why’ questions from a three-year-old; it’s likely that mother will keep answering until the perceived effort just becomes too much. (The most recent edition of Dillman’s book is Dillman, Smyth and Christian (2008), my January 2011 Survey Book of the Month).
Strangely enough, if the reward is excessive for the perceived effort, then that undermines trust – exactly what happened to me in the example above. And the trust wasn’t great in the first place: that dodgy salutation and the mismatch of sender and email address also worked to undermine trust.
The outcome? Well, in the interest of research I did click through – and the survey seemed perfectly genuine but I didn’t qualify because my business has fewer than 500 employees. I wonder what response rate they did get. Would those features that put me off have inspired trust in a busy HR manager is a large business? I suspect not.
A reminder that looks like a reproach
Then a couple of days later, a friend forwarded this survey reminder to me:
Recently, we sent you an invitation to complete a Guest Satisfaction Survey concerning your stay with us at XXX YYYY, where you checked out on August 3, 2011.
We noticed that you did not have time to complete the survey. We are concerned that you may not have responded because we have somehow failed to live up to your expectations.
At XXXX, we are committed to providing a superior guest experience to every customer.
Please take a few minutes to tell us how well we met your expectations.
To complete the survey, please click on the web address below. If that does not work, please copy and paste the entire web address into the address field of your browser.
http://survey.xxxxxx
Thank you again for choosing XXXX. I look forward to hearing about your stay with us.
Sincerely,
Dave ZZZZ
Manager,
XXXXX Hotels
————
Please do not ‘Reply’ directly to this invitation.Technical assistance: Should you have any problems accessing or completing this survey, please e-mail our survey vendor at hhctechsupport.xxxx.com
To unsubscribe: We rely on feedback from guests to ensure that your hotel stay meets and exceeds your expectations. If, however, you prefer not to receive a survey invitation in the future, you may unsubscribe by clicking this link:
http://survey.xxxFor all other requests, please visit http://www.xxxx
My friend forwarded it to me with the comment “How to be annoying. I didn’t know that I am somehow obligated”. Needless to say, she didn’t respond.
Let’s look for the reward. Hmm. I can’t see one. What’s in it for my friend? It seems she’s missing out on an opportunity to complain. But she didn’t have a complaint about her stay – only about her survey reminder!
Another way people can feel rewarded is if they’re made to feel special; if they’re picked out as one of a small group who is being asked to respond to an important survey that will influence a major business decision. So is there anything in this reminder to make her feel like an individual whose opinion is valued?
No, rather the reverse. It says: “At XXXX, we are committed to providing a superior guest experience to every customer”.
Maybe you read this sentence differently, but to me it has the effect of reminding her that she’s just one of the crowd, nothing special.
And finally, look at these mixed messages: “I look forward to hearing about your stay with us.” “Please do not ‘Reply’ directly to this invitation.”
He doesn’t really want to hear from her, because she’s not allowed to write to him. He only wants to increase the response rate on his survey.
A survey invitation has three purposes
Your survey invitation needs to do three things:
- Offer a perceived reward.
- Explain the amount of effort.
- Inspire trust.
Offer a perceived reward
People like to do nice things. If you can make your respondent feel special in some way, that can act as a reward. Obviously, a financial payment can work – but not if using that payment, or getting the payment, will be a huge hassle for the respondent.
A delayed reward is definitely less effective than an immediate one – as I discussed in my post Do incentives help to improve response rates?
Explain the amount of effort
I have to admit that I’m still researching the question of explaining the amount of effort. I think two factors are operating:
- The possible benefit from saying how long a survey is, either in minutes or as a number of questions
- Widespread cynicism about the value of these promises.
Personally, I’ve experienced questionnaires that promised to be ‘short’ but actually took me 30 minutes or more. It’s also tricky to predict how many questions a person will have to answer if your questionnaire has skips in it (places where you are routed around an inappropriate question).
I’m hoping to find some data to tell us how long people expect a questionnaire to be, and whether that varies by topic or by the organisation that’s asking.
Features that inspire trust
I’m still looking for research on the features of a survey invitation or reminder that inspire trust. Meanwhile, I think we can transfer ideas from research on features of websites in general that inspire trust, such as the Stanford Guidelines for Web Credibility.
For example: “Design your site so it looks professional (or is appropriate for your purpose).” ).
If you’re doing a survey on behalf of an organisation that’s known to your users, and I assume respected by them, then I interpret ‘look professional’ to mean: make it look “designed” and make sure that it clearly communicates the brand.
Here are two examples that I got this week. The two brands are John Lewis, middle-to-upmarket department stores, and Champneys, a spa. I’m a happy customer of both, regularly shopping with John Lewis (at the store and online), and occasionally treating myself to a visit to the spa. Here are mini versions of the home pages of their respective websites.
To my eyes, there’s a certain similarity: both brands rely on a reasonably restrained use of a specific colour palette (dark green/grey/white for John Lewis; dusky brown/light brown/gold for Champneys) to complement large, striking images.
Now let’s look at their two survey invitations, side by side. I’ve made them quite small so we can focus on the overall impression of the design.
Both of them look designed and professional, but I see the Champneys one as being much better at conveying the brand – and giving me the idea that filling in their survey might even be a pleasurable experience.
Tips for better invitations and reminders
Your user’s experience of your survey starts with the invitation. If you want to improve the response rate:
- Look first at trust. Have you made it clear who you are? Is the person likely to know you, and if not have you made it easy for them to find out that you’re trustworthy? Does your invitation look as carefully considered and designed as the rest of your brand?
- Then check perceived reward. Why would this person want to be nice to your organisation in the context of this request?
- Then check perceived effort. What will the respondents get in return? Is this all about you, or is there something in it for them?
- And finally: Test! Test! Test! Make sure your testing includes the invitation and any reminders. Don’t let silly mistakes like sending your invitation to #fullname# ruin your survey.
#surveys #surveysthatwork