Seven questions about user-research panels

1. Who will manage the panel?

User-research panels don’t manage themselves. As your panel grows, it will likely require a dedicated person to run and manage it—in addition to the people who actually design and run research activities. Someone has to do all the following things and more:

• Recruit panel members.
• Answer panelists’ questions.
• Keep panelists engaged.
• Control access to the panel.
• Refresh the panel when panelists drop out.
• Recruit people for specific research activities.
• Secure the private data that panelists have provided.

Have you identified the person who will do all of this for your organization and ensured that person has the budget and time necessary to do the job?

2. When you recruit people to the panel, what do you promise them?

• What happens if a panelist happens to be unavailable for an activity you’ve planned?
• How often do panelists expect to take part in research activities?
• What if a panelist is never a match for the types of people you need for specific research activities?

And here are some that relate to incentives and other rewards:

• Is it ethical to expect panelists to be volunteers when everyone else on your project is being paid?
• If you offer an incentive, what should the incentive be? Should you offer an incentive simply for being on the panel or only when panelists take part in a research activity?
• Which incentives should you tell panelists about?
• If you don’t offer an incentive, are you skewing your panel toward people who can afford to spend their time participating in research activities without being paid?

Let’s also think a bit about panelists who will participate in research relating to their professional roles or who belong to a specialized, niche audience. Here are some examples that complicate the concept of a financial incentive:

• Policies may bar government employees from accepting any reward at all in connection with their participation on a panel, even a cup of coffee.
• Highly paid, professional or technical users—for example, a doctor or a system administrator with expertise in a particular technology—might be willing to contribute their time to research activities relating to a product they really care about, but feel insulted if you offered them a relatively trivial amount of money.

3. How representative are the panelists of some particular user population?

People who are willing to sign up for a panel might be different from the people who wouldn’t, in ways that could affect their participation in a research activity. Here are some examples:

• Panelists may be more positive about your organization than people in general would be.
• Their personal reason for joining your panel might be that they consider themselves advocates, with strongly held views on certain topics such as privacy or the rights of the group they represent. While you must respect and welcome their views, the strength of their opinions may make these panelists somewhat different from the full range of people who might use your product or service.
• Panelists may have specialized knowledge about your product or organization that makes them more interested in participating, but their knowledge differs from that of a typical user.

In the language of survey methodology, this problem is similar to non-response error, where the people who don’t respond differ from the people who do respond, in ways that affect the results of the survey.

Some things to consider about the representativeness of your panelists include the following:

• Are the people who join your panel different from the people who don’t?
• If they are different, will those differences affect your specific research activities? Or won’t they matter?
• Is a user-research panel the best way to reach people who see themselves as advocates? Or would some other approach be more appropriate?

4. Once you use a panelist as a participant, does that limit the panelist’s future opportunities to participate?

If you think back to matters relating to Question 1, which included refreshing the panel, you’ll realize that limiting panelists’ participation would add quite a bit of extra work for the person who manages the panel—as well as maybe cost. Would the panel still be cost effective?

If you really don’t want to limit panelists’ participation, it might seem harsh to expel them from the panel after they’ve taken part in just a single research activity. But if they take part in your research repeatedly, will that make them too familiar with the product or service?

In contrast, there’s another level of usage to think about. If most people who use your product or service spend large amounts of time using it, perhaps every day, how can you ensure that your panelists have enough experience to reflect that?

5. How much data will you capture at the start versus along the way?

Another approach is to progressively ask more questions of participants. For example, you might decide to ask for additional details on a monthly cycle, changing the topic on which your questions focus each month. Perhaps you might ask about panelists’ social interests, families, or the technologies they use. Or maybe ask about something that’s relevant to the decisions you’ll be making during that particular month. Of course, when using this approach, it will take a lot longer to build up a complete profile of each panelist.

Here are some things to think about when signing up new panelists:

• What is the least information you could request that would be useful and allow you to contact the panelist again?
• Would panelists expect you to contact them regarding anything other than an invitation to participate in a user-research activity?
• What sort of contact would panelists welcome? What questions are they interested in answering?

6. Will you be linking research data from other studies?

For example, in a specific research activity, you might have asked a participant, “Do you have children?” If you don’t link the data from that prior research, you might get this answer from the participant, “I told you last time that I’ve got three.” Or even worse, the participant may weep because, as she told your colleague last time, her child recently died.

If you link the data that you capture during each specific research activity, you may build up a considerable body of very private data.

Here are some questions to consider:

• Who will look after that data?
• What are the information-security requirements?
• Who should have access to the data?
• Can you fulfill promises of anonymity and confidentiality regarding specific research activities?

7. What technology will you use?

One possibility would be to start with the simplest possible technology—perhaps that paper notebook—then work up to fancier technologies only once you’ve addressed some of the other issues—especially and maybe most importantly, the challenges relating to Question 1: Who will manage the panel?

In a larger organization, a paper notebook simply may not be appropriate. What would happen if it went missing or a colleague at another location needed to use it? So you may need to think about a more robust technology straight away, to confront and explore solutions to information-security and procurement issues from the start. The risk is: you might become overly focused on the technology and fail to put enough effort into the people processes around it.

Some questions to consider include the following:

• What technology could you use to manage the panel?
• If you must procure new technology, what is your budget and who will decide how to spend it? Who will set up the system, and who will maintain it?
• If you plan to use some technology you’ve already got, what will be the impact on that technology? Is it suitable for the purpose?

Summary

We’ve taken a good look at seven questions that you should consider when setting up a user-research panel. At this point, you may be thinking, This is too much to think about, so I’ll just keep using my current recruitment methods.

Don’t be put off by the need to consider all these issues. Using a user-research panel can be a wonderful way of keeping in touch with a wide selection of potential research participants, recruiting hard-to-find people when you need them, and ensuring participants are available when you need them. This is certainly preferable to your having to time your user-research activities around the demands of a recruiter.

Our aim was to get you started putting together your user-research panel, while ensuring that the issues are out in the open right from the start. We hope this will help you to think through all the issues and create a successful panel that meets your needs.