![Kids’ Life and Times (KLT) [3] is an annual survey of 10- and 11-year-old children in Northern Ireland that began in 2008](http://www.effortmark.co.uk/wp-content/uploads/2014/06/kidslifeandtimes-211x300.gif)
Putting the ‘Long’ into Longitudinal: UX lessons from survey research
How long is your typical project? Are you working in 6-week agile sprints? Running monthly usability tests? Trying an A/B test for a week? Updating a Twitter stream hourly? The demands of internet time keep us focused on shorter and shorter time intervals, with experiences measured in days, minutes, or even the first 50 milliseconds of exposure to a Web page, according to a team of researchers at Carleton University in Toronto led by Gitte Lindgaard [1].
What happens if you turn that around and think in terms of months, years, or lifetimes? Longitudinal studies look at long-term user experience. Usually, that means over a few months or possibly a few years. But recently, at the European Survey Research Association Conference, I learned about some much longer-term studies that offer some lessons about how to conduct our rather shorter investigations.
The European Survey Research Association Conference: when long-term means years
on SlideShare.The conference included more than 150 sessions, each with three or four papers, in up to 12 parallel tracks, so it was impossible to attend more than a small selection of them. So what to focus on? As I looked back over my notes, I spotted a theme: many of these presentations were about research that happens over years, decades, or lifetimes. To give just two examples from this conference:
- Todd Hughes from the US Census Bureau talked about the usability tests they conducted in 2010—not to inform the 2010 Census, but looking ahead to 2020. These things take a long time to design.
- Stephan Huber, a professor at the Institut für Bildungsmanagement und Bildungsökonomie in Switzerland, talked about the latest developments in a study of Swiss teenagers that started in 1854.
In this review of the conference, I’m going to discuss what I learned from three other presentations that gave me ideas for my own longer-term studies.
UX tip #1: try asking less, but more often and earlier
Jennifer S. Barber of the Population Studies Center at the University of Michigan talked about a study of 18- and 19-year-old women and what experiences might lead to pregnancy—especially unintended pregnancy—that is, unwanted or unexpectedly earlier than planned. Unintended pregnancy can be a health risk for mother and baby. The challenge is that, once a woman is pregnant, it’s too late to ask her about the finer details of her life in the months before the pregnancy.
So they set up a panel to track around 1,000 women over five years. But what factors to look at? Ideally, the researchers wanted to know pretty much everything about the women’s lives, and one solution might have been to do periodic interviews that covered, well, just about everything.
They started with an hour-long interview with each woman at the start of the project. (I just realized that’s at least 25 person-weeks of interviewing.) The temptation might have been to continue with similar hour-long interviews at rather long intervals throughout the study. Instead, they opted for weekly 5-minute interviews—with a mixture of standard and variable questions each week—and gave participants the option of answering questions on the web or by phone. This has worked very well. Five minutes isn’t too onerous for anyone; the variety of questions helps to keep the interview interesting; panellists like the regular incentive, even though it’s small—just $1 per interview, plus a small bonus every 5 weeks.
Tip for UX studies—If you want to learn about users’ engagement with your product over a longer-term experience, might you try asking your participants less, but more often?
The Centre for Longitudinal Studies at the Institute of Education, University of London, carries out multiple surveys of large numbers of individuals from birth and throughout their lives, tracking three cohorts:
- 17,634 people born in 1958
- a similar sample of people born in 1970
- a similar sample of people born in 2000, from the Millennium Cohort Study (MCS)
The MCS study asked participants’ parents about them when they were nine months old, then again at three and five years. When they were seven, their parents did the main questionnaire, and the children got a little questionnaire of their own, asking about their hobbies, their friends, their feelings, and their attitudes toward school.
If you’re designing for children anywhere, but especially in the UK, the overview of the findings from the study at seven years [2] gives robust insights into the lives of children at that age. Should you be willing to tangle with complex sample analysis in SPSS (originally, Statistical Package for the Social Sciences), the actual data is also available. It’s accompanied by a fearsome—to me, anyway—user’s guide about how to manipulate the data to ensure appropriately statistically reliable results.
The researchers are preparing for the survey next year for age 11—rising 12—and reported on a variety of different topics. The paper I caught looked at the best way to ask about alcohol consumption—yes, 11-year-olds are experimenting with alcohol in the UK.
One point that really struck me: deciding not to ask. The researchers have to balance the desire to ask about everything in the children’s lives with the desire to keep them active in the panel. One topic that got excluded: puberty. This is a subject that was too horrible for 11-year-olds to contemplate; even the word puberty was embarrassing to them.
Tip for UX studies—Do you need to ask about everything, right now? Could you, maybe, postpone a difficult topic for another time, or even exclude it altogether?
UX tip #3: think about sharing your results
Most of the research groups presenting at the conference publish extensively to academic and government audiences. Many, like the Centre for Longitudinal Studies, make their actual data sets available for further analysis.
The final survey I want to tell you about took this a step further: they published their results about a survey of children in a child-friendly format. Kids’ Life and Times (KLT) [3] is an annual survey of 10- and 11-year-old children in Northern Ireland that began in 2008. The children answer the questions themselves during school time. The researchers have obtained permission to use cartoon characters from Bang on the Door to help with kid appeal.
The research team is very committed to making sure that children’s voices are heard—and they extended that to the children themselves. So, each year, in addition to their more formal publications, they publish a comic-book version of the results, shown below [4].
Tip for UX studies—How about publishing more UX survey results? Perhaps not the whole data set, perhaps not even the report. But what about giving participants a headline summary of the key findings from the research, if they want it? This could be the way to help keep our users interested and involved in our ongoing projects.
A final reflection
Most of us in user experience are used to thinking short-term—in weeks or months. But, increasingly, organisations are planning to build long-term relationships with their customers. Few of us will need to think in terms of decades or how to keep users engaged over their entire lifetimes, but I believe that we can learn a lot from longitudinal research projects like those I’ve described.
References
[1] Lindgaard, Gitte, G.J. Fernandes, C. Dudek, and J. Brown. “Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!” Behaviour and Information Technology, Number 25, 2006.
[2] Hansen, Kirstine, Elizabeth Jones, Heather Joshi, and David Budge. “Millennium Cohort Study: Fourth Survey: A User’s Guide to Initial Findings,”PDF 2nd edition. Centre for Longitudinal Studies, December 2010. Retrieved August 7, 2011.
[3] ARK (Access Research Knowledge) Northern Ireland. “Kids’ Life and Times (KLT) Survey,” June 6, 2011. Retrieved August 7, 2011.
[4] ARK (Access Research Knowledge) Northern Ireland. “Kids’ Life and Times Comics,”July 29, 2011. Retrieved August 7, 2011.
- This article first appeared in UX Matters, August 8, 2011
#surveys #surveysthatwork