Back in March, I was part of a focus group for an online shopping site. I’m not a big fan of focus groups - there are too many interpersonal dynamics at work and I think you get better, more authentic responses in an interview. But it was $100, and an opportunity to be on the other side of the table for a change, so I was legitimately psyched walking in to the session.
Unfortunately, it wasn’t the good experience I was hoping for. I’ve sat on my post about it for a while, unsure whether I wanted to say anything publicly, but I think it’s worth it if it helps anyone stumbling across this post run a better user research session in the future.
The first thing I noticed, after checking in at the door, was that there was no release or mutual NDA for me to sign. Then, they immediately wanted to take my picture. I said I wasn’t comfortable with that, and they didn’t pressure me, but the same photographer was snapping away throughout the two hour session, and I am sure there were plenty of photos of me in the mix.
I know that the NDA is an easy piece of paper to forget or handwave away as unnecessary; I’ve done it myself in the past. That was a mistake. Don’t ever skip the NDA, especially if you’re taking pictures or video. I was surprised to discover how much having my picture taken during the session threw me out of the experience and put me on my guard, all because I didn’t have confidence that my picture wouldn’t end up on some corporate promotional site somewhere.
(And for those of you who say “oh, we’d never do that; we’re really careful with our confidential research material!” let me tell you a story I heard from someone who worked in-house: one of her designers took all the pictures they’d taken during a usability session and proudly posted them to the corporate Flickr feed. Only being reminded of the mutual NDA that all the testing participants had signed got the designer to take them down. Now you can say, and rightfully so, that this story proves NDAs aren’t guarantees someone’s privacy will be respected. But at least they give the interviewee some confidence and some recourse if it isn’t. Don’t skip this step.)
The second thing I noticed were the set of cafe tables on the far side of the room, and the group of people who all seemed to know each other sitting there and chatting. Oh, that can’t be the clients, can it? I thought. Yes it can. The moderator, from a small SF-based brand research company, tried to explain that it was more “authentic” for the team from the online shopping site to be in the room with us, that you can’t really grasp all of the nuances of body language and the subtlety of interpersonal communication while eating M&Ms behind a one-way mirror.* If anyone ever tries to sell you, as a client, on this being the best way to do research, please show them the door, because they are talking nonsense. Don’t be in the room observing your focus group.
Photograph © AMCThere is a huge body of research that shows that people’s behavior, perceptions, and attitudes are changed by their environment, and part of their environment is the people who around them. Doing user research, one of your biggest challenges is getting people comfortable with an extremely artificial situation, one in which you are paying them for their opinions on something that the typically believe (rightly or wrongly) that you have some personal investment in, so that you can get past polite awkwardness and into something like truth. When you feel yourself being watched by the people who want your opinions, it’s going to change what you say, and make it more what you think they want to hear.
What’s more, the rationale that the clients could see us better out from behind a one-way mirror was ridiculous on the face of it, because they were sitting behind the table, which meant they were staring at the back of two focus group members’ heads. Lots of ability to see the nuance of what was going on there, indeed.
I’ve spent some time trying to figure out what led the brand consultancy to use this weird setup, and I’ve only come up with two possible answers. One is that they had us get up and look at images pinned to the wall, and in a typical focus group room, that would have been harder. The other is that, since they were specifically targeting people who live in Brooklyn, they wanted to make sure the sessions were in Brooklyn, and they couldn’t find a proper research facility outside of Manhattan. These are both understandable concerns, but neither of them are insurmountable, and neither one is worth giving up whatever measure of critical distance that not having the people paying for the research in the room during the research can provide.
The final thing I noticed was the composition of the group. Out of the ten women at the table, two were in marketing, one worked in branding, and three (including one of the marketers and me) had experience running focus groups. I can almost make an argument that if you want Brooklyn tastemakers, you want to include a few people whose job involves branding and marketing, but even so, you need diversity, and more people who don’t know how the sausage is made. Screen your focus group members effectively.
Portigal Consulting, one of the top user research consultancies on the West Coast, talks about the complexity of writing a good screener on their blog, with links to a sample screener. If you go to that screener (which is a great example of a script for a recruitment firm), you’ll see that a lot of the early questions are looking for, and getting rid of, people just like the ones who were around that table. Why? Because most people don’t work in marketing and user research, and don’t think like them. Just like I wouldn’t want to just take a programmer’s word on what makes for a great piece of consumer software, I wouldn’t take a marketer’s word - or my own word! - on what makes a compelling online shopping experience. We’re not the typical user, and we think about it differently than the typical user would. It’s precisely to get that perspective from the other side of the interaction that you do user research and focus groups in the first place.
I went home that night annoyed in the way that only a job badly done can annoy you, and wrote an angry bunch of notes for this post. Then I unsubscribed from the site’s mailing list. I spent the honorarium on framing art that I bought somewhere else.
* The only downsides I have ever found to being in an observation room in a testing facility are (a) overdosing on M&Ms and (b) since Mad Men started, people who are new to user research come in and say “I feel just like Don Draper!”