Quick Summary: In part 1 of this two-part article I explained the top reasons that remote user testing can be a great way to gather user feedback. But remote testing has also has an ugly side. Here in part two we explore some of the significant drawbacks.
Remote user testing is great… isn’t it?
Yes and no. I previously covered some of the key benefits. To summarize, remote user testing enables you to:
- Test representative users from diverse geographic locations
- Test less expensively than traditional in-person user research
- Test more rapidly, with less lead time
But there are also some serious drawbacks, and it’s important to understand them.
Maybe remote testing isn’t so great
Let’s examine some of the problems that can arise, and you’ll see why remote user testing isn’t necessarily a good choice, and certainly isn’t a replacement for all types of in-person user testing.
1. Greater burden on respondents. Despite some important advances in technology, many remote user testing solutions require users perform some type of setup in order to participate. This is a potential failure point. Some technologies like Cisco’s WebEx require users to install before they can join a session. This isn’t necessarily a problem, but it necessitates that you verify everything’s working on the user’s end before a session can begin. If there’s a problem, it can be difficult to troubleshoot over the phone – and it costs valuable session time. By comparison, with in-person testing users typically just show up to a research facility and sit down at a properly configured computer.
2. Some respondents are unwilling to install software. It’s possible to pre-screen respondents to ensure they’re willing to allow software to be installed on their computers, but many people are understandably distrustful of any request to install software on their computer. In today’s age of virus-laden websites and email attachments this isn’t surprising. So in planning remote user testing, depending on the technology used, it becomes necessary to recruit people who are willing install the required software. And if you’re interested in testing a group that includes less-savvy Internet users this can be problematic.
3. Not everyone has a suitable connection. Depending on what you’re testing, the need for good bandwidth can be an issue. This isn’t a problem if you’re testing respondents with an ideal demographic and geographic profile (for example, tech-savvy professionals living in the Bay Area). But depending on your demographic profile it may be difficult to find respondents who have access to a reliable, fast connection.
4. Greater dependency on consistent bandwidth. Naturally, technical glitches can occur during an in-person session as well as over a remote testing session. But with an in-person session, if your Internet connection fails or your application crashes you have the option of switching to a set of printed screens (you remembered to bring those, right?). By comparison if you lose the connection during a remote user testing session, you’re at a dead stop until the connection can be re-established. This may require additional troubleshooting and remote support for the respondent, and can eat up valuable session time.
5. More “no show” respondents. In my experience, it’s not at all unusual for a remote user testing respondent to fail to answer their phone at the appointed time, resulting in a “no show”. I speculate this is in part because remote respondents have a more tenuous connection to the idea of attending session. In other words, the bar is lower for them – they haven’t been forced to schedule time to leave work and/or drive to a research facility. So perhaps the commitment seems less “real”. Regardless of the reasons, a higher rate of “no shows” makes it more important to have back-up respondents if you wish to stick to a testing schedule. And that means more recruitment, more incentives, and more administrative overhead for the sessions.
6. Loss of subtle (but often useful) cues. This is perhaps my greatest gripe with remote testing. With in-person testing there’s a wealth of information to be had from observing body language, facial expressions and other subtle cues that simply aren’t available in a remote user testing session. Watching a respondent physically interact with an application or website can often reveal important insights into the user experience. But in a remote user testing session the respondent’s presence is reduced to a voice and a mouse pointer. You don’t know:
- Where the respondent is looking on their screen (unless you ask)
- What their body language, expressions, and gestures are indicating
- What they’ve said that’s too quiet for the telephone to pick up (subtle verbal cues like mumbling and muttering can be telling!)
7. Difficulty creating a personal connection. Some people are natural user testing respondents – they’re confident, speak freely, and can clearly articulate their opinions. Then there’s everyone else. Many people are more withdrawn, less confident in their opinions, or have a difficult time expressing themselves. But these people can be great user testing respondents; they just need a little more guidance or encouragement. When your only connection to a person is a distance voice on the phone, it’s very difficult to establish the personal connection that enables a moderator to coach a reticent or reluctant respondent.
So, remote user testing should be avoided, right?