10 tips for unmoderated user testing
Whilst most user testing is still run with a moderator (a.k.a. facilitator), more and more teams are utilising unmoderated user testing to very quickly get user feedback. As the name suggests, unmoderated user testing takes the moderator out of the equation. It’s the self-checkout, the self-assembly furniture, the bring your own picnic of user testing.
Unmoderated user testing participants will consent to having their screen recorded and then via a number of on-screens prompts will be asked to carry out session tasks. This means that teams can set-up sessions using a service such as usertesting.com, lookback.com or userlytics.com and then (in theory) put their feet up as the insights come flooding in.
Unmoderated user testing can be a very quick and efficient way to get user feedback. However, it’s a different beast to moderated user testing. If you’re thinking of utilising unmoderated user testing, here are 10 tips for getting the most out of it.
1. Utilise for evaluative, rather than discovery research
If a team is following the well-known double diamond design process (or something close to it) they will often utilise discovery research to gather insights for a domain and to define the problem(s) to be tackled. They will then utilise evaluative research to evaluate and refine potential solutions and designs.
Unmoderated user testing can be very effective for evaluative research (the second diamond). For example, for testing a concept or potential design with users, often in the form of a prototype. However, it not so well suited to discovery research (the first diamond). This is because discovery research tends to be more exploratory. There are more paths to follow and more avenues to explore. This sort of discovery work invariably requires a 2-way conversation with users, something that simply isn’t possible using unmoderated user testing.
2. Don’t try to cover too much
When setting up unmoderated user testing sessions it can be tempting to try to replicate what you would usually cover in a moderated session. Don’t make this mistake. It’s like expecting a self-service check-out in a supermarket to perform the same as one operated by a cashier. When you don’t have an experienced facilitator running the show you can’t expect to cover as much, nor expect to keep a user engaged for as long.
Unmoderated sessions tend to work best when they are short and focused. For example, 20 minutes rather than the usual 45 or 60 minutes for a moderated user testing session. If you want to cover lots of different tasks and questions, it can be better to set these up as multiple sessions, rather than trying to cram them into one mammoth session.
3. Screen participants
Carrying out user testing with people who are not representative of your users, or worse still are timewasters who have signed up purely to earn some quick money is a complete waste of time (hence the term). This is why it’s so important to screen your unmoderated user testing participants and to exclude any timewasters from future sessions. Most tools allow for screening questions to be set-up, or you can put a two- step process in place. For example:
- Run a screening questionnaire to gather the demographic and behavioural information you need to know about users.
- Only invite those people who meet your criteria to carry out unmoderated user testing sessions.
4. Make it clear that sessions will be recorded
Whilst it might seem obvious that a user’s screen and their video or audio will need to be recorded for unmoderated user testing to work, you’ be surprised at the number of participants who don’t realise this.
It’s important to make it very clear prior to sessions, and then at the start of sessions that in order to take part participants have to agree to their screen and their audio or video being recorded. Give users the option to switch off their video if they want to and remind them that they should ensure that no personal or sensitive information should be present on their screen during the session.
5. Ensure that instructions and tasks are very clear
If a user doesn’t understand an instruction or misinterprets a task during a moderated user testing session it’s not the end of the world because they can always ask the moderator for clarification. However, if this happens during an unmoderated session it can be catastrophic because there is no one there to get them back on track.
It’s therefore important that instructions are very clear, are written in plain English and that tasks only have a single goal for users to achieve. For example, a task asking users to find a particular product on an ecommerce website and then to purchase it, would be better as 2 tasks. It’s also a good idea to have a number of different people review instructions and tasks prior to running sessions. What might be clear to you, might not be to someone else.
6. Provide feedback for non-functional prototypes
In a moderated user testing session the moderator is able to intervene if a prototype isn’t fully working. This isn’t the case for unmoderated usability sessions which can all too easily get derailed by buttons that are not working, links that don’t take the user anywhere and non-functional features. If a feature isn’t working in a prototype (or the interaction is very limited) it’s a good idea to provide feedback for users so that they know what would happen. As shown in the example below, you can even ask users some questions about what they would expect to happen.
7. Include a survey
Because unmoderated user testing allows for lots of sessions to be run with users, by including a survey it’s possible to get a good mixture of qualitative and quantitative insights.
You can use a survey (or surveys, most tools allow questions to be asked following a task) to capture information about users, the ease of completing tasks, and the overall usability.
8. Run a pilot
You don’t want to launch your unmoderated user testing only to find that the results you get back are of little value because the tasks are unclear, participants have mis-interrupted what you’ve asked them, or there is a critical issue with the prototype.
Whilst running a pilot session is a sensible step for moderated user testing, it’s a critical step for unmoderated user testing. It’s important to trial sessions before running with your users, otherwise you risk wasting their time, and your own.
9. Run batches of sessions
Unmoderated user testing can often be a little bit hit and miss. Sometimes you’ll get the sort of insights you want, other times it will highlight that perhaps unmoderated user testing wasn’t the best approach after all. This is why it’s always a good idea to run batches of sessions. You can run an initial set of sessions, find out if you’re getting the sort of insights you need and then make a call as to whether you run more. This also provides the opportunity to tweak instructions and tasks between sessions.
10. Run in combination with moderated user testing
It’s important to view unmoderated user testing as a compliment to moderated user testing, rather than an alternative. Rather than one or the other, it’s often best to run one and then the other.
Unmoderated user testing allows for lots of sessions to be quickly carried out, which can then inform follow up moderated user testing session. For example, perhaps the unmoderated sessions highlight some assumptions that need to be explored with users, or flag some pain points which can be better understood via 2-way conversations during moderated sessions. By running unmoderated user testing in combination with moderated user testing you can ensure that you gather both quantitative and qualitative insights about your users.
See also
- A guide to running unmoderated usability tests (Dovetail)
- Unmoderated User Tests: How and Why to Do Them (Nielsen Norman Group)
- Using unmoderated usability testing (The GitLab Handbook)
Image credits
Photo by Christin Hume on Unsplash
