User testing is often an afterthought, considered only after the design and development of an app. Our team at Savvy Apps believes in evaluating and validating your app from the outset and throughout the creation process. This allows us to iterate on what we learn and make sure your app appeals to users and meets their expectations before its launch. Without user testing, there's no way to know if you're on the right track until it's too late.
As there's no one-size-fits-all approach to user testing, we created this guide to teach you the fundamentals and strategies that will help you create your own testing plan specific to your business needs. We examine everything from defining your user goals, choosing between remote and in-person testing, recruiting your participants, preparing your testing environment and materials, to determining the type of data to collect.
Regardless of whether you use this resource to create your own in-depth user testing approach or contact us for help, by the time you finish reading you'll understand each of the core user testing steps, the best tools to use, and the special considerations you need to get started.
Define User Testing Goals For Your App
The first step in user testing is deciding exactly what it is you want to get out of your testing. Your goals determine everything from your testing methods to the exact environment, tasks, and questions you ask your participants. This impact makes it essential to determine your goals before going any further.
Your user testing goals will change depending on where your app is in its lifecycle. For example, if you plan on conducting user testing before your app's development phase, your goals will be more focused on exploration and discovery. That is, finding out what your target users need, what similar apps/products they already use, and what pain points they need help with. During this phase you won't have a functioning version of your app to test with, but you don't need one to achieve your user testing goals. Interviews with users, focus group discussions, and competitive analysis work well to get at the heart of your target users' wants and needs.
User testing goals during development tend to lean more towards validation and evaluating. That is, finding out if users understand and enjoy your app, and whether the features meet their needs. For this, leverage early versions of your app, prototypes, and mockups. We prefer card sorting and concept testing to zero in on how your users understand and interact with your app's hierarchy, structure, and feature set.
Testing doesn't stop once your app is live in app stores. It's important to continue testing to make sure you didn't miss any opportunities with your users and to keep your app aligned with evolving user expectations. Your goals in the post-launch phase will focus on measurement and further validation. That is, how your app is performing, opportunities for optimization, and whether your user experience appeals to users. For this, give your participants your actual app and try A/B testing to narrow in on what resonates best with your users.
Once you establish your user testing goals, you need to go a step further and determine clear objectives for individual tests. Try to hone in on the most important insights you want to gather rather than trying to test everything at once. Remember, you're not looking for generic feedback, you're looking to uncover specific problems. When you try to discover too much at once, you run the risk of conducting a long and unwieldy test that fatigues your testers. It's also easier to confuse your results and next steps when you try to tackle too much.
Your main goals and specific test objectives come together to form the foundation of your user testing strategy. For example, consider how those elements impacted our user testing for Share Our Strength's healthy eating app, Cooking Matters.
Since we conducted our user test later on in the development of Cooking Matters, we used a stable, pre-launch version of the app. Our main goals focused on evaluating the intuitiveness of the app's navigation and core features. We wanted to observe users' natural interactions with Cooking Matters to identify any red flags before launch. To make sure we established clear objectives for this test, we outlined a few key questions to answer at the end:
- Is the navigation intuitive?
- Can users easily find recipes?
- Can users easily add ingredients to their shopping list?
- Can users schedule and add meals to their meal plan?
By answering these questions and sticking to our main goal throughout the testing process, we discovered a few easy-to-fix issues that we likely would not have noticed before launch.
Choose Between In-Person and Remote Testing
User testing can occur either in person or remotely. For both types of testing, you will want to observe participants using your product and uncover any usability issues. Where in-person and remote testing differ is in their pros and cons.
With in-person testing, you get the opportunity to establish a real connection with your participants and see their reactions from up close. It's much easier to read someone's body language in person than it is over a video conference. In-person testing also makes it easier to use a participant's body language cues to gauge when to ask questions for clarification or dig for more feedback. These in-person interactions come with a cost, however. They often take more time because they involve traveling to a location, setting up a testing environment, and being present for each participant session. You may also incur a cost if you need to rent physical space for the testing.
Remote testing works well for unmoderated sessions (discussed below), as well as when budgets are more limited. This type of testing is also ideal for tighter timeframes as there's no need for travel or testing facilities, and because there is less pressure on test moderators. The major downsides, however, are that you can't interpret body language and it is more difficult to follow up or do a deeper dive into your participants' feedback.
Decide Between Moderated and Unmoderated Remote Testing
Say you decide to go the remote testing route. Then you have one more big decision to make — whether you want to conduct moderated or unmoderated sessions.
Moderated sessions often offer more insights because you have the ability to ask questions in real time. If a user gets stuck on a particular task, you can redirect them or get more information about what may be causing confusion. You also have the option to ask more questions as they arise. For example, someone might say “I wasn't expecting that” and leave it at that. In a moderated session, you could follow up by saying “What were you expecting?” or “Why were you expecting that instead of what you saw?” to further investigate the user's experience.
Moderated remote testing comes with a few downsides. Like in-person testing, there's a higher burden on time and effort as you need to schedule and be present for each session. You also run the risk of receiving less than candid results. Some participants may feel more pressure with someone observing and asking them questions than if they were interacting with your testing materials on their own.
While you don't have to worry about scheduling, moderating, and pressuring participants with unmoderated, remote testing, you lose the ability to ask follow-up questions and redirect participants as needed. You also run the risk of not being there to troubleshoot tech issues or explain tasks to confused participants. These scenarios could result in testing sessions that provide less meaningful insights.
Unmoderated sessions do, however, give you the added benefit of being able to test your app in a unique context or environment. This is important for apps used under specific circumstances. For example, we asked our test participants to use the on-demand ride service, Sprynt, in the most natural context — when requesting a ride. It would have been impossible (or at least, very awkward) to try to simulate similar circumstances with a remote moderator.
Select and Recruit Your Participants
If you recruit the wrong participants, or if you don’t get enough participants, your user testing will not generate the results you need. To find the right participants for your user test, you first need to identify your app's target audience. We often dive into recruiting participants during the discovery and user research process.
Behavior Versus Demographics
It's more important to recruit participants based on their behavior rather than demographics. In choosing your participants, select for behavior more than age, sex, income, location, education, or marital status. The best way to do so is by finding people who are already active in similar apps. If you don't have clear competitors, look for people who do activities similar to your app's feature set. These participants are more like your target user base in that they likely have some prior experience and interest in what your app is trying to achieve. This is often more helpful than a participant who falls into a specific gender or age group.
That doesn't mean you should dismiss demographics altogether. Your user research should indicate the common characteristics of your target users. Selecting participants who match those demographics will help you arrive at more useful insights. How picky you should be in matching those characteristics, though, depends on your target users and testing goals. You will want participants who match a very specific demographic if you have a niche product or user base. This is also important for complicated apps, apps that need a certain level of prior knowledge or experience, or apps with direct competitors. Demographics aren't as important if your app appeals to a wide variety of users. This is also true for apps that anyone can use with minimal instructions or experience, and apps that you can test in 15 minutes or less.
There are several other considerations to keep in mind when selecting the right participants for your user testing. Before reaching out, be sure to determine how technically proficient your participants should be and whether they need access to a specific platform (Android, iOS, or web). Also stay away from recruiting people you know or contacts from social media to keep your user testing results as unbiased and accurate as possible.
How Many Participants Do You Need?
The number of participants you need for your test also depends on your testing goals. You should recruit 5-7 participants to test usability problems. While the Nielsen Norman Group asserts that you only need five participants to uncover 85% of a product's issues, we prefer to schedule seven participants per test. The two extra participants account for no-shows.
Tests that investigate trends and opinions require a larger sample size to gather enough quantitative findings. We segment those users into groups of 5-7 participants based on different characteristics. You might separate them based on their knowledge related to the topic of your product or experience with your existing product. For example, when testing trends for BoatUS, we segmented a large pool of test participants into two groups based on their experience with the existing app. One group of participants had no prior experience with the BoatUS app, while the other had used the app before. We then compared results to see if that prior experience made a difference in how the participants used the app and how it made them feel.
Once you identify the 5-7 (or more) participants you want based on their behaviors and demographics, you need to get them on board. For in-person testing, we generally pull from our existing users. If you don't have existing users, you can lean on your network. We've found particular success by tapping into community organizations and professional associations interested in the problems our apps try to solve. To help entice candidates, we provide compensation like gift cards, but the incentive will depend on what appeals to potential recruits. We often use an incentive that's valued at $15 up to $50. That's up to your budget and how vital feedback is to you. We also try to be flexible about days, times, and the lengths of testing sessions to cater to our participants' busy schedules.
Recruiting With Tools Like User Interviews
There are several effective and affordable services that can help you recruit users to participate in remote testing. We prefer User Interviews. This tool allows us to target specific demographics and pose questions to narrow down the pool of potentials. Each candidate provides a profile and completes a survey for you to review before deciding whether they make a good fit. You provide a description of your project, any requirements, and any compensation. Once you invite a candidate, it's then up to you to provide more details and either run the testing session or provide a link to the remote test.
We leverage services like User Interviews when we do not have an existing pool of users to recruit from for testing. For example, we used User Interviews to set the demographics for ideal Jellies users, provided a screener survey to weed out candidates, and then reviewed the remaining potentials to find our user testing participants.
Recruiting participants takes 3-5 days. This is an acceptable amount of time, especially for the tool's less-expensive price point. User Interviews costs $20 per participant, plus any incentive you want to provide. It can take a lot of time and resources to recruit testing participants on your own, and it's expensive to use a firm that specializes in recruiting.
Still, there are downsides to using User Interviews. As with any outreach, you're at the mercy of the information you're provided. With User Interviews you have to rely on your survey to make sure the candidates match up with your target pool's behaviors and demographics. You don't speak with the participants before the sessions, so you have to trust that they were truthful in their responses.
Schedule Your User Testing Sessions
Scheduling every user session for every test can become a logistical nightmare depending on the number of tests you decide to run. We simplify the process by using a tool called Calendly to eliminate the hassle of scheduling each individual testing session.
Calendly eliminates the confusion and hassle of trying to nail down session dates and times for each participant. It prompts each candidate to select one of your available time slots. When one participant selects a certain date and time, Calendly removes that option for everyone else. This saves you time and headaches by removing the need to reschedule double-booked sessions. We find it especially helpful when dealing with large numbers of testers for moderated remote testing. If you're conducting unmoderated testing via a tool like Lookback (more on that later), your participants need only run through the tasks at a time convenient to them.
The only downside we've encountered is that it can be confusing to complete the initial setup for your user testing events. To help you through this, here's a quick step-through of how we approach the setup for our user testing events:
- First we sync Calendly with Google Calendar so we can get an overview of when we're available for sessions.
- Then we create a specific “User Testing” event type within the Calendly tool.
- We select the duration for each session and date range for the entire testing event. Generally our testing events last about a week. We then assign each session in 60-minute blocks within a normal work-hour timeframe, like between 9 a.m. and 5 p.m. depending on the participant's time zone.
- We adjust further so that Calendly only displays the times when our moderators are actually available.
- We set our availability increments to 15 minutes (found under the advanced tab) and the event buffers to 15 minutes before and after each session. This provides some time between sessions in case something goes wrong or a session runs long.
Once we set up the Calendly event, we send it out via a link in an email to our participants. As they select their session times, they are automatically added to our Google Calendar. We provide directions to participants on the morning of each session.
If you prefer to handle scheduling without a tool, remember to provide enough of a buffer between testing sessions and not overbook yourself. You will juggle a lot of variables during your testing, so you'll need extra time to gather your thoughts, fix a testing snafu, handle a tardy tester, or deal with some other delay. Also consider how exhausting moderating testing sessions can be. You don't want to be sluggish, especially in your afternoon sessions, so give yourself time for breaks and lunch so you're ready to go.
Prepare Your User Testing Materials and Script
Preparing for your user testing sessions is usually done the same way for in-person and remote testing. First, you need to write down the tasks you want your participants to attempt during the testing session. Then create a script to follow throughout the test. The only extra step you might take for in-person testing is printing your tasks out on physical cards. For people that process information better visually, we find it's helpful to have those tasks printed out so you can hand them over one at a time.
Create your testing tasks, materials, and script based on your user testing goals. You can insert metrics questions to gather more insights. For example, record how long it takes for a user to complete a task. That's one measurable data point you can use to hone in on issues with your user experience or navigation. For gauging the difficulty of a task, ask the participant whether the task was easy to complete on a Likert scale (very difficult, difficult, neutral, easy, very easy).
Testing works best when you expose your participants to the same variables. Changing the tasks you want participants to tackle, or the way you explain the tasks and the test as a whole, will skew your results. That means you need to script everything you’re going to say to the user, and you have to follow the script to the letter.
Writing the Script
When writing your script, make sure it addresses your research questions. You do this by translating your research questions into tasks that represent realistic user goals. For each task, set up a user story to make it more realistic for the participant. For example, for a BoatUS test we didn't tell participants to accomplish a certain goal within the app. Instead we told them that the motor on their boat had just stopped working, stranding them a few miles out to sea. We then asked them, “How would you call BoatUS to get back to the marina?”
Your script should also include any interview questions you would like answered, prompts for follow-up questions, and debriefing questions. Be sure to include clear and detailed instructions on how to get to the app. Instead of telling participants to install your app from the App Store, write out every step of the process. This might look something like “Go to the App Store, search for [Exact Name of App], and install it. Once you download the app, please DO NOT open it yet. Proceed to the first task.” If sharing links, it’s best to shorten their length using bit.ly, tinyurl, or another tool.
Don't leave anything up for interpretation. Tell your participants whether their device should be in landscape or portrait orientation with the sound on or off, and other specifications. Note whether they should follow along with an on-screen tutorial or onboarding experience. At the same time, you don't want your script to lead participants. Provide participants with clear directions, a task to complete, and then give them space to interact naturally with your app to get the best results.
Set Up Your User Testing Environment
Your testing environment and tools depend on your testing method. In-person testing gives you full control over your testing environment, while your participant's remote testing experience relies more on the quality of the tools you choose.
The most important thing when selecting a place to conduct your in-person testing is finding an area that's free from distraction. Loud noises, unexpected visitors, and other distractions interrupt sessions and skew your test results. We use a quiet conference room with the door closed so our participants stay focused on their tasks, and our recordings don't pick up extra noises. Also try to limit the number of people in the room to a participant, a moderator, and a note-taker. Participants tend to get nervous when there are several people in the room watching them.
It's important to choose the right tools and familiarize yourself with them before you start your first session. When selecting the right tools for your test, save yourself a headache and prioritize the ones that are easiest to use. Testing tools should be simple to set up and manage during your sessions. The tool needs to be easy to use not just for you, but for the participant. This is especially important if you are running unmoderated remote testing. In that type of test, the participant won't have you to help them if they run into any issues getting the session started.
Your tools also need to be reliable. The last thing you want to deal with is a technical issue when you're trying to run a user test. It can interrupt the flow of the session and cause frustration for you and the participant. Good sound and video quality are just as critical. This is especially true for moderated remote testing, because you're relying on the video chat to observe the participant's body language and tone.
Besides using User Interviews and Calendly for the preparation stages, we choose between AirServer, Vysor, and Lookback for the actual test. The benefits of these tools depend on the type of test you're conducting.
Test Environment For In-person User Testing
We use AirServer and Vysor to cast the participant's iOS and Android app activity to a laptop, respectively. This allows us to view and record the app as it is being used by the participant. In the past, we needed a complicated setup to record the participant's device. By using a tool to cast to your laptop, the participant can hold their device naturally, rather than leave it flat on a table underneath an overhead camera.
Both tools are inexpensive to use. AirServer provides a 14-day free trial. Their consumer license is $19.99 and business license is $39.99. Vysor offers a free version to mirror, control, and take screenshots from an Android device. The pro version includes more features for $2.50 a month, $10 a year, or $40 for a lifetime license.
Note that Vysor is slightly less reliable than AirServer when it comes to consistent mirroring. You may find it disconnects from time to time, and you'll need to ensure that you've “Allowed USB debugging” when prompted on your device. Often it takes disconnecting and reconnecting the USB a couple of times for it to work.
We introduce a third recording tool, Screenflow ($129), to capture the participant's reactions as they interact with your app. Screenflow allows us to not only record the activity on your app (via AirServe and Vysor), but audio and video of the participant during the session. Because we're recording all of these inputs at once, we are creating a useful, concise record to refer to after testing.
After the sessions we edit and export the Screenflow file to an .mp4 or .mov file. You can easily edit the recording in Screenflow to create a highlight reel for any stakeholders that may want to watch the testing. Keep in mind that the files tend to get pretty large, especially if your sessions run an hour or longer. It helps to save the recordings on an external hard drive so that you don't run the risk of running out of disk space. After your sessions are complete, you can transfer the recordings to your Dropbox if need be. You won't have time to do this between sessions.
Test Environment For Remote User Testing
We rely on Lookback.io for recording moderated and unmoderated remote user testing sessions. Not only is Lookback easy to set up and use with reliable audio and video, it provides the most convenient way to record both the participant and their device. There isn't another tool as robust as Lookback on the market right now.
Not only do you get a solid connection between moderator and participant, Lookback enables you to include observers in each session. Any stakeholders who want to watch the sessions live can do so, as well as communicate with you via a chat tool. This comes in handy for including questions the observers might want asked on the fly. Your participants don't see the observers, so they don't create any added stress during the session.
Lookback also streamlines the user testing process by providing each participant with a separate interview room, so you could have multiple sessions running concurrently. After your sessions, you can watch the recordings back and add notes to the recordings or create highlight clips to export.
There are a few downsides to keep in mind when using Lookback for your user testing sessions. We've run into problems, particularly with older Windows computers, of participants not being able to hear us via Lookback. To get around this we use a conference line, such as Uber Conference, as a fallback audio solution. The participant mutes their microphone within Uber Conference to prevent an echo, but we leave our microphone on. This allows the participant to hear us through Uber Conference while Lookback continues to record their audio feed.
Another downside is that Lookback doesn't give you the option to ask participants to complete specific tasks in remote unmoderated sessions. To get around this, you need to communicate your tasks with the participant before their session. We do so via an email or Lookback's project "Welcome Message." Lookback provides some helpful ideas on the best ways to accomplish this.
Another tool to consider for your remote testing is UserTesting. This pricier alternative to Lookback is best for unmoderated sessions. It allows you to choose from their large pool of test candidates or use your own for an added fee on their Pro plan. While Lookback costs $59 a month or $49 a month if billed annually, the price of a UserTesting enterprise contract can run into the $1,000s depending on the size and needs of your company. UserTesting also offers an individual plan that provides for up to 15 video sessions (a cap you'll quickly run up against when conducting quality user testing). If you prefer to use the same tool to recruit and test participants, and have the budget for it, UserTesting might be a better choice for your business.
Test Your Test
Make sure you do a dry run before your first real session. Doing so will give you the opportunity to fix any red flags or issues that come up. We find running a pilot test with a co-worker helps iron out any kinks in the test script and tasks. It gives you a chance to evaluate the flow of the test, add or remove any tasks, and predict any potential road blocks. You'll feel much more prepared and confident for your sessions if you've already remedied possible issues. Think of it like usability testing your usability test!
To maximize the effectiveness of your test, try to do this at least a day or two before your sessions so that you're not scrambling right before participants arrive. Also give yourself time to prepare your device and whatever you’re testing (app, prototype, sketches, etc.) so everything is ready to go.
Conduct Your User Test and Debrief Your Participants
It's finally time! If you're conducting unmoderated testing, there's not much for you to do as your participants should have already received the materials, tasks, and directions. For those of you conducting moderated testing, here are a few more items to consider.
Start your testing sessions by welcoming the participants and explaining their role (using your test script!). Make sure you note that you're testing the app and not them. This takes the pressure off your participants and may help them relax. You'll also need to remind them to think out loud throughout the session. Participants often forget to verbally express their thoughts and may resort to sighing or looking confused. Encouraging them to think out loud will help you gain those valuable insights you're after.
Only one person should be moderating each session, but you can alternate moderators between sessions if necessary. Be aware that switching moderators mid-session can throw off the flow and make participants uncomfortable. As moderator, you start establishing a connection with the participant from the beginning of the session. Bringing in another moderator undoes that progress. This connection between moderator and participant comes in handy when probing for more information or asking participants to clarify their thoughts.
After each session, ask the participant follow-up questions and for any more insights or feedback they might want to offer up. If multiple team members are observing the sessions, chat briefly with the group after each session to see if anyone has anything they would like noted. Make sure to thank each participant, and let them know how valuable their feedback was. It's possible that you may be able to test with them again down the road.
Analyze and Act On Your User Testing Results
Now it's time to watch any recordings and take note of what you see and hear. Look at all of your observations together to try to determine why particular things happened and develop theories about the causes of any friction, frustration, and problems.
Besides direct feedback from your participants, examine measurable data points to arrive at conclusions about your app. Mentioned earlier, an easy way to do this is by examining how long particular tasks took to complete. The time a participant spends on a task can indicate if something in the app is particularly difficult or confusing. It can also reveal if something was fun, engaging, and easy.
If you conduct multiple individual sessions, ask your team members to watch a few sessions each and compare notes. Team members may interpret your participants' statements differently. Hearing everyone's thoughts will help you eliminate bias and arrive at more accurate results.
Once you observe, record, and discuss your findings with your team, start creating an action plan full of recommendations for improving your app experience. We like to put together an easy-to-digest report that summarizes our findings and any trends we noticed throughout the testing process. This report helps us zero in on what changes we need to make and whether we need to revisit the app's branding, design, feature set, or user research to better resonate with users and achieve your business goals. If our user testing shows less than favorable results, we make changes and test again.
Done well, user testing makes sure your app aligns with what your users expect, need, and want. Without it, there's no way of gauging whether you're on the right track throughout the design and development process. We hope this guide gives you a good foundation from which you can build your own successful user testing strategy. You can learn more about our user testing services and contact us for help on the Savvy Apps website.
Join 20,000+ Other Readers
Sign up to be notified of new blog posts and be the first to receive helpful app goodies from Savvy Apps!