The Definitive Guide to App User Testing

When it comes mobile app development when of the most overlooked aspects of the process is user testing, in fact this is often an afterthought and is only considered after the design and development of an app.

by Robbie Allen, xDesign

Fear not, xDesign a leading mobile app development agency located in Edinburgh, UK are on hand with our definitive guide to App user testing.

App User testing allows us to iterate on what we learn and make sure each app appeals to users and meets their expectations before its launch. Without user testing, there’s no way to know if you’re on the right track until it’s too late…

Much like app development there is no one size fits all approach to user testing.

This Definitive Guide to App User Testing has been created using fundamentals and strategies that will help you create your own user testing plan specific to your business case.

Regardless of whether you use this user resource to create your own in-depth user testing or you reach out to an app development company like us by the time you have finished reading this article we hope you understand each of the core user testing steps, the best tools for the job, and the special considerations you need to make from time to time.

Creating user testing goals

The first step when it comes to app user testing is setting clearly defined goals. The goals you set will define everything from the testing methods you implement to the environment you hold the testing through to the tasks and questions you ask participants.

Before you go any further take some time now to set out what your goals are.

Depending on what stage of development your app user testing goals will change. Let’s look at a couple of examples;

Pre Development User Testing Goals

If you plan on conducting user testing before your app’s development phase, your goals will be more focused on functional requirements and discovery. That is, finding out what your target users need, what similar apps/products they already use, and what pain points they need help with

During Development User Testing Goals

User testing goals during development tend to lean more towards validation and evaluating. That is, finding out if users understand and enjoy your app, and whether the features meet their needs. For this, leverage early versions of your app, prototypes, and mockups.

Post Launch User Testing Goals

our goals in the post-launch phase will focus on measurement and further validation. That is, how your app is performing, opportunities for optimization, and whether your user experience appeals to users. For this, give your participants your actual app and try A/B testing to narrow in on what resonates best with your users

Setting Objectives for each User Test

Once you establish your user testing goals, you need to go a step further and determine clear objectives for individual tests.

Try to hone in on the most important insights you want to gather rather than trying to test everything at once. Remember, you’re not looking for generic feedback, you’re looking to uncover specific problems.

Don’t Do it All

When you try to discover too much at once, you run the risk of conducting a long and unwieldy test that fatigues your testers. It’s also easier to confuse your results and next steps when you try to tackle too much.

In Person vs Remote User Testing for your App

User testing can occur either in person or remotely. For both types of testing, you will want to observe participants using your product and uncover any usability issues. Where in-person and remote testing differ is in their pros and cons.

In Person Pros

• With in-person testing, you get the opportunity to establish a real connection with your participants and see their reactions from up close.
• It’s much easier to read someone’s body language in person than it is over a video conference.
• In-person testing also makes it easier to use a participant’s body language cues to gauge when to ask questions for clarification or dig for more feedback.

In Person Cons

• These in-person interactions come with a cost, however.
• They often take more time because they involve traveling to a location, setting up a testing environment, and being present for each participant session.
• You may also incur a cost if you need to rent physical space for the testing.

Remote Testing Pros

• Remote testing works well for unmoderated sessions as well as when budgets are more limited.
• This type of testing is also ideal for tighter timeframes as there’s no need for travel or testing facilities, and because there is less pressure on test moderators.

Remote Testing Cons

• The major downsides, however, are that you can’t interpret body language.
• it is more difficult to follow up or do a deeper dive into your participants’ feedback.

Selecting the RIGHT Recruits

If you recruit the wrong participants, or if you don’t get enough participants, your user testing will not generate the results you need. To find the right participants for your user test, you first need to identify your app’s target audience. We often dive into recruiting participants during the discovery and user research process here at xDesign.

Behaviour over Demographics

It’s more important to recruit participants based on their behavior rather than demographics. In choosing your participants, select for behavior more than age, sex, income, location, education, or marital status. The best way to do so is by finding people who are already active in similar apps.

This doesn’t mean you should dismiss demographics altogether.

How many User Testing Recruits do you need?

The number of participants you need for your test also depends on your testing goals. You should recruit 5-7 participants to test usability problems. While the Nielsen Norman Group asserts that you only need five participants to uncover 85% of a product’s issues, we prefer to schedule seven participants per test. The two extra participants account for no-shows, which do happen!

Test your User Test

Make sure you do a dry run before your first real session. Doing so will give you the opportunity to fix any red flags or issues that come up. We find running a pilot test with a co-worker helps iron out any kinks in the test script and tasks.

It gives you a chance to evaluate the flow of the test, add or remove any tasks, and predict any potential roadblocks. You’ll feel much more prepared and confident for your sessions if you’ve already remedied possible issues.

Think of it like usability testing your usability test!

Testing Time… Finally

• Start your testing sessions by welcoming the participants and explaining their role (using your test script!).
• Make sure you note that you’re testing the app and not them. This takes the pressure off your participants and may help them relax.
• You’ll also need to remind them to think out loud throughout the session. Participants often forget to verbally express their thoughts and may resort to sighing or looking confused. Encouraging them to think out loud will help you gain those valuable insights you’re after.
• Only one person should be moderating each session, but you can alternate moderators between sessions if necessary.
• After each session, ask the participant follow-up questions and for any more insights or feedback they might want to offer up.

What Next?

Now it’s time to watch any recordings and take note of what you see and hear. Look at all of your observations together to try to determine why particular things happened and develop theories about the causes of any friction, frustration, and problems.

That’s a Wrap

Done well, user testing makes sure your app aligns with what your users expect, need, and want. Without it, there’s no way of gauging whether you’re on the right track throughout the design and development process.

We hope this definitive guide to app user testing  gives you a good foundation from which you can build your own successful user testing sessions.