With more companies today putting a stronger emphasis on gaining a
deeper understanding of their customer, it’s not unusual for us to be
called in for a project to find that our clients don’t have a lot of
experience with research and don’t know what to expect. This article is
for every designer, architect, manager, engineer, and stakeholder who
wants to know more about research and is intended to provide you with
the most critical tools for interacting with researchers and
understanding how the work that we do can make your job easier.
This
article will also outline what to expect from researchers and some ways
to recognize when you’re working with a good one. These are indicators,
not standards, based on what we’ve found to be effective. There are
many ways to do research and every research study is different so it
doesn’t mean that a researcher is incompetent if he or she doesn’t
conform to these indicators. One sign of a strong researcher is that he
or she will educate you throughout the process so that you know what to
expect. With that in mind this article is ultimately intended to
provide a useful starting point.
Recruiting
One
of the most critical and time-consuming elements of test preparation is
defining the right target audience and recruiting participants.
Participant recruiting is usually conducted by professional recruiters
who typically consult databases of potential participants. Sometimes
researchers will do the recruiting themselves, but it’s usually more
cost effective to use a specialist.
Recruiting will almost always take two weeks or more
depending on the number of participants and the type of research, so
make sure that you get started early enough for the recruiter to have
enough time to find the appropriate participants for the study.
Recruiting for phone interviews may take slightly less time and any
kind of home visit will likely take longer (ethnography or contextual
interview). Your researcher should be able to provide you with an
estimate at the time of initial engagement.
A week for
recruiting tends to be difficult and any less than that is pretty much
unthinkable. Short-changing the recruiting could result in participants
that don’t properly fit the target market segment, don’t provide
quality feedback, or just don’t show up at all. All of these can have a
negative impact on the data. Even if it is possible to get participants
faster, it’s usually better to take the time to ensure that you are
getting the right people. Your researcher should know all of this and
recruiting participants is where he or she will start after getting a
basic understanding of your product and schedule.
A
recruiter will need a screener to get started. A screener is a
description of the target user with open and close-ended questions
about the participant that will help the recruiter to select the right
people. What you can do to smooth the process along is to have a
prepared concept of your target user. This does not need to be a full
market research report—just an outline of the types of users that will
use your product.
Your researcher should dig deep with
questions that include more than demographic information by asking
behavioral questions. Behavioral questions can include such topics as
TV watching behavior, purchasing behavior, internet use, etc. Typically
behavioral questions will give you a stronger understanding of those
who are being recruited than demographics alone. These are important
elements of market segmentation that are sometimes organized into
profiles called personas.
Personas are useful because they
create a consistent concept of the intended market segment that can
guide the design process through multiple iterations. Personas can also
be adjusted following deeper discovery research, such as in-depth
interviews, as more information about the intended user comes to light.
Within a few days, the researcher should present a screener that
includes behavioral questions as well as demographics.
Scheduling
When creating a schedule for data collection, the researcher should know that you cannot run participants back to back.
It’s generally not feasible to squeeze in 8 one-hour sessions in a
single day, because of all of the activity that must occur between
sessions. In an eight hour day, a researcher can perform four (maybe
five) one-hour sessions but any more than that will take more time.
Here are the reasons why:
One-hour sessions rarely go exactly one hour, some are shorter
and quite a few will run longer. This can be due to a variety of
reasons such as the product malfunctioning, the participant arriving
late, or the participant providing lots of feedback. My rule of thumb
is to allocate 50% of the session length as a buffer between sessions
to allow for overrun, not including time needed to set up for the next
session.
For sessions at an office or lab, some participants
will arrive 10-20 minutes early, at which time they will need to use
the restroom, sign NDAs and consent forms, and generally get
comfortable. Comfortable participants give useful feedback, while
uncomfortable participants tend to clam up and provide short,
unemotional responses.
The researcher needs to set up and get
ready. For usability or experience testing, the test will need to be
reset, notes and documents need to be filed and new ones prepared. For
any kind of home or location visit, the researcher will need to pack up
all equipment and travel to the new location and set up equipment again.
Thus for every one-hour usability or experience testing session, there’s forty-five minutes to an hour of buffer and setup time. Home visits can take much longer.
Test Plan
A
test plan should take no more than a week to develop and the researcher
should give it to you for review and approval before being finalized.
The test plan should specify the research and business goals associated
with the project. During this period the researcher will need a significant amount of time with the product, either with a prototype or available concepts,
while writing and checking the test plan. The better the researcher
understands the intended final product, the more valuable the
information he or she can get from the participant.
For usability or experience testing, the researcher will test the
tasks with the product prior to a pilot test. He or she will need to
make sure that there are no glitches, no unexpected areas under
construction, and nothing giving away future tasks when performing each
of the tasks with the product. With that in mind, it’s important to
give the researcher a stable product or prototype and avoid drastic
changes to the product prior to the test.
You should receive
a well-written and organized test plan that details each research
question and how it will be addressed. For usability testing this will
include a list of tasks, what each task is intended to examine,
approximate wording for the task (avoiding leading language), and
detail on how each task will be scored or evaluated. For discovery
research, it will include a list of topics to be addressed such as
processes, environment and context, and expected pain points and needs.
Data Collection
When
the data collection starts, it’s important to let the moderator work.
During this time, the participant should feel comfortable enough to
open up and provide honest feedback. In order to do this, it’s
important to try to minimize observer impact during the testing session.
If
you don’t have a separate place to watch the session (e.g. behind a
two-way mirror or through a video feed), don’t make it obvious that you
are paying close attention. Think about bringing in a laptop during the
session to make it look like you’re doing other work. One way of doing
this is telling the participant that you are also a researcher but
you’re just going to be taking notes.
When you’re observing, remain objective and don’t make judgments based on one or two participants.
It’s not uncommon to see a couple participants have a completely
opposite reaction to a product compared to ten other participants. The
researcher’s job is to sort through all the noise and report the real
trends in the research. Take what you see with a grain of salt and
listen to your researcher.
At the same time, it’s important
to try to observe as many sessions as possible and give your researcher
feedback between sessions if there are certain aspects of the user
experience you want to know more about. The researcher should put the
participant at ease and extract a great deal of information, including
details that might have been overlooked or emotions that the person
experiences. Different researchers will tend to achieve this in
different ways as everyone has their own style, but you’ll notice by
paying attention to the participant and seeing if they feel relaxed or
nervous throughout testing.
Findings
Frequently,
stakeholders will want to make immediate changes to a design, product,
or prototype and won’t have the time to wait for the researcher’s final
report. People have schedules that need to be met so it’s
understandable that a project can’t always wait for the final report
but the researcher should be able to provide you with quick findings within 24 hours of the last session.
For
usability research, these quick findings should consist of a couple of
short paragraphs including problems in the interface, possible
solutions to these problems, and participants’ general reactions to the
product, its look and feel, and expected usage. For ethnography or
other forms of discovery research quick findings will tend to consist
of expected usage of the product, expected value, high and low value
features, and general trends about the intended user. Quick findings
aren’t comprehensive and come before the researcher can get a complete
look at the data, but it will provide you with the overall themes from
the study.
When you do get the final report, make sure you take a look at it. It will tell you two things:
- Detailed findings regarding the interface, product, features, and intended user
- The quality and clarity of the report will tell you quite a bit about the quality of your researcher.
There’s one other thing to keep in mind when you are processing the
findings from a usability test. The participants will tend to focus on
the more obvious problems with a product or interface. There could be
other, smaller or more abstract problems that are not identified in the
first pass of usability testing. It’s usually a good idea to perform
another test on the product after making changes to ensure that the
changes you made were effective and identify any additional issues.
Summary
In summary, here are the most important points for non-researchers to know about the research process:
- Recruiting will almost always take two weeks or more.
- For
every one-hour usability or experience testing session, there’s
forty-five minutes to an hour of buffer and setup time, home visits can
take much longer. - The researcher will need a significant
amount of time with the product (prototypes or concepts) while writing
and checking the test plan. - Try to minimize your impact during the testing session.
- Remain objective and don’t make judgments based on one or two participants.
- Ask your researcher to provide you with quick findings within 24 hours of the last session.
