Since launching the Askable platform, we’ve helped over 100+ companies recruit people to help them test their products. From super broad to super niche we’ve worked with almost every kind of demographic you can think of.
We’ve found that one of the key aspects of getting the right user testers and focus group participants to your door is making sure the questions on the screener are bulletproof. Sadly there’s not many resources out there for people working in UX or UA roles and we’ve definitely learned some hard lessons along the way. If you screw it up, you’ll end up wasting a huge amount of time and money interviewing people that don’t really match your criteria.
In this article I’m going to cover some of the common mistakes that we’ve seen first hand and how you and your team can avoid them. That way you can be confident that the people who are showing up at your usability tests are going to be the right people and ultimately give you the most actionable insights.
Mistake #1 – Ambiguous Wording
Wording. If you’ve ever read a legal document and wondered why it sounded like complete gibberish, that’s because in law, wording is everything. It can mean difference between winning your case or having it thrown out by the court completely. Legal documents have sentences crafted in a way that every possible variation or combination is thoroughly covered off, minimising room for misinterpretation.
And thats great… if you’re a lawyer.
The problem is that for most people they’re super hard to read.
Your job as a UX designer looking to interview the right person is to make sure that your screening questions walk the fine line between being easy to understand for the average person, whilst also being thorough and clear enough that you don’t leave room for misinterpretation.
Let’s look at some examples:
Say for example you’re looking to interview small business owners for a financial app you’re building. You could create a screening question like:
“Are you a business owner?”
From experience, we know this delivers really varied results. You’ll end up with some people who are a sole trader and run a one person operation, and others who own medium sized businesses with 50+ employees. This is because “business owner” is a really broad term and covers a whole range of demographics. Unless your app is targeted towards any business of any type, this question is probably too broad.
“Are you a small business owner?”
That’s looking better, with this you should get a more specific group of people — owners of small businesses. But you can be even more specific to prevent further misinterpretation.
“Are you a small business owner (5–10 employees)?”
Much better! Now you can be certain that the people who make it through your screening questions are going to be exactly the right target demographic.
The main point is to remember that you need to eliminate ambiguity where possible, whilst still keeping your questions easy to understand.
Mistake #2 Questions that are Too Long
Most participants who are signing up to attend your user interview will probably be on their mobile phones, especially if they have seen it posted on social media or received an SMS. Having really long, wordy questions can increase dropoff rate significantly. Long questions are intimidating and can mean losing otherwise great participants who just gave up on finishing your screening questionnaire.
Where possible, split your questions up or re-write them to get rid of as many unnecessary words as possible. Try and keep questions short and to the point:
“Given your current household circumstance, please select the option that you believe best describes the frequency at which you prepare meals during a typical week.”
“How often do you cook at home?”
Mistake #3 Too Many Questions
Similar to the previous point, giving participants too many questions to answer can cause a significant increase in applicants that leave. Typically we’ve found anything under 15 questions is not a problem, but keep in mind also that the incentive on offer also affects a participant’s willingness to answer large numbers of questions.
Interestingly, we’ve also found that certain brands attract participants who are more willing to answer lots of questions up front because of their loyalty to the brand. But on the whole, try not to go over 15 questions in your screening questionnaires. Sometimes it may be necessary to just screen for the critical requirements, and then later collect further categorical information from your participants in person.
Mistake #4 Compound Questions
A compound question is where you’re asking two different things in one question. This makes it hard to work out if the answers or options presented (assuming you’re making multiple choice questions) relates to the first or second part of your question. It also makes it difficult for participants to answer correctly if they have to mentally keep track of the different cases you’re presenting. In most cases you can probably eliminate one or two of the extra questions but otherwise, just split them out into separate individual questions.
Have you made an insurance claim in the last 12 months, called regarding a claim or requested more information using an existing claim number?
This makes it really hard to work out what exactly it is you are asking. Are you asking if the participant has made an insurance claim in the last 12 months? Or are you asking if they have just made an enquiry? Or are you asking about the method they used to make that enquiry?
Mistake #5 Creating Unnecessary Arbitrary Filters
Whilst you want to make sure you write questions that will give you the exact demographic that you want, it can be easy to overdo it. It might be that you just got in the zone writing screening questions but forgot to think about whether or not each question is actually important, or if it’s relevant to the demographic that you’re looking for.
Mistake #6 Question Order
Few things are more frustrating to a potential participant than filling out a 20 part screening questionnaire only to realise on the last question that they are obviously ineligible. This is particularly true if you target demographic which requires your participants to get past a really unique screening question. E.g. you’re looking for people with a specific disability or people who own a specific product.
It’s always better to move those questions up to the front. The participants answering your screener will appreciate it. We also recommend putting super specific requirements like this in your description or your initial messages. It may seem counter intuitive but we’ve found that being up front with who you’re looking for can dramatically increase application completion rates.
Mistake #7 Too Many Options
If you’re creating multiple choice questions, it can be hard to know exactly how many options to provide as answers. Sadly there isn’t really a magic number or rule of thumb, since the number of options you should provide depend on the question. Generally speaking though, try and condense where possible, unless you absolutely need to filter really specifically.
Take participant age for example. You could create a question asking for age and include as your answers every single possible age (18, 19, 20, 21, 22 etc). But unless you are screening for people who are a specific age, in most cases you’d be better off creating the answers as a range (18–25, 26–35, 36–45 etc).
Mistake #8 Options that are Vague
In the same vein as mistake #1, creating multiple choice answers that are vague can easily give you the wrong participants. Be careful of using words that describe duration or timeframes. Words like “recently, soon, frequently” can mean different things to different people. It’s always better to provide an example of what you mean. Using “Recently (in the last week)” will give you much better results than just saying “Recently”.
Mistake #9 No Red Herrings
It’s important when creating questions that you throw in a few curveballs. Not because you’re trying to be nasty, but as an added safety net to disqualify participants who aren’t eligible.
And while most people aren’t trying to be dishonest, some participants may approach the questions through a filter of their own or glance over details in their eagerness to get accepted.
Lets take a look at an example:
Say you’re looking to do some user testing with people who follow major league baseball.
Here’s the question with no red herring:
Do you follow Major League Baseball?
Yes / No
And then with red herrings
Which of the following sports do you follow?
None of the above
You can see in the second example, it’s much harder to see at a glance which answer is the ‘correct’ answer.
Mistake #10 Using Negatives in Questions
Questions with negatives are almost guaranteed to be misunderstood. At best they’re much harder to read, and at worst you’ll end up wasting time interviewing completely the wrong type of person. This is easy to avoid, look out for any questions that use the word ‘not’ — For example:
“Would you agree or disagree that you do not consider yourself a technical person?”
“Would you consider yourself to be a technical person?”