How to Verify Research Participants When Product Teams Own the Call

One of the most common pieces of advice I give product teams and designers running their own research is this: if you’re seven interviews into a generative study and no patterns are emerging within the same segment, the answer is not more interviews. Stop. A more sophisticated synthesis tool won’t fix it, and neither will asking AI to find the signal in your transcripts.

You may have asked the wrong questions, but more likely you asked the right questions of the wrong people. The problem started before you even said hello.

Participant quality is now a product decision

For most of my career, participant quality sat with researchers. They were the gatekeepers. They owned the screener, the panel, the inclusion criteria, and the moderation. When something looked off in a participant, they caught it before it became a “finding” because trained researchers know what to look for.

That’s changed. More product managers and designers are running their own studies now, often without a researcher in the loop, often using AI tools that promise speed and scale. But nobody told them that participant quality came with the job: who they talk to and what they ask is now their call, not a researcher’s.

The gatekeeping function didn’t move. It disappeared.

What ‘the right people’ really means

When researchers train, we learn that good research lives or dies on participant quality. Skip this, and the rest doesn’t matter. And the right people are rarely ‘anyone we could recruit fast,’ ‘five users from the panel,’ ‘customers,’ or ‘people who signed up.’

A trained researcher defines participants by behavior, attitudes, context, role, frequency, expertise, and relationship to the product. Not all are necessary for every study, but think about these dimensions carefully when matching the question set you’re exploring. Each one changes what someone can credibly tell you. A heavy user of X can answer questions a light user can’t. A customer who churned can answer questions that a current happy customer won’t. A prospect answers different questions than the person who bought.

Because each of those people answers a different question, you rarely want just one type in a study. What you’re usually after is how the answer changes as one variable moves, since the difference between a novice and an expert, or a new user and a veteran, is often the finding itself. So a strong sample deliberately spans the dimensions that produce those differences, and they go well beyond demographics: workflows, skill levels, device types, environments, geographies, tenure, and use frequency. The teams who get this right also recruit edge participants on purpose: heavy users, failed adopters, switchers, support-heavy customers. They reveal the system more clearly than the middle does.

If your sample isn’t defined this way, your learnings can’t be either.

The fraud you can’t see in the data

The discipline of making sure you have the right people does more than get you a relevant sample and builds confidence in the results. It’s also your best defense against the ones who were never real participants at all. Fraud is at the extreme end of the wrong-people problem.

A few cohorts ago, one of my Ask Like A Pro students caught it in real time. She was running a generative interview, sharing a dashboard prototype. Early in the session, she asked the participant what was working and what wasn’t. The participant described a feature in detail. The feature was on a screen the student hadn’t shown yet. Then a smaller tell. It also looked dark outside their window. They said they were in New York, where it was noon.

The participant was a fraud, part of a coordinated ring running screeners dozens of times to learn the “happy path,” using stolen LinkedIn identities and proxy IPs to mimic East Coast locations. We caught three of them across nine active studies in that cohort.

If a trained researcher hadn’t been watching closely, the data would have shipped. That is what makes fraud so dangerous. It doesn’t look like fraud in your results. It looks like clean, confident data. You can’t catch it in the analysis, and you shouldn’t have to catch it live.

For most of my career, keeping fraud out was a manual fight, screener by screener and session by session. We asked where each participant came from, whether they were real and identifiable, whether someone had screened them both in and out, or whether they were just self-reported volunteers from a list. The durable fix has always been the same: stop fraud at the door, before anyone reaches a session, by confirming that everyone in the study is a real, identifiable person.

What’s changed is that the technology has caught up. That work is now largely handled at the platform level, before a participant ever reaches you, rather than resting on whoever happens to be in the room. Askable, for instance, owns its participant network across 50+ countries and verifies every person in three layers: LinkedIn and identity checks, phone validation, VPN and proxy blocking, and AI fraud detection at sign-up; an AI-led onboarding interview before anyone can appear in a study; and a tech check at the start of every session. That is precisely the stack that stops the ring I described above, stolen LinkedIn identities and proxy IPs and all, before they ever reach your screen.

The difference between a verified participant and an unverified one is the difference between a confident finding and, well, pixie dust.

How to verify a research participant

Almost all of this is won before fieldwork starts. Here’s the sequence I use to make sure the people in a study are the right ones, and real ones. Skip a step, and the wrong person slips through.

1. Write your inclusion criteria like a product spec. Who counts as “our user” for this specific question set? Then write your exclusion criteria. Yes, exclusion criteria. Pro tip? Recruit fewer people if it means recruiting the right ones. And check what already exists before you recruit anyone. In my experience, the answer is often already in the building: prior studies, support tickets, sales calls, analytics. And definitely have a conversation with someone on your Customer Support team. They are gold mines. Most teams I work with don’t have a research problem. They have an amnesia problem.
2. Screen for behavior, not opinions. “Do you care about productivity?” almost always returns “yes.” Ask what someone has done, not what they believe. How many times in the last three months? Which activities did they personally handle? Screeners that ask people to demonstrate rather than describe filter out the people who can’t back up their claims.
3. Plant a bogus item. Ask “Which of these booking tools have you used?” and slip in one response option that doesn’t exist. If they select it, they’re not who they said they were, and the rest of their answers are suspect. It’s called a bogus item, and the technique for catching careless, low-effort respondents this way has been in the methods literature for decades. Meade and Craig gave it one of its best-known treatments in 2012, in Psychological Methods. In my experience, most teams running their own research have never heard of it. I use one in every single screener. If they select it, they’re out. No follow-up, no benefit of the doubt. One bogus item, one decision rule. (It’s fun coming up with the bogus names, too.)
4. Run a saturation check. This is how you find out, mid-study, whether the first four steps actually worked. As a rule of thumb, in generative work, the kind that explores what people think and do to shape something new, I start to worry when seven or eight interviews in one segment turn up nothing. In evaluative work, usability testing, concept testing, or checking a prototype against defined criteria, that number is closer to five or six within the same segment. When the patterns don’t come, don’t blame the data. Look upstream, at who you recruited and what you asked them.