AI-Moderated Studies: When They Work, When They Don't, and What Makes Them Different
Jack Laidley
•
•
We've been running AI-moderated interviews on Askable for a while now. 38,000+ of them, across industries, in 15+ languages. We didn't announce it with a splashy launch because we wanted to understand where it belongs and where it doesn't before telling anyone.
Now we know.
The questions that come up in conversations with researchers aren't the "what is it" questions anymore. You've moved past that. Now you want to know if it'll actually work for your situation, or if this is just another shiny thing that'll disappoint you like every other AI tool that promised the moon and delivered a slightly better autocomplete.
Fair. Let's get into it.
AI-Moderated Studies aren't trying to replace human moderators. They're not even trying to be human moderators.
This is a different research method entirely. One that lets you run conversational interviews at a scale that was never possible before, without sacrificing the adaptive, follow-up-question depth that makes qualitative research valuable in the first place.
Think of it less as "automated interviews" and more as "qualitative research that doesn't hit a ceiling at 10 participants because your one senior researcher is already triple-booked."
You need 40 interviews to segment by persona with confidence. Your researcher has capacity for 7. Your PM wants results by Thursday. We've all been there.
AI-moderated studies run conversational sessions 24/7, in participants' timezones, with adaptive follow-ups based on what they actually say. Not scripted questions that barrel ahead regardless of the answer. When someone gives a vague response, the AI probes deeper. When someone gives a rich answer, it moves on. When someone mentions something unexpected (a competitor feature, a workflow hack, a frustration), it follows that thread.
Eight interviews is a guess. Fifty is evidence.
Here's what that looks like: a B2B software company wants to understand why at-risk customers are about to churn. With AI moderation, they interview 60-70 customers in a week, something that would take months with traditional moderation. They discover that integration failures are the real issue, not the product itself. That changes the retention strategy before those customers walk out the door.
Running the same study in Spanish, French, German and Japanese markets simultaneously? With human moderators, you're looking at $5K-10K per market for native speakers, plus weeks of coordination.
Askable's AI moderator supports 15+ languages natively. Participants engage in their language, transcripts are auto-translated, and you can compare cultural differences in themes without the coordination nightmare. Or the budget nightmare. Or the "it's 3am somewhere and someone needs to moderate" nightmare.
Same study. Four markets. Four languages. Simultaneously. No coordinator. No timezone logistics.
Here's something we didn't expect: participants are often more candid with AI than with humans.
When a customer told our AI moderator they were considering switching to a competitor, they weren't softening the blow to protect a relationship. They shared specific feature gaps, integration needs, and exactly what it would take to stay. That honesty is harder to get when participants know they're talking to someone from the company.
No social pressure = more truth.
Here's the play smart teams are running: 40-50 AI-moderated interviews for breadth and patterns, then 10-15 human-moderated deep dives on the juiciest findings.
You get statistical confidence from the scale (enough participants to actually segment properly) and strategic depth from the human conversations. It's not AI versus human moderation. It's AI plus human moderation, each doing what they're best at.
You couldn't afford 50 interviews just to figure out which 10 to do properly. Now you can.
We could pretend AI moderation is perfect for everything. But we'd rather you trust us on the things we recommend.
If you only need 5-10 interviews for early discovery, AI moderation adds complexity you don't need. Use your standard moderated tracks. They're designed for exactly this.
When research requires deep human connection, contextual observation, or building rapport over a two-hour conversation about something deeply personal? That's human moderator territory. AI can follow a thread. It can't read a room. The energy shift when someone describes their old workflow, the gentle redirect when a participant is circling something they haven't articulated yet.
AI outputs need human review before big decisions. For safety-critical or regulated contexts, human oversight isn't optional. It's the whole point.
The smartest teams pick the right tool for the question. Not the tool they happen to like.
This is where most AI interview tools fall apart. Not in the conversation, but in what happens after it.
Most platforms offer some form of automated analysis now. The problem is that it's almost always built on generic LLMs. They'll give you themes and summaries, but the underlying models are prone to hallucination, they flatten nuance into generic patterns, and the insights they surface come without clear evidence trails. When the output says "28 participants mentioned pricing concerns," can you click through to every quote? Can you see who said it, when, and whether it was unprompted or led? Usually not. A hallucinated pattern looks identical to a real one.
The teams that skip platform analysis entirely aren't better off. They download transcripts and run them through ChatGPT or Claude, which strips participant identity. A first-time user and a power user saying "it was confusing" are different findings, but the model sees identical text. And every analysis starts from zero. Nothing accumulates. Nothing connects to the next study.
If you can't trace a finding to its source, it's not a finding.
When an AI-moderated interview ends on Askable, analysis starts automatically. Every session goes through the same pipeline.
The AI identifies themes across all participant responses and clusters them by topic: "onboarding confusion," "feature requests," "pricing concerns." But it goes deeper than labels. Every meaningful statement is extracted as a specific finding: not just "checkout friction" but "users abandon checkout when shipping costs appear unexpectedly on the final screen."
Each finding links directly to the participant quote, their demographic profile, their screening responses, and the conversation moment. Every theme shows the number of participants who mentioned it. You're always one click away from the actual participant words that support a finding.
Not all evidence is equal, and the analysis reflects that.
Each insight carries a confidence score. Was this volunteered or prompted by a leading question? Does it confirm the obvious or challenge assumptions? Does it contain specifics or vague generalities? A participant saying "yeah I guess the checkout was frustrating" after a leading question gets scored differently than someone who volunteers "I had a $200 cart, spent 20 minutes looking for shipping costs, and just closed the tab."
Both exist in the system. Only strong evidence shapes the top-level findings.
Think of the AI handling 80% of the pattern recognition and quote extraction, while your researchers focus on the 20% that requires strategic judgment and interpretation.
Most teams analyse each study in isolation. Nothing connects to the next project. Every analysis starts from zero.
With Askable, findings from every AI-moderated study feed into your research repository automatically. Same process every time. A churn study from six months ago connects to an onboarding study from last week if the evidence links to the same behaviour.
And with Ask AI, you can query across all your studies in natural language. When a VP asks "what do customers think about our pricing?" you get answers drawn from 12 months of evidence, with confidence scores and source quotes, in minutes. No "we should probably run a study on that."
Someone new joins and wants to understand feature X? Instead of "ask Sarah, she ran a study on that once," they query the knowledge base and get every relevant finding with source material.
These come up in almost every conversation we have. If you're thinking them, you're not alone.
"I find AI moderators get repetitive and pushy on the same subject. Can this be limited?"
Yes. You control conversation depth through settings (Shallow, Moderate, or Deep) which determines how many follow-up questions the AI pursues per topic. If you've experienced frustrating "five whys" loops with other tools, try Moderate or Shallow depth. The AI is also designed to recognise when it's extracted enough from a topic and move on, rather than interrogating participants into exhaustion.
"Can you influence the style? Q&A style vs open conversation?"
You can set interview style: Professional (formal, structured), Conversational (friendly, natural, recommended for most studies), or Explorative (open-ended, encourages storytelling). The AI is designed for natural conversation flow, not robotic interrogation. It acknowledges what participants share and asks contextually appropriate follow-ups.
"Have participants given feedback on being interviewed by AI?"
We're transparent about it from the start. Participants know they're engaging with an AI interviewer, and most find the experience natural. Many appreciate the flexibility (complete on their schedule, in their timezone) and the lack of social pressure. Some participants are more forthcoming precisely because there's no relationship to manage. When you're not worried about what the interviewer thinks of you, you say what you actually think.
"What if the AI hallucinates or makes things up?"
This is where the thematic analysis pipeline matters. Every insight comes with confidence scoring and direct links to source quotes. When the analysis surfaces a theme like "pricing concerns mentioned by 28 of 68 participants," you can click through to every quote that supports that claim. It's not a black box. You can verify any finding in seconds.
"We don't need 50 interviews. 20 is plenty for qual."
For pure exploratory work? Absolutely. Twenty well-conducted interviews can be gold. But say you're researching three personas. With 20 total interviews, that's roughly 6-7 per persona. Not enough to confidently say "this segment thinks X while that segment thinks Y." With 50, you've got 15+ per persona. Now the segments actually separate.
Many teams use AI moderation as a first pass: run 50 interviews to find the patterns, then do 10-15 human deep dives on the most interesting threads. Scale for discovery, depth for understanding.
"Do you use AI moderators all the time, or are there projects where human moderation works better?"
Both have their place. AI-moderated interviews are ideal when you need scale (20+ interviews), speed (results in days), global research (multiple languages), or quantitative backing for qualitative insights. Human-moderated interviews are better for exploratory research with small samples, complex emotional topics, contextual observation, and building deep empathy.
The question isn't which is better. It's which is right for this question.
Languages: 15+ natively supported.
Turnaround: 24-48 hours for 50 participants.
Participants per study: Up to 15 (Lite) or 50 (Full Track).
Interview length: 10-15 minutes typical; you control depth.
Security: SOC 2 Type II, ISO 27001, GDPR compliant. Zero data retention with LLM providers.
We built Askable around a belief: the best decisions come from real conversations with real people. Everything here exists to make those conversations happen more often, for more teams, with better evidence behind them.
AI-Moderated Studies are available now, self-serve.
Ready to see how this works in practice? That's the "why" and "when." In Part 2, we cover the "how": real examples of how teams are using AI-moderated interviews, what good study setup looks like versus bad, and how to get started without overcomplicating it.
[Read Part 2: How to Run AI-Moderated Studies That Actually Work →]
Askable is the modern research platform for velocity-obsessed teams.
Let's chatGet a sneak peak into the product, and everything Askable can do for you.
Contact sales.svg)
.jpg)
