The bottleneck was never the interview
Every article about AI and user research opens the same way: research is slow, AI makes it fast. That framing is not wrong, but it is imprecise enough to be useless when you are deciding what to actually change on Monday morning.
Here is the more accurate version. Running a moderated interview takes sixty minutes and always will, because a person has to talk for sixty minutes. What used to take four days was everything around it — transcribing, tagging, finding the moment where three participants said the same thing in three different vocabularies, and turning that into something a product manager could act on before the sprint closed.
That is where the time went, and that is precisely where language models are strong. They are pattern-matchers over text. Research synthesis is a pattern-matching problem over text. The fit is real.
But the same property that makes them good at synthesis makes them dangerous at it. A model that finds patterns will find patterns whether or not any exist, and it will describe the ones it invented in exactly the same confident register as the ones that are there. In research, a confident wrong theme is worse than no theme, because the wrong theme gets built.
So the question is not whether to use AI in user research. Everyone already does. The question is which parts of the process you hand over, what you check before you trust the output, and what you refuse to automate at all. That is what this article is about.
Where AI actually earns its place
We map AI against the research process stage by stage rather than treating it as a single decision. Different stages have very different risk profiles.
| Stage | What AI does well | What it gets wrong | Our rule |
|---|---|---|---|
| Recruitment & screening | Drafting screeners, spotting contradictory answers in screener responses, flagging likely professional respondents | Over-filters for articulate participants; screens out the frustrated users you most need | Assist only. A human approves the final panel. |
| Discussion guide | Generating a first draft from the research question, proposing follow-up probes, checking for leading questions | Produces generic guides that test what is already known | Draft accelerator. The researcher rewrites at least half. |
| Moderation | Real-time note-taking, live tagging, suggesting probes to a human moderator | Cannot read hesitation, discomfort, or the pause before a polite lie | Never autonomous. Support tool for a human moderator. |
| Transcription | Speaker separation, timestamping, first-pass cleanup | Turkish transcription accuracy drops sharply with domain jargon, brand names, and code-switching | Automate, then spot-check 10% against audio. |
| Coding & tagging | Applying an existing codebook consistently across hundreds of transcripts at speed | Inventing new codes mid-corpus; collapsing distinct problems into one comfortable label | Automate with a fixed, human-authored codebook. |
| Synthesis | Clustering, surfacing cross-participant patterns, drafting the first narrative | Confuses frequency with importance; smooths over the dissenting participant | Draft only. Every theme traced to source before it ships. |
| Continuous feedback analysis | App store reviews, support tickets, NPS verbatims, chat logs at volume no team can read manually | Sentiment scoring reads sarcasm and cultural indirectness badly | High-value automation with sampling checks. |
| Artifact building | Personas, journey maps, opportunity trees from validated inputs | Produces plausible artifacts from thin or absent evidence | Only from verified research, never from the model's world knowledge. |
Two rows in that table are worth more than the rest combined.
Coding at scale. If you have a codebook a human wrote, applying it consistently across 40 transcripts is exactly the kind of tedious, rule-following work models are reliable at. This is where most of the recoverable time actually is.
Continuous feedback analysis. Most companies sit on thousands of app store reviews, support tickets, and survey verbatims that nobody has read since they arrived. This is the highest-return, lowest-risk use of AI in the whole discipline, because you are not replacing research that was happening — you are reading data that was being thrown away. We cover this in depth in How AI Can Transform User Feedback into Actionable Product Insights.
The four failure modes we keep catching
These are not theoretical risks. They are the specific things that go wrong on real projects, in roughly descending order of how often we catch them.
1. Frequency masquerading as importance
Ask a model to summarise twenty interviews and it will foreground what was said most often. But research value is usually not in the modal response. It is in the one participant who abandoned the flow entirely, or the two who described the same workaround independently, or the segment that never made it to the step everyone else discussed.
A model summarising a checkout study will tell you that shipping cost transparency came up repeatedly. It will not tell you that the only two participants who completed the purchase were both already logged in — which is the actual finding, and which appears in the corpus only as an absence.
Our check: we ask explicitly for the minority position. "Which participant disagreed with the emerging consensus, and what did they say?" If the model cannot name one, we treat the summary as incomplete rather than as evidence of agreement.
2. Fluent invention
Models produce quotes that sound exactly like something a participant would have said, attributed to a participant who did not say it. Not often. Often enough. And the failure is invisible on inspection, because a fabricated quote reads better than a real one — real people speak in fragments.
Our check: every quote that reaches a client deck carries a transcript reference and a timestamp. If it cannot be located in source audio, it does not ship. This is a mechanical rule, not a judgement call, because judgement is exactly what fails here.
3. Sycophantic synthesis
Give a model your hypothesis and then your data, and the synthesis will support your hypothesis. Give it the same data with the opposite hypothesis and the synthesis will support that. This is the single most expensive failure mode in a consultancy context, because it produces exactly the deliverable the client wanted while destroying the reason they hired you.
Our check: synthesis prompts never contain the hypothesis. The model gets the corpus and the research question, nothing else. When we want a hypothesis tested, we run it as a separate, adversarial pass: "find the strongest evidence in this corpus against the following claim."
4. Synthetic users
The most confidently marketed and least defensible application: generating simulated participants and interviewing them instead of people. It produces plausible transcripts fast, and the plausibility is precisely the problem. A model trained on the internet will tell you what a persona sounds like, which is a summary of how such people are written about — not how they behave when your form validation rejects their address on the third attempt.
Synthetic users are useful for one thing: rehearsing a discussion guide before you spend a real participant on it. We use them for that. We do not use them as evidence, and we do not report their output as research findings. If a supplier is offering you a research programme with no human participants in it, they are selling you a very expensive way to confirm what you already believe.
Our working protocol
This is the actual sequence, not an idealised one.
1. Humans write the codebook first. Before any model touches a transcript, a researcher reads three to five interviews and builds the coding frame by hand. The frame comes from this study's data, not from a generic UX taxonomy. Everything downstream inherits its quality from this step, which is why it is the step we never compress.
2. The model applies the frame, and only the frame. Fixed codebook, no new categories mid-run. When the model encounters something the frame does not cover, it flags it as unclassified rather than forcing a fit. The unclassified pile is often the most interesting output of the entire run.
3. Double-code a 15% sample. The same transcripts get coded by a human and by the model independently, and we measure the disagreement rate. Under 10% and we proceed. Above 20% and the codebook is wrong, not the model — we go back to step one. The disagreement rate is a diagnostic on the frame, and treating it that way has saved us more than treating it as a quality gate on the tooling.
4. Synthesis draft with no hypothesis in the prompt. The model gets corpus and research question. It produces clusters and a first narrative.
5. Traceability pass. Every claim in the draft gets linked to at least two participants, with timestamps. Claims that cannot be traced are deleted, not softened. A claim supported by exactly one participant is labelled as a single observation, which is a legitimate and often valuable thing to report — but not as a theme.
6. Triangulation against behavioural data. Stated behaviour and observed behaviour diverge constantly. We check interview findings against session recordings and heatmap data — rage clicks, dead clicks, drop-off points — before anything reaches a client. When participants say the filter is fine and the recordings show them abandoning the filter, the recordings win. This is also where AI-driven qualitative work is most likely to be caught out, because behavioural data does not care how coherent the narrative was.
7. A human writes the recommendation. Findings can be assembled with assistance. What to do about them, in this organisation, with these constraints and this roadmap, is a judgement call made by someone who sat in the interviews.
What we do not automate
Short list, and we hold it firmly.
- Moderation of interviews with vulnerable or distressed users. Healthcare, financial hardship, accessibility research. The moderator's job there is partly duty of care, and that is not delegable.
- The recommendation. See step seven.
- Accessibility evaluation. Automated tooling — including our own WCAG audit tool — catches roughly a third of WCAG issues. The rest require manual testing with assistive technology. Any vendor claiming full automated coverage is describing a scan, not an audit.
- Deciding what not to research. Scoping is where most research value is created or destroyed, and it is a strategic conversation, not a summarisation task.
A note on Turkish-language research
Most published guidance on AI-assisted research is written and validated in English, and the performance gap is real and under-discussed.
Turkish transcription degrades noticeably around brand names, sector jargon, and the English-Turkish code-switching that is completely normal in Istanbul product and e-commerce teams. Sentiment classification handles Turkish indirectness poorly — the polite hedge that precedes a serious complaint frequently reads as neutral or positive to a classifier trained predominantly on English. Agglutinative morphology also means that keyword-frequency approaches to open-ended survey data undercount badly, because a single concept appears in a dozen inflected forms the tokeniser treats as unrelated.
Practical consequences: raise your transcription spot-check rate above the 10% you would use for English, never rely on automated sentiment as a standalone signal in Turkish corpora, and validate any classifier on a hand-labelled Turkish sample before you trust its output at volume. We run our own validation set for exactly this reason.
How to tell whether it is working
Speed is the wrong metric. Everyone gets faster; the question is whether the output survives contact with reality. Three measures we actually track:
Insight survival rate. Of the findings delivered in a research report, what proportion were still considered valid six months later? This is uncomfortable to measure and worth every bit of the discomfort.
Human–model disagreement rate. From step three above. Track it over time. Rising disagreement usually means the research domain has shifted and the codebook has not.
Time from last interview to first testable hypothesis. This is the number AI genuinely moves. If synthesis used to take eight days and now takes two, that is two extra experiment cycles per quarter — which is the actual business case, and it is a good one. It just is not the case most vendors make.
Going deeper
This article is the overview. Four companion pieces cover the individual stages in detail:
- Creating Data-Driven User Personas with Artificial Intelligence — building persona artifacts from validated evidence rather than model priors
- From Data to Decisions: How AI Can Streamline User Research Synthesis — the synthesis stage in depth
- How AI Can Transform User Feedback into Actionable Product Insights — reviews, tickets, and NPS verbatims at scale
- Streamlining Product Discovery with AI-Powered User Research — applying all of it inside a discovery cycle
Frequently asked questions
Can AI replace user interviews entirely? No. It can replace some of the work around interviews — transcription, coding, first-pass synthesis — and it can help you prepare better ones. It cannot generate evidence about how people behave, because it has no access to their behaviour. Simulated participants tell you how such people are described in text, which is a different thing and frequently a misleading one.
How much time does AI-assisted research actually save? In our experience the saving concentrates almost entirely in synthesis and coding, where it is substantial. Fieldwork, recruitment, and stakeholder alignment are unchanged. A realistic expectation for a moderated study is that the post-fieldwork phase compresses considerably while total project duration falls by much less — because fieldwork was always the long pole.
Are synthetic users ever legitimate? As a rehearsal tool for discussion guides and as a way to pressure-test survey wording, yes. As evidence in a research finding, no. The distinction matters: one is a preparation aid, the other is fabricated data with a research label on it.
What is the biggest risk? Confident, fluent, wrong output — and specifically the version of it that agrees with whatever the team already believed. Fabricated quotes are caught by traceability rules. Sycophantic synthesis is harder, because it produces a deliverable everyone is happy with. The only reliable defence is separating the hypothesis from the synthesis prompt and running an explicit adversarial pass.
Does AI-assisted research work as well in Turkish? Not out of the box. Transcription accuracy, sentiment classification, and keyword-frequency methods all degrade on Turkish relative to English. It is usable and we use it daily, but it requires higher sampling rates, validated classifiers, and a researcher who reads the language.
Want a second opinion on your research process?
If you are running AI-assisted research and want to know whether your findings would survive scrutiny — or you are deciding what to automate and what to protect — we will review your current process and tell you plainly where the risk sits.
