A 2022 systematic review in Social Science & Medicine found that across 23 empirical tests, saturation in interview studies arrived between 9 and 17 interviews. That is a defensible sample. The problem is what happens next: one analyst reads those transcripts, applies codes, names three themes, and the themes become backlog items. Nobody checks whether a second competent researcher reading the same transcripts would have found the same thing.
This guide is for research leads, UX teams and CRO practitioners who hand qualitative findings to people who will spend money on them. By the end you will be able to run an intercoder agreement check on a defined slice of your data, pick the right statistic, set a threshold you cannot move afterwards, and recognise when the exercise is the wrong tool.
What an agreement check actually tests
It answers one narrow question: applying this coding frame to this text, do two independent people make the same decisions? It does not tell you the codes are the right codes, or that your themes are true. It tests whether your frame is defined clearly enough for someone other than its author to use.
That narrowness is why it is worth doing. A frame only its author can apply is not a frame, it is a habit. When a client challenges a finding six months later and the original analyst has left, a documented agreement figure is the difference between a defensible study and an anecdote.
When to run the check, and when it is the wrong tool
Run it when coding is deductive or codebook-driven: a defined frame, more than one coder, and output informing a decision someone is accountable for. O'Connor and Joffe (2020) treat that as the condition under which reliability is coherent at all.
Do not run it on reflexive thematic analysis. Braun and Clarke (2021) are explicit that there, researcher subjectivity is a resource rather than a threat, and demonstrating between-coder agreement is "illogical, incoherent and ultimately meaningless." A kappa on reflexive TA signals you have misread the method.
A 2024 review in the American Journal of Qualitative Research adds the sharper point: used for internal reassurance only, intercoder agreement creates an "illusion of rigor" and crowds out negative case analysis and triangulation.
The seven-step procedure
1. Freeze the coding frame first
Draft the frame, test it informally on one transcript, revise, then freeze. Checking agreement against a frame you are still editing measures nothing. Keep it small: O'Connor and Joffe cite practical limits of roughly 20 codes (Hruschka et al., 2004) and 30 to 40 (MacQueen et al., 1998). Reliability falls as the frame grows, because more codes means more near-neighbours to confuse.
2. Pick the subsample
Double-code 10% to 25% of your data units, selected randomly or by stratified sampling so the slice represents the whole. On a twelve-interview study that is two to three transcripts, and the coders do not choose which.
3. Code independently
At least two coders, working separately on the same subsample, no conferring. If your project has one analyst, bring in a second person for the subsample only.
4. Pick the statistic, and report it per code
Do not report raw percentage agreement alone: it is inflated by chance. Use a chance-corrected statistic: Cohen's kappa for two coders on nominal codes, Krippendorff's alpha for more than two coders or mixed data levels. Chi-square, Cronbach's alpha and Pearson's r are not appropriate. Report agreement for each code, never as one pooled number; a pooled figure hides the two failing codes behind the eighteen that are fine.
5. Set the threshold before you look at the result
There is no universal cutoff, which is why you commit to one in advance.
| Benchmark | Value | Source |
|---|---|---|
| Landis & Koch bands | 0.41–0.60 moderate; 0.61–0.80 substantial; 0.81–1.00 almost perfect | Landis & Koch, 1977 |
| McHugh's floor | Below 0.60 is inadequate | McHugh, 2012 |
| Neuendorf | Above 0.80 acceptable to many, above 0.90 to all | O'Connor & Joffe, 2020 |
| Chance warning | Around 0.60 may be chance alone | Krippendorff, via AJQR 2024 |
We recommend 0.70 per code for commercial research that informs spend, with anything below 0.60 revised and retested or discarded. Raise it for medical, regulatory or large financial decisions.
6. Resolve disagreements by a rule you wrote down first
A third coder with majority rules, deference to the most experienced coder, or negotiated consensus. Negotiation works. Campbell et al. (2013), cited in O'Connor and Joffe, report agreement rising from 54% to 96%. It also invites deadline-driven capitulation, so name the rule in advance.
7. Recode the whole dataset with the final frame
A check on 20% of the data does not license leaving the rest coded under an earlier draft.
Worked example: why 94% agreement can mean almost nothing
Two coders, 100 segments, one code. Identical observed agreement, wildly different kappa.
Case A, a common code ("price anxiety"). A applies it 10 times, B 8 times, they agree on 6.
| B: yes | B: no | |
|---|---|---|
| A: yes | 6 | 4 |
| A: no | 2 | 88 |
Observed agreement = (6 + 88) / 100 = 0.94. Expected = (0.10 × 0.08) + (0.90 × 0.92) = 0.836. Kappa = 0.104 / 0.164 = 0.63. Substantial, and just over McHugh's floor.
Case B, a rare code ("distrust of the installment table"). A applies it 6 times, B 4 times, they agree on 2.
| B: yes | B: no | |
|---|---|---|
| A: yes | 2 | 4 |
| A: no | 2 | 92 |
Observed agreement = (2 + 92) / 100 = 0.94, identical. Expected = (0.06 × 0.04) + (0.94 × 0.96) = 0.9048. Kappa = 0.0352 / 0.0952 = 0.37. Unusable.
Both codes show 94% agreement. One is reportable, one is noise. That is the argument for chance correction and per-code reporting, and it is why rare codes, often the interesting ones, need more double-coded data or a stated caveat. McHugh suggests 30 comparisons as a floor for an interpretable kappa.
The Turkish-and-English problem
If you run research in Turkey and report to stakeholders in English, translation sits inside your analysis rather than after it. The 2022 review noted that multi-country studies were among the outliers needing larger samples than the 9-to-17 range. van Nes and colleagues (European Journal of Ageing, 2010) worked through the mechanics; applied to a Turkish study reported in English:
- Code in the source language. Both coders code Turkish transcripts in Turkish. Translating before coding means you measure agreement on a translator's choices.
- Translate at the interpretation stage, when the team must explain findings in English, and budget discussion time for it. Near-synonyms carry different weight: their example was "engaging" and "absorbing" read as equivalent, losing the contrast.
- Keep culturally bound terms untranslated, with a gloss. A one-word equivalent strips the cluster of meaning. Their case was the Dutch gezellig flattened to "cosy". Turkish studies hit the same wall with the words participants use about trust and obligation.
- Use back-translation as a check on loss, not a guarantee of recovery.
If your coders do not share the source language, you cannot run this check properly. Say so in the method section rather than reporting a kappa on translated text as if it were equivalent.
Where this breaks down
- It is not validity. Two coders can reliably apply a frame that misses the point. Agreement is a precondition for a defensible study, not evidence of a correct one.
- Coders can agree on a shared bias. Researchers trained by the same lead and briefed by the same client converge, and the statistic cannot tell convergence from correctness.
- Rare codes resist it. The arithmetic above is not a flaw in your study; it is what chance correction does when a code is scarce.
- It is expensive. Double-coding a quarter of twelve hour-long interviews is real analyst time; on a small discovery sprint, triangulation may buy more credibility per hour.
[INTERNAL DATA NEEDED: Switas per-code kappa ranges from our own moderated studies, typical values for behavioural vs. attitudinal codes, and how often a frozen frame needed a second revision round. Would replace the generic 0.70 default with our own figure.]
Pre-flight checklist
- Frame frozen and documented, under about 20–30 codes
- Subsample at 10–25% of units, random or stratified
- At least two coders, coding independently
- Statistic chosen in advance: kappa, or alpha for more than two coders
- Threshold written down before results are seen, per code, not pooled
- Disagreement-resolution rule named in writing
- Minimum 30 comparisons per code, or a stated caveat
- Source-language coding confirmed, translation stage identified
- Plan for low-scoring codes: revise, discard, or retain with justification
- Full dataset recoded with the final frame
FAQ
How much of my data do I have to double-code?
Between 10% and 25% of your data units, selected randomly or by stratified sampling so the slice represents the whole dataset. On a twelve-interview study that is two to three transcripts; smaller datasets sit at the upper end.
Is percentage agreement ever acceptable on its own?
Rarely. It is inflated by chance, and the example above shows two codes at identical 94% agreement with kappa of 0.63 and 0.37. McHugh allows that it may suffice when coders are well trained and guessing is unlikely, but report it alongside a chance-corrected figure, never instead of one.
Cohen's kappa or Krippendorff's alpha?
Kappa for two coders on nominal codes. Alpha when you have more than two coders or data at different measurement levels, which is why recent methods guidance leans towards it. Both are chance-corrected; alpha is harder to compute by hand.
What kappa should I aim for?
No universal cutoff exists, so commit to one before you look. For commercial research informing spend, set 0.70 per code with a 0.60 hard floor; medical, regulatory or large financial decisions warrant 0.80 or above.
Does this apply to usability findings as well as interviews?
Yes, wherever you apply a defined frame to open-ended text or observed behaviour: session notes, open survey responses, support tickets, review mining.
Can AI coding assistants replace the second human coder?
Not as a reliability check. Agreement between a human and a model tells you about the model, not about the frame's clarity for your team. Use it to pre-code and speed up the human pass, then run the check between humans.
Run this on your next study
Take the ten-point checklist to the study you are closest to shipping and count how many points you can answer today. Under seven, and the findings are less defensible than they look. Have us run the study, or audit the analysis behind one you have already delivered.
Sources
- O'Connor & Joffe (2020), International Journal of Qualitative Methods 19
- Hennink & Kaiser (2022), Social Science & Medicine 292:114523
- Braun & Clarke (2021), Qualitative Research in Psychology 18(3):328–352
- McHugh (2012), Biochemia Medica 22(3):276–282
- Inter-Coder Agreement in Qualitative Coding, American Journal of Qualitative Research (2024)
- van Nes et al. (2010), European Journal of Ageing
