- Mental health diagnosis reliability is lower than widely assumed, with retests producing the same result only 65% of the time.
- A McMaster University meta-analysis of 57 studies found mental health diagnosis reliability varies sharply by condition type.
- Schizophrenia spectrum disorders showed the worst consistency, with retests agreeing just 55% of the time — barely better than a coin flip.
- Researchers are calling on clinicians to stop treating standardized diagnostic interviews as the definitive gold standard.
- Mental health diagnosis reliability is lower than widely assumed, with retests producing the same result only 65% of the time.
- A McMaster University meta-analysis of 57 studies found mental health diagnosis reliability varies sharply by condition type.
- Schizophrenia spectrum disorders showed the worst consistency, with retests agreeing just 55% of the time — barely better than a coin flip.
- Researchers are calling on clinicians to stop treating standardized diagnostic interviews as the definitive gold standard.
The Tool Psychiatry Trusts Most Has a Reliability Problem
Mental health diagnosis reliability has long been treated as a settled question — interview a patient using a structured tool, score the answers, and arrive at a diagnosis. Clean, consistent, defensible. But a major new meta-analysis published in JAMA Network Open is putting serious cracks in that assumption, and the implications stretch far beyond academic debate. We’re talking about real patients walking away with the wrong label — or a different one depending on which day they were interviewed.
The study, led by researchers at McMaster University in Canada and co-authored with colleagues from the University of Copenhagen and UMass Chan Medical School in Boston, pooled data from 57 studies covering more than 8,000 adults across 26 countries. The focus was on standardized diagnostic interviews — or SDIs — the structured questionnaires that psychiatrists and psychologists use to formally categorize mental illness. What they found was uncomfortable: these tools are significantly less consistent than the field generally assumes.
Same Patient, Different Day, Different Diagnosis
The core finding is straightforward and troubling. When the same patient undergoes the same standardized interview just days apart — sometimes with the same clinician — there’s a meaningful chance the two sessions will produce different diagnostic conclusions. For mental health diagnosis reliability specifically, the retests matched the original result only about 65% of the time. That’s not a rounding error. That’s one in three patients potentially receiving a different psychiatric label depending on when they showed up.
Of the 57 studies examined, 46 were selected for detailed cross-comparison using Cohen’s kappa — a statistical measure that accounts for the possibility that some matches between tests happen purely by chance. Even with that correction applied, the numbers held up poorly. Substance use disorder assessments did better, with about 72% consistency across test-retest pairs. Opioid use disorder specifically came in at 81%, the highest of any category studied.
Bipolar disorder was also relatively consistent, with retests matching the original diagnosis roughly 74% of the time. But non-affective psychoses — a category that includes schizophrenia spectrum disorders — landed at just 55%. That’s essentially a coin flip. For some of the most serious and life-altering psychiatric diagnoses available, the diagnostic tool most clinicians rely on performs barely better than chance.
“If we give the same interview to the same person twice, we would like to think the interview would produce the same result, but that’s not always the case,
