Patients with: (1) psychotic symptoms; or (2) mood, anxiety, or stress-related symptoms without psychotic symptoms were identified by a clinician working at the outpatient psychiatric service. Identification was based on the normal intake interview performed by a second-year psychiatry resident; the intake interview is basically intended to triage patients. The information yielded by these interviews includes sociodemographic data, current reason for consultation, basic information about the course and clinical presentation of the problem, which was used to select the protocol for the patient. In the presence of psychotic symptoms, the patient was referred to protocol 1, and in the presence of mood, anxiety or stress-related symptoms without psychotic symptoms, the patient was referred to protocol 2. We used this screening procedure to select an enriched sample of study participants likely to display the conditions that were the focus of the study (Reed et al., 2018).

After receiving a comprehensive explanation of the nature and aims of the study, and giving their written informed consent, all participants were interviewed simultaneously by two clinicians. One clinician in the pair was designated as the primary interviewer for that particular patient and the other as the observer.

Clinician raters were psychiatrists, or fifth-year psychiatry residents actively engaged in clinical work (i.e., involved in the assessment or treatment of people with mental health conditions) for an average of 10 or more hours per week.

All clinician raters participated in a half-day training session on the diagnostic guidelines and study procedures. ICD-11 diagnostic guidelines for the four disorder groups included in the study were provided to participating clinicians, who were asked to read them in detail prior to the face-to-face training session. The training curriculum and materials used for the face-to-face training, developed by WHO, comprised a presentation of the innovations proposed for the ICD-11 diagnostic guidelines for each diagnostic group included and the main conceptual features of the diagnostic guidelines for each category. As part of the training, clinician raters practiced applying the diagnostic guidelines to case vignettes, and discussed the issues that arose during this process. Clinician raters were also provided with information on the study purpose, rationale, and methods, including a tutorial on how to use the Electronic Field Study System for data entry.

The local Institutional Ethics Review Board approved all the procedures used as a part of this study, including the consent forms for both service users and clinicians. Although clinician raters had not been informed of any diagnostic formulation made by the referring clinician before conducting their diagnostic interview, they were provided with a brief clinical summary of the participant prepared by the second-year resident conducting the triage intake interview that did not include diagnoses or psychotropic medications.

During the training, clinician raters were informed that they could also review other clinical information on the patients if necessary and available (including laboratory tests and brain images), with the proviso that both clinicians should look at the same information. Clinician raters then conducted a diagnostic interview of the participant in the way they deemed most appropriate. No specific instructions were provided for the interview except that in Protocol 1 (participants with psychotic symptoms), they should ensure they assessed Schizophrenia and Other Primary Psychotic Disorder, and in Protocol 2 (participants without psychotic symptoms but with affective, anxiety- or stress-related symptoms), they should ensure they assessed Mood Disorders, Anxiety and Fear-Related Disorders, and Disorders Specifically Associated with Stress. They were also instructed to assess any other diagnostic area appropriate to the participant's presentation, just as they would in a regular diagnostic interview. The member of the dyad designated as the interviewer for that participant conducted the interview, but the observer was allowed to ask additional questions at the end of the interview.

Clinician raters individually and autonomously entered the results of the diagnostic interview into a secure web-based electronic data capture system (the Electronic Field Studies System, developed using the Qualtrics survey platform specifically designed by the WHO Field Studies Coordination Group) for these studies (Reed et al., 2018). Clinician Raters selected up to three diagnoses they thought were applicable for the service user they had seen, or indicated that no diagnosis was warranted, and then provided diagnostic evaluation information including a thorough review of the essential features of each selected diagnostic category. This was done to ensure clinicians to include at least one of the diagnosis under study (Schizophrenia or Other Primary Psychotic Disorder in Protocol 1, and a Mood, Anxiety and Fear-Related, or Stress-Related Disorder in protocol 2), as well as the principal comorbid diagnoses within the same group of disorders or in other one.

In addition, clinician raters provided data on the severity of the service user's symptoms and their functional status, and answered questions about the clinical utility of the ICD-11 diagnostic guidelines as applied to the particular service user.

On the basis of earlier descriptions of the concept (Keeley, Reed, Roberts, Evans, Medina-Mora et al., 2016; Reed, 2010), the clinical utility of a classification construct or category for mental and behavioral disorders depends on its: (a) ease of communication (e.g., among practitioners, patients, families, administrators); (b) implementation characteristics in clinical practice, including goodness of fit (i.e., accuracy of description), ease of use and the time required to use it (i.e., feasibility); and (c) usefulness in selecting interventions and making clinical management decisions.

Accordingly, in the present study, the clinical utility of ICD-11 diagnostic guidelines was evaluated using a 4-point Likert scale to rate the different elements of these domains through a self-reported questionnaire applied to a particular patient. This scale was developed for the field studies designed to test the modifications proposed for ICD 11 (Keeley, Reed, Roberts, Evans, Medina-Mora et al., 2016; Reed, 2010) (see Table 7). Its factorial structure and internal consistency were evaluated prior to the main analyses regarding the clinicians’ perception of the guidelines’ clinical utility.

General characteristics of clinicians and patients were described using means and standard deviations for continuous variables and frequencies and percentages for categorical variables. All the variables were compared between protocols (1 and 2), using independent sample t-tests or chi-square tests depending on the type of variables. Frequencies and percentages were also calculated to evaluate the general level of agreement (No agreement/Overall agreement) between interviewers and observers across all diagnostic groupings. Comparisons of frequencies of each diagnosis provided by the interviewer and observer were made using McNemar tests. Kappa values were calculated in order to summarize the level of diagnostic agreement between interviewers and observers.

Basic psychometric properties of the clinical utility measurement were obtained by calculating an exploratory factor analysis (using likelihood maximum extraction, Oblimin rotation and Kaiser-Meyer-Olkin measure of sampling adequacy (KMO), and a confirmatory model (IBM SPSS Amos 21) for factorial or construct validity, as well as total and subtotal Cronbach's alphas for internal consistency or reliability.

Lastly, in order to analyze clinical utility information, the frequencies and percentages of each item were described for both interviewers and observers. Total means were compared between interviewers and observers using a t-test for independent samples. The significance level for all tests was established at p=.05.

Table 3 presents the Kappa's coefficients for the diagnostic guidelines of each ICD-11 diagnostic group.

The Scale of Clinical Utility of the ICD-11 Mental and Behavioural diagnostic guidelines was first evaluated in terms of its construct validity (factorial validity) and reliability (internal consistency). Table 4 presents the results of the exploratory factorial analysis of the scale, as well as internal consistency coefficients for the total and subtotal scores.

Two factors with eigen values above 1 together account for 60% of the variance. These factors involved the same general type of items. Factor one grouped together items regarding the clinical utility of the guidelines for case identification and management, while factor two included items concerning the evaluation of implementation characteristics. Cronbach's alphas were over .90 for the total and subtotal scores.

Confirmatory model is presented in Figure 1, showing a good adjustment of the factor structure resulted from the exploratory analysis (χ2=95.69, df=68, p=.015, GFI=.956, RMR=.01, CFI=.991, RMSEA=.038, CI90%=.017–.054).

Figure 1.

Scale of Clinical Utility: Confirmatory model.

Note: n=287 with the whole sample of interviewers and observers; χ2=95,69, df=68, p=.015, GFI=.956, RMR=.01,CFI=.991, RMSEA=.038, CI90%=.017–0.054.

(0.26MB).

According to this scale, a high proportion of clinicians considered that all the diagnostic guidelines studied are quite or extremely useful (Table 5).

In general terms, the more frequent answer option was, by far, the one referring to a good clinical utility (i.e., quite easy to use, quite easy to apply, quite useful, etc.), following the one related to a very good clinical utility (i.e., extremely easy to use, extremely easy to apply, extremely useful, etc.). When adding the frequency of both answer options, the clinical utility of the ICD-11 guidelines under study, given their implementation characteristics (ease of use, goodness of fit, clarity, amount of time required, etc.), were good or very good for more than 85% of the clinicians; ranging between 85.6% (for the description of the boundary between the patient's disorder and other disorders) to 91.5% (concerning their ease of use and clarity). Consistently, the clinical utility of the guidelines for identification and management of cases (including their utility to communicate with and educate patients and family) was rated as good or very good by more than 80% of the clinicians; ranging between 81.3% (for guidelines’ utility to asses patient's prognosis) to 88.9% (for qualifiers as helpful to select a treatment for the patient).

According to Cohen's criteria (Cohen, 1960), diagnostic agreement between raters using ICD-11 guidelines can be rated as strong for Schizophrenia and Other Primary Psychotic Disorders, moderate for mood and stress-related disorders, and week -although acceptable- for anxiety and fear-related disorders. However, consistent with McHugh (2012), “Cohen's suggested interpretation may be too lenient for health-related studies because it implies that a score as low as 0.41 might be acceptable” (pp. 276), and being strict, a kappa below to 0.60 as in the case of diagnoses for anxiety and fear-related disorders, indicates inadequate agreement among the raters.

This might be explained in part given such group of disorders was less common in the sample. Consequently, specificity was high for all the diagnostic groups under study while sensitivity was lower for anxiety and fear-related disorders (as well as other diagnoses). Another plausible explanation could be related with the high comorbidity of anxiety disorders with the other diagnosis under study. Thus, it is possible that, being such a common manifestation, hinders the diagnostic separation even though clinically it is more accessible for expert clinicians.

Still, although we did not provide any guidance on how the interview was to be conducted, and the majority of the cases presents with a clinically significant severity and comorbidity (given they were recruited in a specialized institution), observed kappa indexes were similar to those achieved using more complex and time-consuming instruments (such as structured or semi-structured clinical interviews) (Pies, 2007). And even though our results are not comparable to DSM-5 reliability studies, which used a different methodology, they challenge the assumption that a less rigid diagnostic guidelines are inherently less reliable (Craddock & Mynors-Wallis, 2014), probably because in their attempts to communicate the essence of the disorder, they are more similar to how clinicians think.

The present study also provides information on the perception of clinicians regarding the clinical utility of the diagnostic guidelines evaluated. This is important because of the emphasis on increasing the clinical utility (Keeley, Reed, Roberts, Evans, Robles et al., 2016; Reed, 2010) of the classification as a whole in order to provide a tool that will help reduce the global burden of disease though early identification and the treatment of health conditions.

Regarding the diagnostic guidelines for psychotic, mood, anxiety and stress-related disorders proposed for ICD-11, we can infer from our results that Mexican clinicians with extensive experience of attending psychiatric patients consider that they are of value in terms of their implementing characteristics (mainly regarding their ease of use and clarity) as well as for the identification and management of patients, specially their qualifiers to select a specific treatment. This important finding (given that the ultimate goal of a clinical useful classification is to help in the decision of a proper case management) seems to be in line with several WG's proposals, including a different system of qualifiers for Schizophrenia and other Psychotic Disorders, which considers the evaluation of the level of cognitive impairment that may indicate the need for cognitive remediation interventions.

Additionally, concerning the classification of depressive disorders, one of the common mental disorders responsible for a large burden of disease in Mexico, Latin America and globally (Medina-Mora et al., 2007; World Health Organization WHO, 2017), although ICD-11 classification was not been substantially modified, the proposed diagnostic guidelines include new severity qualifiers that were expected to improve their clinical utility (Chakrabarti, Berlanga, & Njenga, 2012) especially regarding treatment selection, which might varies considerably from a mild to a severe case. However, there are some space for additional improvements, mainly in terms of the guidelines’ utility to asses patient's prognosis, which could require, in many cases, not just a systematic effort to include the information needed to do so, but the generation of such scientific data by psychiatric entity. An additional contribution of this study is the psychometric evaluation of the Scale to Measure Clinical Utility (Keeley, Reed, Roberts, Evans, Medina-Mora et al., 2016; Reed, 2010) in a reliable, valid manner for future studies in the field.

According to our results, the ICD-11 would appear to constitute a reliable, clinically useful diagnostic system, at least as regards clinician consistency when the guidelines are used to identify mental disorders that account for the greatest proportion of years lived with disability, and for which there is a considerable treatment gap in both developed and developing countries (Pan American Health Organization PAHO, 2013; Wang et al., 2007).