Participants were drawn from the Global Clinical Practice Network (GCPN), a worldwide network of mental health professionals established for the purpose of the ICD-11 case-controlled field trials. Mental health professionals were invited to join the GCPN through national and regional professional associations; international and national conferences in psychology, psychiatry, and related disciplines; professional listservs; and professional word-of-mouth. For more information on the history and development of the GCPN, see Reed et al. (2015). The study was implemented in three languages: English, Spanish, and Japanese†. At the time the study was launched, 3,669 GCPN members met eligibility requirements: (a) self-rated advanced proficiency or fluency in one of the three study languages, and (b) currently providing clinical services to patients or engaged in direct clinical supervision. Eligible GCPN members were sent a personalized initial invitation email, followed by reminder emails two and four weeks later. Data collection lasted two months for each language. Following planned data inspection procedures, forty-three responses were dropped from the database, either for completing the study twice (e.g., in multiple languages) or otherwise being ineligible for the study (e.g., being a member of a group with a formal role in aspects of ICD-11 revision directly related to the study). Of the eligible participants, 2,320 (64%) responded to the survey link and began the study. Of the 2,320, 1,738 (74.9%; 47.37% of total) completed enough of the study for data analysis.

Thus, the final sample consisted of 1,738 mental health professionals representing 76 different nationalities. The regional distribution of participants is shown in Table 2, along with their gender, profession, mean age, and years of experience. Participants who completed the study were no different than those who were invited but did not participate in terms of age or years of experience. The participants who completed the study were slightly more likely to be male and to come from the Asian region (virtually all Japanese-speaking participants came from this region).

The study was administered through Qualtrics, a web-based survey program. The materials in the study included proposed diagnostic guidelines for Disorders Specifically Associated with Stress for ICD-11 (PTSD, Complex PTSD, Prolonged Grief Disorder, Adjustment Disorder, Acute Stress Reaction, and Other Disorder Specifically Associated with Stress) and ICD-10 (PTSD, Enduring Personality Change after Catastrophic Experience, Adjustment Disorder, Acute Stress Reaction, and Other Reaction to Severe Stress), a series of 11 case vignettes specifically developed for diagnostic comparisons, and diagnostic and clinical utility questions regarding the vignettes.

Specifically, participants provided a diagnosis (or indicated that no diagnosis was warranted) for each vignette with which they were presented and then answered specific questions about the presence or absence of each essential feature of the diagnosis as specified in the diagnostic guidelines, using wording adapted from the ICD-10 or ICD-11 diagnostic guidelines. Participants also rated the ease of use of the diagnostic guidelines on a four point scale (Not at all easy to use to Extremely easy to use), the goodness of fit of the diagnostic guidelines for describing the vignette on a four point scale (Not at all accurate to Extremely accurate), how frequently they encountered patients with that diagnosis on a five point scale (Never to Very frequently: multiple times per week), how similar patients in their regular practice were to the vignette on a four point scale (Not at all similar to Extremely similar), the level of functional impairment of the person in the vignette on a five point scale (No impairment to Complete impairment), and the severity of the symptoms in the vignette on a five point scale (No symptoms to Extremely severe symptoms).

A total of 11 vignettes were generated for the study (see Table 3). Because of the specific content required for each vignette, and a desire to create case examples that matched both expert definition of the symptoms and actual presentation of those symptoms in real clinical situations, content experts (members of the Working Group for Disorders Specifically Associated with Stress) drafted initial versions of the vignettes based on explicit instructions. One common criticism of vignette studies is the artificiality of cases (see Evans et al., 2015); vignette writers were therefore instructed to draw upon their clinical experience and to base the vignettes on actual cases (obscuring personal details for confidentiality). The vignettes covered a range of ages and included both men and women, but did not include child cases. Although written by clinicians from diverse countries, the vignettes did not explicitly identify specific social or cultural groups.

Vignettes were explicitly developed to exemplify certain characteristics, such as a case that displayed all PTSD symptoms except for clear functional impairment (Comparison 2; see Tables 1 and 3). These experimental manipulations of the case descriptions were designed to test specific changes or additions to ICD-11. Vignettes were pre-tested by content experts (a) to determine whether the desired characteristics were indeed present, (b) to confirm the presence or absence of required features, and (c) to confirm expert consensus about the correct diagnosis for the case. Based on the results of the pretest, some vignettes were modified to clarify content or ensure that specific symptoms were recognizable.

All materials were developed in English and then translated into Spanish and Japanese. Translators followed a rigorous forward and backward translation procedure utilizing bilingual content experts. Occasionally, translation efforts suggested wording clarifications to materials in English in order to maintain conceptual equivalence.

Participants received an email invitation to participate in the study through Qualtrics. Upon entry to the study, participants were randomly assigned to view either the ICD-10 or ICD-11 diagnostic guidelines for the disorders included in the study. Participants were blind to their assigned condition. Once participants had reviewed the guidelines, they were randomly assigned to one of eight comparisons (see Table 1). Within each comparison, participants viewed two vignettes, paired to highlight a specific change from ICD-10 to ICD-11 (see Table 3). The order of presentation of the two vignettes was counterbalanced across participants. After reading a vignette, participants were asked to provide a diagnosis from a preset list (including the relevant disorders from ICD-10 or ICD-11, as assigned), write in a different diagnosis, or indicate that no diagnosis was warranted. Participants could refer to the assigned diagnostic guidelines while making their decision. They then rated the specific presence or absence of each of the essential features as specified in the ICD-10 or ICD-11 diagnostic guidelines (as assigned) for the diagnosis they had selected. After reviewing the specific diagnostic requirements, clinicians were given the option of changing their final diagnosis. Participants then rated the diagnostic and clinical utility questions described above. If the selected final diagnosis was incorrect, participants received an additional differential diagnosis question, inquiring why the person had assigned the selected diagnosis rather than the correct diagnosis (without explicitly identifying the selected diagnosis as incorrect). Participants then completed the sequence again for a second vignette (i.e., the vignette comparison was a within-participant variable). Participants were unable to go back and change their responses as the study progressed. The detailed description of the specific comparisons follows.

Comparison 1 (Vignettes 1A and 1B). The first comparison examined the new ICD-11 diagnostic requirement for PTSD that the event must be re-experienced as though it were occurring in the present, rather than simply remembered. Vignette 1A included a description of re-experiencing in the present, whereas vignette 1B included only memories of the stressful event. Otherwise, the two vignettes expressed similar symptoms of PTSD. Expert pretesting confirmed that no re-experiencing (as defined by ICD-11) occurred in vignette 1B. Under ICD-10 diagnostic guidelines, both would fit within the diagnosis of PTSD; however, under ICD-11 diagnostic guidelines only 1A should be labeled PTSD with 1B receiving another diagnosis (e.g., Other Disorder Specifically Associated with Stress).

Comparison 2 (Vignettes 1A and 1C). The second comparison tested the inclusion of functional impairment in the diagnostic requirements for PTSD under ICD-11. Vignette 1A included associated functional impairment where vignette 1C did not. All other symptoms of PTSD were equivalent between the two vignettes. Expert pretesting confirmed that functional impairment was not present in vignette 1C. Under ICD-10 diagnostic guidelines, both cases should receive a diagnosis of PTSD, while only 1A should be diagnosed as PTSD under the proposed ICD-11 diagnostic guidelines.

Comparison 3 (Vignettes 1A and 1D). Comparison 3 targeted the possible misapplication of the PTSD category by participants who assigned the diagnosis based on the nature of the stressor rather than on the nature of the symptoms. Vignette 1A described a traumatic event that is commonly associated with PTSD (a severe car accident), where 1D had identical symptoms but a less prototypic event (having part of a ceiling fall on him while conducting home repairs). Expert pretesting confirmed that the individual experienced a traumatic event as defined by ICD-11. Under ICD-11, both cases should be given a diagnosis of PTSD; under ICD-10 it would be justifiable not to assign a PTSD diagnosis to vignette 1D.

Comparison 4 (Vignettes 1A and 2A). Comparison 4 examined the differentiation of PTSD and Complex PTSD. Vignette 1A described classically defined PTSD (meeting all ICD-11 requirements) where vignette 2A met all ICD-11 requirements for Complex PTSD. A correct classification under ICD-10 would be a diagnosis of PTSD for vignette 1A and either PTSD or Enduring Personality Change after Catastrophic Experience for vignette 2A; under ICD-11 the correct response would be a diagnosis of PTSD for vignette 1A and Complex PTSD for vignette 2A. Evidence for the efficacy of the new category would be a higher proportion of participants selecting the correct diagnosis for vignette 2A using ICD-11 than ICD-10.

Comparison 5 (Vignettes 2A and 2B). Comparison 5 examined whether clinicians could discriminate between the symptoms required for a diagnosis of Complex PTSD and the nature of the stressor. Specifically, vignette 2B described an individual exposed to a chronic and extreme stressor from which escape was difficult or impossible that would commonly be associated with Complex PTSD. However, the person in vignette 2B displayed standard PTSD symptoms without any of the additional three symptoms required for Complex PTSD. Expert pretesting confirmed that the three additional Complex PTSD symptoms were not present in vignette 2B. If clinicians provided a diagnosis of Complex PTSD for vignette 2A and PTSD for vignette 2B under the ICD-11 system, they would be utilizing the system as intended.

Comparison 6 (Vignettes 3 and 4). Comparison of vignettes 3 and 4 examined the addition of the new Prolonged Grief Disorder category for ICD-11. Vignette 3 described a culturally normative grief reaction, where vignette 4 described a reaction that outlasted cultural expectations and fulfilled the diagnostic description for Prolonged Grief Disorder. It was predicted that under ICD-10 vignettes 3 and 4 would be more likely to receive no diagnosis, and those diagnoses provided for vignette 4 would have little consistency (e.g., Depressive Episode, Adjustment Disorder, Other Reaction to Severe Stress). Under ICD-11, vignette 3 should receive no diagnosis and vignette 4 should receive a diagnosis of Prolonged Grief Disorder. This comparison tested whether clinicians could differentiate the diagnosis of Prolonged Grief Disorder from normal grief and whether the addition of the diagnosis improves the ICD-11's coverage of psychopathological conditions representing appropriate targets of treatment.

Comparison 7 (Vignettes 5A and 5B). Comparison 7 examined whether clinicians could discriminate between the symptoms required for a diagnosis of Adjustment Disorder and the nature of the stressor. Vignette 5A fulfilled the diagnostic guidelines for ICD-11 Adjustment Disorder, evidencing both preoccupation with the stressor and functional impairment. Vignette 5B described an individual with the same symptomatic presentation of Adjustment Disorder, over the same time span, but who had experienced a type of stressor that is more commonly associated with PTSD. If clinicians provided a diagnosis of Adjustment Disorder for both vignette 5A and vignette 5B under both the ICD-10 and ICD-11 systems, they would be utilizing the systems appropriately. If, however, the nature of the stressor overrode the symptomatic presentation in the minds of clinicians, they would assign a diagnosis of PTSD for vignette 5B. Because the diagnostic guidelines proposed for ICD-11 for both Adjustment Disorder and for PTSD were more explicit than those in ICD-10, it was expected that this effect would be reduced for ICD-11.

Comparison 8 (Vignettes 5A and 6). Comparison of vignettes 5A and 6 addressed the changes for Adjustment Disorder in ICD-11, specifically the requirement of preoccupation with the stressor and the explicit inclusion of functional impairment. As noted, vignette 5A fulfilled the diagnostic guidelines for Adjustment Disorder, evidencing both preoccupation with the stressor and functional impairment. Vignette 6 evidenced a similar level of symptoms and the same time span as vignette 5A, but without preoccupation with the stressor or impairment in functioning. Under the ICD-10 diagnostic guidelines, both vignettes 5A and 6 should receive Adjustment Disorder diagnoses. Under the ICD-11 diagnostic guidelines, only vignette 5A should receive a diagnosis of Adjustment Disorder, while vignette 6 should receive no diagnosis.

Participants’ diagnostic decisions were compared across diagnostic systems for a single vignette or across vignettes within a single system using χ2 tests. When diagnoses were compared across diagnostic systems and vignettes (i.e., a three-way interaction), we used the G2 statistic, which is a log-linear transformation of the χ2 distribution (Rao & Scott, 1984).

This comparison was designed to examine whether clinicians recognized the proposed requirement that trauma be re-experienced in the present for PTSD in ICD-11. PTSD diagnoses dropped from 1A to 1B using ICD-11 (consistent with the changes in the diagnostic guidelines), but an equivalent drop occurred for ICD-10, χ2(1)=0.79,ns (see Figure 1). Thus, using ICD-10 as a baseline, there was not a significant reduction in the use of PTSD as a diagnosis for vignette 1B in ICD-11. The lack of re-experiencing in the present seemed to make clinicians less likely to give a diagnosis of PTSD under the ICD-10 diagnostic guidelines as well. Further, PTSD remained the most common diagnosis for 1B under ICD-11.

Figure 1.

Percentages of diagnostic choices for Comparison 1: Do clinicians appropriately recognize the required symptom of re-experiencing and apply it correctly as a diagnostic requirement for the ICD-11 diagnosis of PTSD?

Note: Correct diagnoses are in Table 3.

(0.1MB).

Given that clinicians did not follow the pattern expected for ICD-11 diagnoses of vignette 1B, it was instructive to see why they gave the diagnosis of PTSD and whether they endorsed the presence of re-experiencing in the present. As can be seen in Table 4, among the participants who gave a PTSD diagnosis for vignette 1B using ICD-11 guidelines (n=75), there was more uncertainty about re-experiencing than for any other element of the PTSD diagnostic guidelines. Many participants affirmed the presence of re-experiencing in contradiction to expert evaluations.

Comparison 2 examined the proposed ICD-11 requirement of functional impairment for PTSD. Specifically, vignette 1A evidenced functional impairment whereby vignette 1C showed the same symptoms but was still working and had intact relationships. As with comparison 1, there was not the expected shift for individuals using ICD-11; they gave diagnoses of PTSD at the same rate for both vignettes, χ2(1)=0.02,ns (see Figure 2). Of those who assigned a PTSD diagnosis to vignette 1C using ICD-11 (see Table 5), most endorsed the presence of functional impairment, but more indicated uncertainty than for any other element of the PTSD diagnostic guidelines.

Figure 2.

Percentages of diagnostic choices for Comparison 2: Do clinicians appropriately recognize functional impairment and apply it correctly as a diagnostic requirement for the ICD-11 diagnosis of PTSD?

Note: Correct diagnoses are in Table 3.

(0.09MB).

Comparison 3 examined how clinicians would respond to an event that is subjectively experienced as frightening or horrific but that is less commonly associated with PTSD (vignette 1D) versus a traumatic event that is more prototypic for PTSD (vignette 1A). The diagnosis of PTSD should not change from vignette 1A to 1D in ICD-11. Under ICD-10, other diagnoses would technically be correct, but there was nothing explicitly excluding vignette 1D from receiving a diagnosis of PTSD. Treating any diagnosis other than PTSD as incorrect, there was not a significant difference across vignettes using the ICD-11 or ICD-10, G2(4)=5.74,ns (see Figure 3). In other words, clinicians were equally likely to diagnose both cases as PTSD under both diagnostic systems. Subjectively defined, non-prototypic traumas were considered by clinicians using either system to be legitimate precursors for the diagnosis of PTSD.

Figure 3.

Percentages of diagnostic choices for Comparison 3: Do clinicians appropriately base the ICD-11 diagnosis of PTSD on the presence of the required core symptoms, or do they tend to over-diagnose PTSD based on a history of specific types of stressors?

Note: Correct diagnoses are in Table 3.

(0.1MB).

The proposed addition of Complex PTSD to ICD-11 necessitates an assessment of clinicians’ ability to differentiate it from regular PTSD. If Complex PTSD is an improvement over Enduring Personality Change after Catastrophic Experience in ICD-10, the diagnostic preference for vignette 2A (Complex PTSD) should be clearer for ICD-11 than for ICD-10. Clinicians were able to successfully distinguish cases of PTSD and Complex PTSD using ICD-11, χ2(2)=112.70, p<.0001. Further, the three-way interaction indicated that the change in correctness between vignettes shifted significantly from ICD-10 to ICD-11 in the expected direction, G2(4)=110.90, p<.0001 (see Figure 4), indicating that Complex PTSD was a cleaner solution than using the concept of Enduring Personality Change. In other words, the ICD-11 concept of Complex PTSD appeared to clarify the diagnostic decision for vignette 2A relative to the ICD-10 system.

Figure 4.

Percentages of diagnostic choices for Comparison 4: Can clinicians differentiate the proposed ICD-11 diagnostic requirements of Complex PTSD from those of PTSD?

Note: Correct diagnoses are in Table 3.

(0.1MB).

Comparison 5 examined the effect the nature of the stressor had in assigning a diagnosis of Complex PTSD. The analyses below focus only on those who gave diagnoses of PTSD, Complex PTSD, or Enduring Personality Change after Catastrophic Experience, ignoring other incorrect diagnoses as these are not relevant to differentiating PTSD and Complex PTSD (see Figure 5). An interesting split occurred. Participants diagnosing vignette 2A under ICD-11 did not significantly differ from those using ICD-10, χ2(1)=2.19,ns However, for diagnoses of vignette 2B, participants using the ICD-10 were more likely to diagnose the case with PTSD, χ2(1)=17.92, p<.001. The three-way interaction indicates that the shift between diagnostic systems was not equal, G2(4)=68.14, p<.0001. Participants using the ICD-11 were more likely to give a Complex PTSD diagnosis to vignette 2B on the basis of the nature of the traumatic event, while this option was less enticing for Enduring Personality Change after Catastrophic Experience in ICD-10.

Figure 5.

Percentages of diagnostic choices for Comparison 5: Do clinicians inappropriately diagnose Complex PTSD based on a history of a severe and long-lasting stressor rather based on the required symptoms?

Note: Correct diagnoses are in Table 3.

(0.11MB).

An examination of those participants using ICD-11 who gave either diagnoses of PTSD or Complex PTSD for vignette 2B follows. For those clinicians (n=50) who selected Complex PTSD, all but one endorsed that the trauma was extreme (see Table 6). However, many endorsed the presence of persistent changes in affect regulation, a sense of worthlessness accompanied by guilt, and sustained problems maintaining intimate relationships. Nonetheless, a number of individuals indicated that those symptoms were not present, or they were unsure. Of those clinicians who gave a diagnosis of PTSD (n=46), 16 recognized that a diagnosis of Complex PTSD was possible. The nature of the trauma swayed clinicians’ decisions, indicating that the diagnostic guidelines needed further clarification.

The addition of a diagnosis of Prolonged Grief Disorder assumes clinicians’ ability to differentiate it from a normal grief process. First, vignette 4 was nearly always recognized as Prolonged Grief Disorder under ICD-11, but a wide variety of diagnoses were offered under ICD-10 (see Table 7). Thus, the addition of Prolonged Grief Disorder provides a clearer mechanism for capturing this sort of case. Second, for ICD-11, clinicians distinguished clearly between the cases, such that vignette 4 received a different pattern of diagnoses than vignette 3, χ2(2)=116.65, p<.0001. However, clinicians under either system did not do particularly well in recognizing that vignette 3 should not receive any diagnosis. Treating any diagnosis as incorrect, clinicians using the ICD-11 system (50% correct, 50% incorrect) were marginally better those using ICD-10 (37% correct, 63% incorrect), χ2(1)=3.38, p=.06. Of the 20 individuals who gave a Prolonged Grief Disorder diagnosis to vignette 3, there was disagreement as to whether the grief reaction had lasted for an abnormally long period of time or had been a culturally normal response (50% Yes, 35% No, 15% Not Sure).

This comparison examined whether clinicians could discriminate between symptoms required for Adjustment Disorder and the nature of the stressor. The majority of clinicians identified vignette 5A as a case of Adjustment Disorder across both ICD-11 and ICD-10. Examining just those who gave diagnoses of PTSD and Adjustment Disorder while lumping all other diagnoses into an “other” group, a higher percentage of Adjustment Disorder diagnoses occurred under ICD-11 than ICD-10, G2(7)=135.14, p<.0001, even though both groups of clinicians had difficulty with the distinction (see Figure 6). Thus, the distinction between PTSD and Adjustment Disorder diagnoses was clearer using ICD-11 versus ICD-10 diagnostic guidelines.

Figure 6.

Percentages of diagnostic choices for Comparison 7: Do clinicians appropriately differentiate PTSD from Adjustment Disorder based on the required symptoms, or do they tend to inappropriately base this distinction on the nature of the stressor?

Note: Correct diagnoses are in Table 3.

(0.12MB).

For those who gave a diagnosis of PTSD for vignette 5B, the diagnostic system made a difference in participants’ level of certainty about the presence of particular guidelines. Specifically, under ICD-11 (n=28), there was little agreement about the presence of all three core symptoms of PTSD, with the majority indicating no presence of avoidance or hyperarousal (see Table 8). Under ICD-10 (n=51), that disagreement only occurred for symptoms of avoidance or emotional numbing.

Two important changes were proposed in the ICD-11 diagnostic guidelines for Adjustment Disorder: (a) functional impairment is necessary and (b) the person must exhibit preoccupation with the stressor. For vignette 5A, for both ICD-11 and ICD-10 the majority of diagnoses were Adjustment Disorder, and the relative rates of giving an Adjustment Disorder diagnosis versus another diagnosis versus no diagnosis were equal across systems, χ2(2)=4.33,ns. For vignette 6, the rate of assigning an Adjustment Disorder versus another disorder versus no disorder diagnosis did not change by diagnostic manual, χ2(2)=1.88,ns. Nonetheless, no diagnosis was more likely for vignette 6 when controlling for the lack of a difference between manuals, G2(4)=80.28, p<.0001 (see Figure 7). Thus, the ICD-11 Adjustment Disorder diagnostic guidelines did not provide a clearer basis for identifying cases that should not receive the diagnosis.

Figure 7.

Percentages of diagnostic choices for Comparison 8: Do clinicians appropriately exclude diagnoses of Adjustment Disorder that do not evidence preoccupation and functional impairment as required by proposed ICD-11 guidelines?

Note: Correct diagnoses are in Table 3.

(0.12MB).

The results of this study should be interpreted in light of its limitations. No vignette methodology can address all questions relevant to the development of ICD-11 diagnostic guidelines. Specifically, clinic-based field studies and epidemiological work are necessary to address the fit of the guidelines to actual clinical cases. Second, the GCPN is not intended to be a representative sample of all mental health professionals; the participants in the GCPN are specifically interested in contributing to the improvement of mental health diagnosis or they would not have volunteered to participate in the network. Thus, the results may not generalize to some clinicians, situations, contexts, or languages. However, the GCPN represents a broad sample of mental health professionals and the study received greater participation than most other studies of diagnoses. Third, the vignettes described adult cases and the clinical utility of the proposed changes for children and adolescents remains to be established. The key changes to PTSD and Adjustment Disorder are of particular importance in this regard because the greater specificity of the guidelines may lead to missed cases if child-specific expression of re-experiencing and preoccupation is not taken into account (Scheeringa, Myers, Putnam, & Zeanah, 2012). Fourth, we chose to collapse the results across all three languages in which the study was conducted. Small linguistic differences did occur, but none impacted the overall pattern of results presented here. It is worth noting that the Japanese sample was more homogenous, but this is also reflective of diagnostic practice in Japan. In Japan, only physicians may assign a diagnosis and the large majority of psychiatrists are male. Nonetheless, differences in culture and language are important in the interpretation and implementation of diagnostic guidelines, and deserve further attention.

Finally, there is an important question regarding whether diagnosis is best viewed as a process of systematic hypothesis-testing or as matching information to well-learned pre-existing templates or exemplars (Elstein & Schwartz, 2002). Investigating the recognition of changes to existing diagnoses, particularly relatively small changes, favors the hypothesis-testing approach and its application of specific rules. Clinicians using a pattern-matching approach may be slow to adjust their templates, such that performance levels only improve after repeated exposure to and practice with the diagnostic changes. Thus the vignette methodology may be best at identifying problems with the initial application of the new rules and have less to say about levels of accuracy that can be achieved over the longer term.