Content validation using Aiken methodology through expert judgment of the first Spanish version of the Eppendorf Schizophrenia Inventory (ESI) in Peru: A brief qualitative report

Abstract

Background

The main objective of this study was the validation through the expert judgment of the first Spanish version of the Eppendorf Schizophrenia Inventory (ESI-Sp).

Materials and methods

A translation and cultural adaptation procedure as a first stage was carried out, which was evaluated by four psychiatrists with clinical experience. In a second stage, we recruited a group of clinical psychiatrists to evaluate the translated instrument by judging the items’ clarity and relevance. Content validity was performed using the Aiken methodology (Aiken's V).

Results

For the first stage, we did not find any items that needed alteration during translation and adaptation. Concerning the raters’ stage, 30 clinical psychiatrists (47.07 ± 13.62 years) completed the rating of the ESI-Sp. Of them, the majority worked in an institute (10/30; 33.33%) and had “15 or more” years of experience (12/30; 40.00%). The null hypothesis was rejected for all items and demonstrated the content validity of each item. Similar results were found for the factors of the ESI. Finally, the total Aiken's V for the ESI did not include the values of the null hypothesis in the confidence intervals (V = 0.78; CI95 [0.70; 0.85]), demonstrating content validity.

Conclusions

ESI-Sp is a valid and representative instrument to determine subjective experiences in patients with a high risk of schizophrenia. The construct validation of the ESI-Sp is pending, which our research group will carry out after this first step.

Introduction

Basic symptoms (BS) in schizophrenia are defined as subtle and subclinical alterations experienced in various areas of mental life.¹ Although the presentation of the BS varies between subjects, these subjective experiences englobe mostly neurocognitive alterations, affective alterations, and subclinical alterations of perception and thoughts.²

In the last decade, many research groups have worked on developing instruments that measure BS since their assessment could also help detect early schizophrenia.³ An example of those instruments is the Eppendorf Schizophrenia Inventory (ESI),^4–6 which measures BS phenomena such as speech and attention, ideas of references, auditory uncertainty, and dispersion of perception. Initial results have shown the advantages of ESI for measuring BS, demonstrating superiority over the Frankfurt Discomfort Questionnaire (FCQ).⁵ Additionally, its application was uncomplicated for the participant, being quick to fill out and untroubled to answer.

The diversity of populations worldwide and cross-cultural and international research indicate a necessity for validated instruments for different groups of people and languages. In the case of the ESI, for instance, there are several translations published (e.g., Korean or Italian),^6,7 except for the Spanish language, being the latter one of the most spoken languages, with about 491 million speakers worldwide.⁸ Additionally, in some emerging Spanish-speaking nations, such as Peru, where scientific production is progressively increasing, tools, such as the ESI, are required for schizophrenia research and clinical purposes. In the case of subjective symptoms of schizophrenia, an instrument such as the ESI will enable the physician to determine the individuals at high risk for developing psychosis. In addition to being unknown and necessary in Spanish-speaking countries, such as Peru, this information must be disseminated to the world scientific community since few instruments evaluate initial symptoms of schizophrenia in the Spanish language.

Therefore, the main aim of this study is to apply content validation to the first Spanish version of the ESI (ESI-Sp) using the Aiken methodology through expert judgment.

Materials and methods

Brief Description of the Instrument – ESI: The Eppendorf Schizophrenia Inventory (ESI) was created by Mass et al. and initially written in German⁵ to evaluate schizophrenia symptoms, especially at the early onset of the psychotic disorder. This instrument consists of 40 items (response options: 4 – true, 3 – mostly true, 2 – sometimes is true, 1 – that is not true) and four different factors⁵: attention and speech impairment (AS, 10 items), ideas of reference (IR, 7 items), auditory uncertainty (AU, 8 items), and deviant perception (DP, 9 items). As “control” questions, the “Frankness” group (FR, 5 items) was included, questions that assess the willingness or ability of the subjects surveyed to admit minor personal flaws in a general way.⁵

Translation and cultural adaptation: The original version in German was translated into Spanish by one of the authors (BPP). Then, we invited four psychiatrists from Lima (Peru) with the following characteristics: ten years of clinical experience and active professors in clinical Psychiatry. Our main goal for these four experts was to suggest changes regarding the wording and readability without changing the content of the items. Afterward, one of the authors (WH) re-translated the adapted instrument. The re-translated version was sent to the original author of the ESI to compare and contrast with the original version in German, obtaining thus the approval to use. Table 1 shows the adapted items in Spanish.

Table 1.

Spanish version of the Eppendorf Schizophrenia Inventory (ESI).

1. No puedo retener bien lo que pasa a mi alrededor.

2. Algunas veces mi oído es muy fino, por lo que escucho los sonidos habituales con mayor intensidad y nitidez.

3. Me he percatado que veo objetos que no son visibles para las otras personas.

4. Ocasionalmente se dicen cosas (por ejemplo, en programas de radio), que siento que guardan relación conmigo, aunque sé que esto no puede ser posible.

5. Cuando alguien hace oraciones largas al hablar, me cuesta comprender lo que se me dice.

6. Aún si escucho algo claramente, me siento inseguro y pienso que puede ser producto de mi imaginación.

7. Cuando miro objetos de uso cotidiano (por ejemplo, mesas o sillas), tengo a veces la sensación como si estos fueran extraños para mí.

8. A veces tengo la sospecha de que alguien está planeando algo malo en contra de mí.

9. A veces he hecho comentarios desagradables sobre otras personas.

10. A menudo me pongo a analizar lo que significan algunas palabras que la gente usa cotidianamente.

11. A veces siento que alguien o algo ajeno a mí controla mis pensamientos, comportamientos o sentimientos.

12. A veces tengo la impresión de no percibir bien el movimiento de mis brazos y piernas.

13. Tengo la sospecha de que alguien, que no soy yo, controla lo que estoy pensando.

14. Cuando veo la televisión, me cuesta mucho entender los programas y seguir al mismo tiempo las palabras e imágenes.

15. Frecuentemente me percato de que confundo algunos sonidos con voces.

16. A veces me pasa que algunas partes de mi cuerpo se ven más pequeñas de lo que realmente son.

17. A veces siento como si la disposición de las cosas a mi alrededor tuviera algún significado oculto para mí.

18. De vez en cuando me alegro un poco del mal ajeno.

19. Últimamente me percato de que algunas palabras de uso cotidiano tienen un significado oculto.

20. A veces escucho dentro de mi cabeza las voces de personas que no están presentes y creo que pueden ser almas o espíritus (por ejemplo, Dios, algunos ángeles o el demonio).

21. A menudo cuando veo un objeto, percibo algunas de sus partes con mayor intensidad sin estar enfocándome en estas (por ejemplo, veo los dedos de una mano con mayor intensidad que la mano misma).

22. A menudo creo que recibo señales del exterior que son importantes, personales y de las cuales sólo yo me percato.

23. De vez en cuando digo una mentira.

24. He olvidado muchas cosas que habitualmente solía hacer.

25. A veces oigo mis pensamientos casi tan claramente como si alguien estuviese hablando en el exterior.

26. He tenido por momentos la sensación de que mi cuerpo presenta alguna deformidad.

27. A menudo siento que algo extraño e inusual ocurre a mi alrededor.

28. A veces postergo cosas que debería de hacer de inmediato.

29. A veces me cruzo con personas y sólo después de verlas me doy cuenta de que las conozco bien.

30. Últimamente tengo la sensación como si alguien o algo me quitase mis pensamientos.

31. A menudo me irrita cuando alguien habla o realiza alguna actividad a mi alrededor.

32. Existen personas que pueden leer mis pensamientos por métodos que no puedo precisar.

33. Últimamente me irrito con más facilidad, cuando las cosas no resultan de acuerdo con lo que quiero.

34. Me resulta estresante participar activamente en una conversación y decir mis propias ideas en ese momento.

35. Últimamente algunos sonidos familiares (o usuales) suenan extrañamente diferentes para mí.

36. Tengo la impresión que en algunas ocasiones, puedo leer los pensamientos de otras personas (por ejemplo, a través de la telepatía o similares).

37. Me he sentido al borde de una revelación trascendental.

38. Cuando una persona me habla, no entiendo con frecuencia el significado de las palabras que esta emplea.

39. A veces me pasa que pierdo el contacto del entorno y no sé muy bien lo que está pasando a mi alrededor.

40. He respondido las preguntas de la forma más correcta y precisa posible.

Expert judgment and validation: After translation and cultural adaptation, we sent the translated version to different raters, a group of Peruvian psychiatrists with clinical experience. The selection of the raters was made by using the snowball method. Here, we introduced the translated version on an online free survey website (Google Forms) and distributed it using different social media or electronic mail addresses.

Ethical statements: This expert validation study was approved by the ethics committee of the Peruvian University “Cayetano Heredia”. The ethical procedures of the study were carried out according to the Helsinki Declaration. Finally, our study group received the translation rights of the ESI from the Hogrefe Editorial for research purposes.

Content validity using Aiken Methodology (Aiken's V): Content validity was calculated using the Aiken methodology (Aiken's V, V = 0: total disagreement, V = 1: total agreement, V = 0.5: indecision), which evaluates the number of agreements between experts. To measure the agreement between judges, we formulated two questions for each item (Clarity: Do you consider this item clear? Representativeness: Do you think this item's representativeness is adequate?). Both questions were scored using a Likert scale of 5 categories.

Aiken's V^9,10 was calculated from the judges’ information on both elements. Confidence intervals corresponding to 95% (CI95)^9,11 were also calculated, which allowed estimating the population parameters for the value of V (V_P). The null hypothesis was defined as the probability that the V_P takes values equal to or less than 0.50. Therefore, to accept the null hypothesis, the CI95 must include the value of V ≤ 0.5 within the range of values, giving the possibility that V_P ≤ 0.50. To calculate each CI95 for Aiken's V, we used the methodological proposal from Penfield and Giacobbi.⁹ Finally, the formulae for the Aiken methodology concerning content validation were calculated in Microsoft Excel 2019, following the recommendations published elsewhere.^9,11

Results

Translation and cultural adaptation: We did not find any items that needed alteration during translation, cultural adaptation, and retranslation. The contents of the questions remained stable with the ones of the original version. The final result of this phase is presented in Table 1.

Brief description of the judges: Initially, 32 judges were summoned to participate. However, two of them retired their participation for unknown reasons, not filling out the online survey for the ESI evaluation. Therefore, thirty judges (clinical psychiatrists; n = 30; 47.07 ± 13.62 years) completed the study and were included in the analysis. The majority of our participants were psychiatrists working in an institute (n = 10, 33.33%), followed by psychiatrists working in a hospital (n = 6, 20.00%), private practice (n = 6, 20.00%), community centers (n = 4, 13.33%), university (n = 2, 6.67%), forensic institutions (n = 1, 3.33%), and private clinic (n = 1, 3.33%). Regarding the years of experience, the majority had “15 or more” years of experience (n = 12, 40.00%), followed by “between 2 and 10” years of experience (n = 11, 36.67%). Finally, a small group of colleagues indicated “between 11 and 15” years of experience (n = 7, 23.33%). Concerning the field of expertise, most of the raters worked in adult psychiatry (n = 20, 66.67%).

Content validity using Aiken's V: We performed the content validity analysis using Aiken's methodology. The analysis was performed for each item, the factors, and the whole instrument, as presented in Table 2. Concerning the 40 items of the ESI, the null hypothesis, defined in the methods sections, was rejected for all items and demonstrated the content validity of each item (Table 2). Similar results were found for the four factors of the ESI: AS (V = 0.75; CI95 [0.67; 0.82]), IR (V = 0.81; CI95 [0.74; 0.87]), AU (V = 0.81; CI95 [0.73; 0.87]), and DP (V = 0.81; CI95 [0.73; 0.87]). Finally, the total Aiken's V for the ESI also did not include the values of the null hypothesis in the confidence intervals (V = 0.78; CI95 [0.70; 0.85]).

Table 2.

Content validity of the ESI – Spanish version (40 items and 6 factors). In this table, Aiken's V from 30 raters and its corresponding CI95 were represented for each item, factor, control item (“Item 40”) and the whole instrument.

Items (1–20)	V	CI95	Items (21–40)	V	CI95
Item 1	0.65	[0.56; 0.73]	Item 21	0.79	[0.71; 0.85]
Item 2	0.71	[0.63; 0.79]	Item 22	0.85	[0.78; 0.91]
Item 3	0.83	[0.75; 0.89]	Item 23	0.66	[0.57; 0.74]
Item 4	0.80	[0.72; 0.87]	Item 24	0.71	[0.62; 0.78]
Item 5	0.77	[0.69; 0.84]	Item 25	0.88	[0.80; 0.92]
Item 6	0.79	[0.71; 0.86]	Item 26	0.83	[0.75; 0.88]
Item 7	0.80	[0.72; 0.86]	Item 27	0.86	[0.79; 0.91]
Item 8	0.88	[0.80; 0.92]	Item 28	0.70	[0.61; 0.77]
Item 9	0.69	[0.60; 0.76]	Item 29	0.60	[0.52; 0.69]
Item 10	0.73	[0.65; 0.80]	Item 30	0.89	[0.82; 0.94]
Item 11	0.88	[0.81; 0.93]	Item 31	0.66	[0.57; 0.74]
Item 12	0.74	[0.66; 0.81]	Item 32	0.88	[0.80; 0.92]
Item 13	0.91	[0.84; 0.95]	Item 33	0.68	[0.60; 0.76]
Item 14	0.78	[0.69; 0.84]	Item 34	0.68	[0.60; 0.76]
Item 15	0.83	[0.75; 0.88]	Item 35	0.79	[0.71; 0.85]
Item 16	0.86	[0.79; 0.91]	Item 36	0.88	[0.81; 0.93]
Item 17	0.76	[0.68; 0.83]	Item 37	0.67	[0.58; 0.75]
Item 18	0.65	[0.56; 0.73]	Item 38	0.76	[0.67; 0.83]
Item 19	0.80	[0.72; 0.87]	Item 39	0.85	[0.78; 0.90]
Item 20	0.89	[0.82; 0.94]	Item 40 (Control)^b	0.82	[0.74; 0.88]

Factors	V	CI95
Factor “AS”	0.75	[0.67; 0.82]
Factor “IR”	0.81	[0.74; 0.87]
Factor “AU”	0.81	[0.73; 0.87]
Factor “DP”	0.81	[0.73; 0.87]
Factor “FR”	0.68	[0.59; 0.75]
Total^a	0.78	[0.70; 0.85]

a

Results for the total number of items (1–40).

b

Item 40 is defined as the “control” factor for the ESI instrument.

Abbreviations: CI95 = 95% confidence interval, AS = attention and speech impairment, IR = ideas of reference, AU = auditory uncertainty, DP = deviant perception, FR = frankness.

Discussion

Our results suggested that the first ESI-Sp has adequate representativeness and a degree of clarity following Aiken's methodology. Furthermore, the wording and readability changes during translation, cultural transformation, and retranslation remained stable for all the items of the ESI-Sp, showing similarities with the literature recommendation concerning qualitative validation.^12,13

Content validity analysis of the items individually and four factors through Aiken's V methodology showed significant results, consistent with the experts’ judgment that the first ESI-Sp is valid. However, given our study's design, we cannot wholly validate this instrument in people with schizophrenia onset or ultra-high-risk psychosis since the study for construct validation is missing. This qualitative expert validation study is the first step for an ESI validation project; we will proceed with a construct validation study with these results.

We have detected some strengths and limitations of our study and highlighted them. The primary strength of this short communication is the translation, cultural adaptation, and Expert judgment of the first Spanish version of this instrument. However, the main limitations of our study should be addressed. Firstly, even though the number of experts calculated for the sample size is enough, the sampling technique is non-probabilistic, increasing the risk of sample bias. Finally, the study design is a limitation of this study since it is impossible to make a statistical inference or, in this case, a construct validation based solely on the experts’ opinions.

In conclusion, our study suggests that the first ESI-Sp is a valid and representative instrument to determine subjective experiences in patients with a high risk of schizophrenia. The construct validation of the ESI-Sp is pending, which our research group will carry out after this first step.

Conflict of interest

The authors declare they have no conflict of interest.

Acknowledgments

The authors of the study would like to thank Hogrefe publishing house for the permission and license authorization of the Eppendorf Schizophrenia Inventory (ESI) translation in the Spanish language (Agreement for translation rights on 28/08/2018).