Teacher's reports about child psychopathology are often useful because they make it possible to compare children's development within their normative peer group. The EDUTEA questionnaire aims to provide clinicians and researchers with a brief tool that can be used to screen autism spectrum disorders and social communication disorders in school settings. Method: It was designed according to DSM-5 criteria and validated in a sample of 2,660 Spanish schoolers. Results: The EDUTEA showed a two-factor structure with high internal reliability: Social communication impairments (α=.95) and Restricted behaviour patterns (α=.93). The ROC curve showed that the area under the curve was highly predictive (.90). We propose using a cut-off score of 10, which gives high values of sensitivity (87%), specificity (91.2%) and positive predictive value (.87). Moderate correlations were found with the severity score of the Autism Diagnostic Observation Schedule (ADOS-2) and the child pragmatic competence. Conclusions: The EDUTEA could be useful in ASD screening protocols in schools.
La información que ofrecen los maestros sobre los problemas psicológicos suele ser frecuentemente muy útil dado que éstos pueden comparar el desarrollo del niño con su grupo de iguales. El cuestionario EDUTEA se creó para proporcionar a profesionales del ámbito clínico y de la investigación una herramienta para la detección de los trastornos del espectro del autismo y de la comunicación social pragmática en el entorno escolar. Método: Fue diseñado en base a criterios DSM-5 y validado en una muestra de 2.660 escolares españoles. Resultados: Presentó una estructura bifactorial con una elevada fiabilidad interna, comprendiendo los factores de Alteraciones de la comunicación social (α=0,95) y Patrones restrictivos de comportamiento (α=0,93). El análisis de curvas ROC indicó una elevada predictibilidad del área bajo la curva (0,90). Se propone utilizar la puntuación total de 10 como punto de corte, ya que proporcionó altos valores de sensibilidad (87%), especificidad (91,2%) y valor predictivo positivo (0,87). El cuestionario mostró correlaciones moderadas con la puntuación de severidad de la Escala de Observación para el Diagnóstico del Autismo (ADOS-2) y la competencia pragmática evaluada en los niños. Conclusiones: Consideramos que el EDUTEA puede ser una herramienta útil para la detección de TEA en las escuelas.
Neurodevelopmental disorders include several conditions that affect the child's ability to socially interact and communicate with others (American Psychiatric Association, APA, 2013). Autism spectrum disorder (ASD) is characterized by a dyad of impairments in social communication/interaction, and restricted or repetitive patterns of behaviour and interests (Mandy, Charman, & Skuse, 2012; Shuster, Perry, Bebko, & Toplak, 2014). Meanwhile, those children that have problems with the pragmatic aspects of social communication but do not have restricted behaviour patterns are now included in the social pragmatic communication disorder (SCD), whereas previously they were diagnosed as having non-specified pervasive developmental disorders. As stated by Romero et al. (2016), this change has contributed to find more severe profiles of nuclear and comorbid symptoms in children with a diagnosis of ASD. Recent international studies have estimated ASD prevalence to be around 1.2-1.5 among children (Christensen et al., 2016; Saemundsen, Magnusson, Georgsdóttir, Egilsson, & Rafnsson, 2013; Sun et al., 2015). In contrast, few studies have explored SCD prevalence, which has been estimated to be around 0.5% (Kim et al., 2014).
Early detection and diagnosis is essential if child intervention is to be properly addressed and guidance provided to families and schools. Both child intervention and family guidance can improve child developmental outcomes and parental well-being (Zwaigenbaum et al., 2015). The screening process is an important step in the diagnostic procedure and family and school play an important role. The most widely used ASD screening questionnaires in Europe are addressed to parents or caregivers: the Social Communication Questionnaire (SCQ; Rutter, Bailey, & Lord, 2003), the Checklist for Autism in Toddlers (Baron-Cohen et al., 2000) or the Modified Checklist for Autism in Toddlers (M-CHAT; Robins, Fein, Barton, & Green, 2001). In contrast, other questionnaires such as the Social Responsiveness Scale (SRS; Constantino, 2005) and the Childhood Autism Rating Scale (CARS; Schopler, Reichler, & Renner, 1988) are addressed to clinicians, parents and also teachers (Ashwood, Buitelaar, Murphy, Spooren, & Charman, 2015). As far as psychometric properties are concerned, Charman et al. (2015) found that the M-CHAT showed a sensitivity of 82% and a specificity of 50%, with an area under the curve (AUC) of the ROC curve of 66% in 18 to 48-month-old children referred to clinical services. Likewise, SCQ showed a sensitivity of 64% and a specificity of 75%, with an AUC of 70% in the same population. In contrast, previous validity studies in a sample of 18 to 70-month-old children described a sensitivity of 70% and a specificity of 38% for the CAST and 70% and 52% for the SCQ, respectively (Snow & Lecavalier, 2008). In 9 to 10-year-old children, Chandler et al. (2007) found a sensitivity of 88% and a specificity of 72% for the SCQ. And in a sample of 4 to 18-year-old children with ASD and their unaffected siblings, the current teacher version of SCQ reached a sensitivity of 60% and a specificity of 95% and the SRS reached values of 69% and 95%, respectively (Schanding, Nowell, & Goin-Kochel, 2012).
On the other hand, pragmatic language competencies are often directly assessed with the children themselves but some questionnaires are also designed for parents or teachers (Bishop, 2006; Russel & Grizzle, 2008). However, they were designed before the DSM-5 defined social communication disorder. As far as psychometric properties are concerned, the Children's Communication Checklist was adapted for a Spanish population (Crespo-Eguílaz, Magallón, Sánchez-Carpintero, & Narbona, 2016). Results indicated a sensitivity of 91% and a specificity of 100% for the detection of pragmatic difficulties, and provided information to discriminate co-occurent pragmatic deficits in children with other neurodevelopmental disorders, such as language disorder or Attention Deficit Hyperactivity Disorder (ADHD).
In fact, social communication and pragmatic skills are impaired in many children with ADHD (Green, Johnson, & Bretherton, 2014; Ronald, Larsson, Anckarsäter, & Licnhtenstein, 2014). At the same time, around 40-86% of the children with ASD exhibit clinically elevated ADHD symptoms(Joshi et al., 2014; Kaat, Gadow& Lecavalier, 2013; Mansour, Dovi, Lane, Loveland, &Pearson, 2017) and, in some cases, ADHD may overshadow ASD features and delay diagnosis (Miodovnik, Harstad, Sideridis, & Huntington, 2015).
Teachers’ reports of child social communication and restricted behaviour patterns are often useful and reliable because they make it possible to compare and follow up children's development within their normative peer group from a professional point of view (Morales, Domènech-Llaberia, Jané, & Canals, 2013). However, in our context, one of the main obstacles mentioned by teachers and educational psychologists is the lack of time. Given this situation, there is a need to develop a brief screening tool designed for teachers so that school professionals can make appropriate referrals and clinicians or researchers can collect reliable information in school settings, and supplement the information provided by the family.
We want to be able to discriminate whether the social problems reported in children are related to an underlying autism condition or to a social communication disorder, which is commonly undetected. Likewise, we would like to have a tool to help improve the rate of detection of ASD in Spain, which is still far from that reported in other countries (Aguilera, Moreno, & Rodríguez, 2007; Belinchón, 2001; Frontera, 2005; Sevilla, Bermúdez,& Sánchez, 2013). Therefore, the aim of this study was to design and evaluate the psychometric proprieties of EDUTEA, a questionnaire based on the DSM-5 criterion for screening ASD and SCD in school settings. Reliability, factor validity and convergent validity have been explored and cut-off scores have been established. We will also assess the ability of the EDUTEA to discriminate between ASD/SCD and ADHD and provide standardized scores in Spanish school population.
MethodStudy designAccording to the classification of Montero and León (2007), this was a descriptive survey study. This study was part of the Neurodevelopmental Disorders Epidemiological Research Project (EPINED), a double-phase cross sectional study aimed to determine the prevalence of ASD, SDC and ADHD in Tarragona, Spain (Figure 1). The study was approved by the Ethics Committee at the Sant Joan University Hospital.
Participants
The total sample consisted of 2,660 children. In the first phase, 1,301 children between the ages of 10-12 (Primary school) and 1,359 children between the ages of 3-5 (Nursery school) participated in the study. The participation of the families was low (49.7%), but we were given permission by the Department of Education of the Catalan Government to collect anonymous data from teachers about the children of non-participating families, so that we had data about the whole sample. In the second phase, 291 children were assessed individually. Of these, 63 were at risk of ASD/SCD, 79 were at risk of ADHD, 40 were at risk of both disorders and 109 were controls. Controls were randomly selected and paired by age, gender, ethnicity and school. The time lapse between the first and the second phase was an average of 2 months.
Questionnaire designThe EDUTEA is an 11-item questionnaire for the teachers about children's current communication, social interaction and behaviors. Items are based on DSM-5 criteria; the first seven items gather information about ASD criteria and the remaining four about SCD. The questionnaire provides an overall score based on a 4-point Likert scale (0=“never or almost never”, 1=“sometimes”, 2=“often”, and 3=“always or almost always”), resulting in a minimum score of 0 and a maximum of 33, which means full endorsement of all ASD/SCD items. The EDUTEA can be found at the following link https://psico.fcep.urv.cat/Q4/EduTEA/.
The items were written by mental health professionals with expertise in neurodevelopmental disorders, following the wording of the DSM-5. Each item contains a header (e.g. “Difficulties in the use of nonverbal communication during social interaction”) and a set of statements all together (e.g. “Poor eye contact; does not use gestures when speaking or uses them in a peculiar way; poor facial expression; does not react to gestures and expressions in others or has difficulty understanding them”). Respondents have to indicate how often the child has difficulties in any of the statements. The instructions specify that if the child's behavior fits any of the descriptions provided, it is enough to assess the presence of this problem.
We sent the first version of the questionnaire in Catalan language to five school teachers in order to assess their understanding of the terms used. In qualitative terms, they all indicated that the items were easily understandable and applicable to their students. Then, a pilot study was conducted with 276 participants to check the performance of the EDUTEA and evaluate teacher responses to potential problems. All the questionnaires were answered without leaving any blank item, except in the case of a student with nonverbal autism. Because of this, we decided to add a specific instruction in the header of the questionnaire to address these sorts of cases. Teachers reported no other concerns about the questionnaire. In this step, preliminary cut-off scores were set at percentile 95 for ASD and SCD risk.
Other measuresThe Childhood Asperger Syndrome Test (CAST; Scott, Baron-Cohen, Bolton, & Brayne, 2002) is a 37-item parent-report questionnaire for screening ASD manifestations and social communication problems in 4 to 11 year old children. The cut-off point for detecting at-risk children is 15, which had a sensitivity of 100% and a specificity of 97% for detecting ASD (Williams, 2005). The internal reliability of the questionnaire in the Spanish population was .83 (Morales-Hidalgo, Roigé-Castellví, Vigil-Colet, & Canals Sans, 2017).
The parents also answered the Child Behavior Checklist to assess psychopathological symptoms among preschoolers (CBCL1½-5; Achenbach & Rescorla, 2000) and school-aged children (CBCL6-18, Achenbach & Rescorla, 2001). The Youth Self Report version (YSR; Achenbach, 1991) was administered to school aged children. We specifically used the DSM-5 ASD scale among pre-schoolers and the social problems scale in school-aged children.
The Conners indexes parents’ and teachers’ forms were used to assess ADHD risk symptoms. The Conners Early Childhood Global Index (Conners, 2009) is a 10-item scale that assesses the presence of general psychopathology over the previous month in children between 2 and 6 years old. Items are divided into two subscales that assess symptoms related to ADHD: restless-impulsive and emotional lability. The Conners 3 ADHD Index (Conners, 2008) is a 10-item scale that assesses the presence of the most prominent ADHD symptoms over the previous month in children between 6 and 18 years old. The scales have a cut-off for elevated scores (T65-69) and for very elevated scores (T≥70), which indicate more and many more concerns of ADHD risk. High internal reliability (.92-.98) was found in Spanish population (Morales-Hidalgo, Hernández-Martínez, Vera, Voltas, & Canals, 2017).
The Autism Diagnostic Interview Revised (ADI-R; Rutter, Le Couteur, & Lord, 2003) and the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2; Lord et al., 2012) were used by specifically trained clinicians to collect information about the developmental areas commonly impaired in ASD–language and communication, reciprocal social interactions, and repetitive behaviors and interests. For diagnostic purposes, we used the ADOS-2 calibrated severity score (1-3 nonspectrum classification; 4-5 ASD-classification; and 6-10 autism classification) described by Gotham, Pickles, and Lord (2009), and the cut-off scores of the ADI-R diagnostic algorithm, based on the child's behavior at 4-5 years old.
The pragmatic subtest of the Objective Language Criteria Test–Screening Revised (BLOC-S-R; Puyuelo, Renom, Solanas, & Wiig, 2007) was administered to estimate the pragmatic competence of the school-aged children. The test provides percentiles, which are distributed in the following levels of children's competence and need for help: 70-100 (upper level), 60-70 (transition level), 30-60 (emergence level) and 25-30 (alarm level).
The Wechsler Scales of Intelligence for preschool and school-aged children (WPPSI-IV, Wechsler Preschool and Primary Scales of Intelligence; WISC-IV, Wechsler Intelligence Scales for Children; Wechsler, 2003, 2012) were used to assess verbal comprehension, perceptual reasoning, working memory and processing speed, and to estimate the global intelligence quotient (IQ).
The Schedule for Affective Disorders and Schizophrenia (K-SADS-PL; Kaufman et al., 1997) is a semi-structured psychiatric interview that yields DSM diagnoses and has been widely used in studies of child psychopathology. We used the K-SADS-PL to collect information on child ADHD symptomatology. The Spanish version has shown a high inter-rater reliability (K=.91) for the ADHD scale (Ulloa et al., 2006).
A socio-demographic questionnaire addressed to parents and teacher was developed ad-hoc in order to gather information from the children and their families. Socioeconomic level was estimated according to Hollingshead index (Hollinsghead, 2011).
ProcedureIn the first phase of the study the teachers answered the EDUTEA, Conners and provided general socio-demographic data. The teachers must have known the children for at least three months; if they had not, the questionnaires were answered by the previous year's teacher. Parents also provided socio-demographic data and filled in the CAST and Conners. We considered children to be at risk of these disorders if they scored above the cut-off point on either EDUTEA (Pc ≥ 95; established in the preliminary study) or CAST (total score ≥15; Morales-Hidalgo, Roigé-Castellví et al., 2017; Williams, 2005). In regard to ADHD screening, we considered children to be at risk of ADHD if they scored above T65 on both parent and teacher reports. The “and” criterion that takes into account information from parents and teachers was chosen to avoid false positives, since parents often report risk symptoms in a high percentage of children (Bied, Biederman, & Faraone, 2017; Morales-Hidalgo, Roigé-Castellví et al., 2017).
In the second phase, all the children who met criteria for risk symptoms and a control group without risk scores (matched for gender, age and school) were contacted and individually assessed. A clinical diagnosis was performed using DSM-5 criteria and an individual neuropsychological and psychopathological assessment of the children and their families. The parents of subjects at risk and a subsample without risk answered the CBCL, and they were administered the ADI-R and the K-SADS. The ADOS-2, the pragmatic subtest of the BLOC, WPPSI-IV/WISC-IV and the YSR was administered to children. At the end of the study, families received a complete report of the results. The inter-rater reliability was assessed several times throughout the research to ensure the reliability of the data. At least once in each school, two clinicians performed a complete assessment in which one was the “examiner” and the other was non-participant observer. The interviews were then scored separately. Scores and diagnostic orientation were compared in qualitative terms (that is, whether they complied or not with diagnostic criteria or obtained clinical or subclinical scores). This procedure was also conducted in those cases in which the test results were inconsistent. We also made a joint analysis of those cases in which information from the various participants was inconsistent. The inter-rater agreement was very high in all the assessments compared.
Case ascertainmentFor a positive diagnosis of ASD, the ADOS-2 severity score had to be above four and all domains in the ADI-R (communication, social interaction, restrictive, and repetitive behaviour and onset) had to be above the cut-off points. For a positive diagnosis of SCD, child communication must be at the “emergence level” or “alarm level” in the BLOC pragmatic subtest. Moreover, communication or social interaction in the ADI-R had to be above the cut-off point and there had to be evidence of communication difficulties in the ADOS-2, which means a severity score at least between three and four. Ultimately, for a diagnosis of ADHD, the K-SADS-PL criteria had to be positive.
The information provided by the parents during the interview and the results of the tests performed with children enabled other neurodevelopmental disorders to be diagnosed, such as obsessive compulsive disorder (OCD), intellectual disability or tic disorders. In the case of OCD and tic disorders, comprehensive information was collected using the K-SADS-PL. Intellectual disability was diagnosed on the basis of parental report about adaptive skills and the cognitive profile. Specific learning disorders were not assessed in our study. However, those cases confirmed by the teams of educational psychologists were taken into account.
Statistical analysesStatistical analyses were performed by IBM SPSS 23 and factorial analyses by FACTOR 10.3.01 (Ferrando & Lorenzo-Seva, 2013). Descriptive statistics were used to provide the average score for the various age groups and genders. T tests for independent samples were performed to compare the scores from boys and girls. Pearson correlations between EDUTEA and other tests were conducted to provide convergent validity. The reliability (internal consistency) of the overall scales and factors was assessed using the reliability estimates of the factor scores, which take into account the ordinal nature of the item responses. To assess the reliability of the overall scale, the first canonical factor was extracted (Ferrando & Lorenzo-Seva, 2013) and its reliability was estimated. In order to estimate the EDUTEA cut-off point, we computed ROC curves, sensitivity and specificity.
To determine the factor structure of the EDUTEA, Factor Analysis was used (FA). Exploratory Factor Analysis (EFA) was conducted on one randomly selected half of the sample. Semi-Confirmatory Factor Analysis (S-CFA) was performed on the other half using a semi-specified target matrix. Finally, the same S-CFA was replicated with the whole sample. For the whole FA, we used an optimal implementation of Parallel Analysis (Timmerman & Lorenzo-Seva, 2011), and an unweighted least squares’ extraction method and, to obtain an oblique rotated solution, we used the direct oblimin procedure (Browne, 1972). Polychoric correlation dispersion matrixs were used due the ordinal nature of the data and several items showed absolute skewness or kurtosis values greater than one in absolute values. For the final solution we considered the following statistical indexes: the Goodness of Fit Index (GFI; Bentler & Bonett, 1980) should be equal to or greater than .90, the Kaiser-Meyer-Olkin test (KMO; Harman, 1962) the value of which should be more than or equal to.50, the Root Mean Square of the Residuals (RMSR; Harman, 1962) which should be smaller than or equal to .08, and Kelley's criterion (Harman, 1962; Kelley, 1935) which must be smaller than the RMSR. The congruence indexes were also reported. As Lorenzo-Seva and ten Berge (2006) suggest, congruence values in the range .85-.94 suggest that the factors show a fair similarity and values higher than .95 imply that the two compared factors (or components) compared can be considered equal.
ResultsThe top panel in Table 1 shows the socio-demographic and psychopathological data of the participating and non-participating families in the first phase of the study. The bottom panel shows the same data for the three main diagnostic groups (ASD/SCD, ADHD and controls) in the second phase of the study. No significant differences in socio-demographic variables were found between participants and non-participants or between the first- and second-phase samples. In the second phase, a total of 20 children in the EPINED met criteria for ASD, half of whom had been previously diagnosed by clinical services. These children obtained a calibrated severity score of 5.8 in the ADOS-2. Only three children were diagnosed with SCD, all of whom had previously been diagnosed with speech delay but not with any communication disorder. Of these children with ASD/SCD, 6 presented a co-occurring ADHD diagnosis. The individual assessment also resulted in 75 children with a diagnosis of ADHD, 116 controls and 77 children with a diagnosis of subclinical ASD/ADHD and various other neurodevelopmental disorders such as obsessive compulsive disorder, intellectual disability, tic disorders and specific learning disorders.
Sample characteristics.
| FIRST PHASE (N=2,660) | ||||||
|---|---|---|---|---|---|---|
| Teacher recruitment | Parent recruitment | t/χ2(gl) | ||||
| Participants’ characteristics | NE | PE | NE | PE | NE | PE |
| Participants, n (%) | 1,359 (100) | 1,301 (100) | 667 (49.08) | 656 (50.42) | – | – |
| Age, years, M (SD) | 4.80 (.46) | 10.82 (.49) | 4.78 (0.46) | 10.81 (0.48) | 2.86 (1355) | .19 (1291) |
| Gender, male, n (%) | 678 (49.89) | 640 (49.19) | 323 (48.43) | 297 (45.27) | 1.12 (1) | 8.13 (1) |
| Ethnicity, Caucasian n (%) | 952 (70.05) | 1029 (79.09) | 520 (77.96) | 527 (80.34) | 3.04 (1) | .306 (1) |
| EDUTEA score, M (SD) | 2.61 (4.55) | 3.13 (5.12) | 2.26 (4.29) | 2.71 (4.82) | 2.77 (1357) | 2.93 (1278.87) |
| CAST score, M (SD) | – | – | 7.67 (3.97) | 7.35 (3.63) | – | – |
| SECOND PHASE (N=214) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ASD/SCDa (N=23) | ADHDb (N=75) | Controlsc (N=116) | NE | PE | ||||||||
| Participants’ characteristics | NE | PE | NE | PE | NE | PE | F | pab | pac | F | pab | pac |
| Participants, n (%) | 14 (2.10) | 9 (1.37) | 16 (2.40) | 59 (8.99) | 55 (8.25) | 61 (9.30) | – | – | – | – | – | – |
| Age, years, M (SD) | 4.91 (0.35) | 11.10 (0.53) | 4.88 (0.43) | 10.90 (0.44) | 4.76 (0.41) | 10.84 (0.43) | 1.06 | .999 | .710 | 1.35 | .628 | .325 |
| Gender, male, n (%) | 12 (85.71) | 8 (88.89) | 12 (75.00) | 42 (71.19) | 35 (63.64) | 34 (55.74) | 1.42 | .850 | .341 | 2.84 | .168 | .154 |
| Ethnicity, Caucasian n (%) | 12 (85.71) | 7 (77.78) | 14 (87.50) | 46 (77.97) | 38 (69.09) | 52 (85.25) | 2.06 | .999 | .366 | 0.91 | .999 | .999 |
| EDUTEA score, M (SD) | 16.36 (6.65) | 14.56 (9.94) | 7.88 (5.56) | 7.68 (7.51) | 3.69 (6.09) | 2.66 (4.59) | 24.58 | .001 | .001 | 17.75 | .010 | .001 |
| CAST score, M (SD) | 17.00 (5.25) | 14.56 (5.88) | 12.44 (3.85) | 10.49 (4.15) | 7.71 (4.37) | 6.92 (3.22) | 27.22 | .018 | .001 | 22.38 | .012 | .012 |
| BLOC-SR, M (SD)* | – | 34.00 (28.70) | – | 59.21 (27.70) | – | 62.54 (24.15) | – | – | – | 3.62 | .097 | .028 |
| Total Wechsler IQ, M (SD) | 85.18 (21.11) | 93.50 (20.03) | 84.69 (14.23) | 95.14 (16.09) | 101.80 (14.32) | 102.85 (15.06) | 11.02 | .999 | .005 | 4.01 | .999 | .358 |
| CBCL, M (SD) | ||||||||||||
| Internalizing problems | 70.79 (10.03) | 63.67 (9.82) | 63.81 (9.53) | 62.25 (9.82) | 56.34 (10.88) | 53.07 (9.53) | 11.73 | .221 | .001 | 14.39 | .999 | .008 |
| Externalizing problems | 67.64 (10.77) | 51.56 (13.31) | 72.06 (9.73) | 61.21 (10.53) | 54.09 (10.15) | 49.30 (9.31) | 24.41 | .717 | .001 | 20.00 | .028 | .999 |
| Total behavior problems | 71.93 (11.40) | 61.67 (10.02) | 70.19 (10.34) | 64.44 (9.32) | 56.75 (10.78) | 50.84 (9.34) | 17.00 | .999 | .001 | 30.72 | .999 | .005 |
| ADI-R, M (SD) | ||||||||||||
| Social interaction | 12.00 (4.19) | 12.11 (9.32) | 2.44 (2.10) | 3.19 (3.20) | 1.96 (1.90) | 1.93 (2.27) | 97.13 | .001 | .001 | 31.98 | .001 | .001 |
| Social communication | 9.64 (3.59) | 10.22 (5.09) | 2.63 (1.50) | 2.50 (2.23) | 1.40 (1.56) | 1.38 (1.58) | 93.91 | .001 | .001 | 59.89 | .001 | .001 |
| Restrictive patterns of behavior and interests | 4.57 (2.21) | 2.67 (1.32) | 1.13 (0.81) | 1.00 (1.09) | 0.78 (1.03) | 0.51 (0.70) | 51.07 | .001 | .001 | 21.35 | .001 | .001 |
| ADOS-2 severity score, M (SD) | 5.86 (1.23) | 4.44 (1.67) | 1.06 (.44) | 1.49 (1.36) | 1.45 (.92) | 1.21 (.61) | 142.75 | .001 | .001 | 34.37 | .001 | .001 |
Note. Mean (M) and standard deviation (SD) or percentage (n; %). NE: Nursery Education (4-5 years old), PE: Primary Education (11-12 years old); p≤.01.
Children with ASD/SCD diagnoses scored significantly higher than the ADHD and control groups on the EDUTEA, ADI-R and ADOS-2 severity score. In comparison with the control group, but not with the ADHD group, they also had higher scores on most of the CBCL scales. In comparison with the control group, but not with the ADHD group, preschool children with ASD/SCD showed a lower IQ. In general terms, differences between primary education children with ASD/SCD, ADHD and controls were less pronounced.
Boys [Mean (SD): 3.19(5.08)] scored significantly higher than girls [2.02(3.86)] in both nursery education (p≤.001)] and primary education [3.19(5.08) versus 2.02(3.86); p≤.001]. Besides, school-aged children showed significantly higher scores on factor scores (F1: p≤.006; F2:.027) and the total score (p≤.006) than pre-schoolers. As far as the socioeconomic status (SES) of the whole sample is concerned, 20.5% of the children were low SES, 58.5% middle SES and 21.0% high SES. No significant differences were found between participants and non-participants in the first phase and between children with and without diagnoses in the second-phase subgroups.
Factorial structure and internal reliabilityFactor analysis was conducted to test the invariance across age. The results show the same evidence of internal structure in preschool and school-aged children. Global analyses were performed and the results are described below. Exploratory factor analysis (EFA) was conducted using half of the total sample (N=1,330). As the Kaiser-Meyer-Olkin (KMO) index value was .92, we concluded that the correlation matrix was suitable for factor analysis. The root mean square of the residuals (RMSR) was .029 and the Kelley criterion was .027, which suggests that the data had an underlying bi-dimensional factor structure. To confirm the stability of the factor structure, we performed factor analysis on the other half of the sample (cross-validity), and obtained similar statistics (KMO=.90, RMSR=.049, Kelley criterion=.027). We then performed the analysis on the total sample. The sedimentation graphic, which was aimed to identify the number of factors to be extracted, highlighted the presence of two factors.
We obtained two factors that explained 81% of the variance (see table 2). These factors were labelled Social communication impairments (Factor 1) and Restricted behaviour patterns (Factor 2). The first factor contained seven items that referred to the social (pragmatic) communication and social interaction impairments (ASD+SCD). The second factor contained four items that referred to restricted or repetitive patterns of behavior and aspects of hyper- or hypo reactivity to sensory inputs or unusual interests in sensory aspects (ASD). In general, factor loadings ranged from .49 to .99. Items with loadings greater than .30 on two factors were included in the factor with higher clinical congruence and loading. The internal reliability of the overall scale was .97.
Factor structure of the EDUTEA.
| Factor 1a | Factor 2b | |
|---|---|---|
| Deficits in social-emotional reciprocity | .76 | .17 |
| Deficits in nonverbal communicative behaviors used for social interaction | .79 | .10 |
| Deficits in developing, maintaining, and understanding relationships | .55 | .37 |
| Stereotyped or repetitive motor movements, use of objects, or speech | .07 | .84 |
| Insistence on sameness, adherence to routines, ritualized patterns behavior | .13 | .71 |
| Highly restricted, fixated interests that are abnormal in intensity or focus | −.08 | .97 |
| Hyper- or hypo reactivity to sensory input or unusual interests in sensory aspects | .22 | .64 |
| Deficits in using communication for social purposes | .49 | .40 |
| Impairment of the ability to change communication to match context | .56 | .31 |
| Difficulties following rules for conversation and storytelling | .94 | −.04 |
| Difficulties understanding nonliteral or ambiguous meanings of language | .99 | −.12 |
| Explained variance (%) | .74 | .06 |
| Factor reliability | .95 | .93 |
| Matrix statistics | ||
| Bartlett's statistic | 19249.6 (df=55; p=.001) | |
| Kaiser-Meyer-Olkin (KMO) test | .91 | |
| Residual statistics | ||
| Goodness of Fit Index (GFI) | 1.00 | |
| Root Mean Square for Residuals (RMSR) | .038 | |
| Kelley's criterion | .019 | |
The convergent validity was evaluated between the factor and total scores of the EDUTEA and several tests and questionnaires, such as ADI-R, ADOS-2, CAST, BLOC and pragmatic subtests (Table 4).
Convergent validity of the EDUTEA.
| EDUTEA | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Social communication impairments | Restricted behaviour patterns | Total | |||||||
| NE | PE | Total | NE | PE | Total | NE | PE | Total | |
| ADI-R | |||||||||
| Social interaction and communication | .40*** | .28*** | .34*** | .43*** | .36*** | .39*** | .44*** | .32*** | .38*** |
| Restricted and repetitive behaviours | .39*** | .04 | .22*** | .37*** | .06 | .21*** | .41*** | .05 | .23*** |
| ADOS-2 | |||||||||
| Social interaction and communication | .45*** | .24*** | .35*** | .41*** | .26*** | .33*** | .47*** | .26*** | .36*** |
| Restricted and repetitive behaviours | .58*** | .27*** | .44*** | .48*** | .43*** | .43*** | .60*** | .34*** | .47*** |
| Severity score | .47*** | .32*** | .40*** | .46*** | .36*** | .41*** | .51*** | .35*** | .43*** |
| Achenbach scales | |||||||||
| CBCL (1.5-5) ASD DSM scale | .32*** | – | .32*** | .25** | – | .25** | .33*** | – | .33*** |
| CBCL (6-18) Social problems | – | .21** | .21** | – | .16* | .16* | – | .21** | .21** |
| YSR (11-18) Social problems | – | .19** | .19** | – | .08 | .08 | – | .17* | .17* |
| BLOC pragmatic subtest | – | −.40*** | −.40*** | – | −.30*** | −.30*** | – | −.39*** | −.39*** |
| CAST | .42*** | .28*** | .36*** | .32*** | .24*** | .29*** | .42*** | .28*** | .36*** |
Note. NE: Nursery Education (4-5 years old), PE: Primary Education (11-12 years old). Significance levels
In general, correlations were low to medium. Correlations were highest between the EDUTEA and the ADOS-2, especially for the repetitive behaviors scale(r=.47; p=.001) and the severity score (r=.43; p=.001). The child pragmatic competence showed a consistent correlation with the EDUTEA social communication impairments factor (r=-.40; p=.001). Correlations were lowest between EDUTEA and the CBCL social problems scale in both parent reports (r=.21; p=.006) and self-reports (r=.17; p=.024) from school-aged children. In contrast, results were slightly higher for the preschool-aged children on the CBCL ASD scale (r=.33; p=.001) and also ADI-R and CAST. As far as age is concerned, correlations were higher for the nursery education group in all factors and total scores.
Sensitivity and specificity of the cut-offResults for the sensitivity and specificity of the EDUTEA cut-off score for the total score are shown in Table 3. We propose 10 to be the best cut-off score. It gave a sensitivity of 87.0% and a specificity of 91.2% with a Positive Predictive Value of .86 and a Negative Predictive Value of .99 for the detection of ASD and SCD risk symptoms. Figure 2 shows the ROC curve. Taking into account the total score, the area under the curve (AUC) of the ROC curve was .896 (95% Confidence Interval=.81–.98; p=.001).
Sensitivity and specificity of the EDUTEA cut-off for total score and percentiles of the total and factor scores.
| PC | EDUTEA | ||||
|---|---|---|---|---|---|
| Social communication impairments | Restricted behaviour patterns | Total | Total score cut-off Sensitivity | Total score cut-off Specificity | |
| 99 | 17 | 7 | 23 | .13 | .99 |
| 98 | 15 | 6 | 19 | .39 | .98 |
| 97 | 13 | 5 | 17 | .47 | .97 |
| 96 | 12 | 4 | 15 | .56 | .96 |
| 95 | 11 | 4 | 13 | .69 | .94 |
| 94 | 10 | 3 | 12 | .73 | .93 |
| 93 | 9 | 3 | 12 | ||
| 92 | 8 | 3 | 10 | .87 | .91 |
| 91 | 8 | 2 | 10 | ||
| 90 | 7 | 2 | 9 | .87 | .89 |
| 85 | 5 | 1 | 7 | .87 | .85 |
| 80 | 4 | 1 | 5 | .87 | .78 |
| 75 | 3 | 0 | 4 | .87 | .74 |
| 70 | 2 | 0 | 3 | .91 | .68 |
| 65 | 2 | 0 | 2 | .91 | .60 |
| 60 | 1 | 0 | 2 | ||
| 55 | 1 | 0 | 1 | .91 | .51 |
| 50 | 0 | 0 | 0 | 1.0 | 1.0 |
According to this cut-off point, around 9.7% of the participants were at risk of ASD and SCD, with boys having higher prevalence rates than girls (12.9 versus 6.1%; p=.001). There was a slight tendency to find more at-risk subjects in primary education than in nursery education (10.6 versus 9.4%; p=.006).
The ability of the EDUTEA to discriminate between ASD/SCD and ADHD was assessed by ROC curve analysis. The cut-off score of 10 gave a sensitivity of 83% and a specificity of 73%, and the AUC was .78 (95% Confidence Interval=.66–.89; p=.001).
Furthermore, we provide the percentiles for each factor (Table 3) so that clinicians can place each of the scores obtained by the subject in relation to the reference group.
DiscussionThe EDUTEA showed a two-factor structure with high internal reliability: Social communication impairments (.95) and Restricted behaviour patterns (.93). Despite being so brief, the two-factor solution explained a greater proportion of variance than the unifactorial solution (81% versus 74%) and had greater clinical coherence. This structure is consistent with other models of ASD symptoms that separate both domains (Mandy et al., 2012; Shuster et al., 2014) and also includes social pragmatic communication difficulties. Thus, the instrument differentiates between those children with high scores on both factors, who are at risk of ASD, and those who only show high scores on the social communication factor, who are at risk of SCD.
The EDUTEA questionnaire showed an AUC of the ROC curve (.89) that was highly predictive of ASD/SCD diagnoses. We propose to use a single cut-off score of 10 (Pc 91.5), which allowed us to identify 20 of the 23 children later diagnosed with ASD and was capable of discriminating between ASD and ADHD (AUC=.78). Additionally, the severity of the teacher ratings on each factor could be described through percentiles, if necessary. Specific cut-off scores for each factor were not provided because of the shortness of the questionnaire and because of the fact that it was mainly designed for screening purposes.
In regard to convergent validity we found moderate correlations between the EDUTEA and the ADOS-2 severity score and the BLOC-SR pragmatic subtest. These findings suggest that EDUTEA was appropriately measuring restrictive patterns of behaviour, social interaction and communication difficulties. On the other hand, correlations with ADI-R were slightly lower perhaps because the scores were obtained through the diagnostic algorithm (4-5 years old), which did not coincide with the current age of the entire sample. Another reason could be that the information had come from a different (teacher versus parents) source (Kaat et al., 2013; Rescorla et al., 2014; Stratis & Lecavalier, 2015). Likewise, correlations with the CBCL scales were lower. One possible explanation for this is that these scales have not proved to be useful for ASD-specific screening, specifically in schoolchildren (Havdahl, von Tetzchner, Huerta, Lord, & Bishop, 2016).
The level of severity of ASD symptoms was moderate in most of the children with ASD diagnosis, with mean scores between ASD and autism on the classification by Gotham et al. (2009). Very few cases of severe ASD were diagnosed since the sample was obtained from a community sample of regular schools. Five children showed co-occurring intellectual disability, which could explain the difference in the average IQ between the ASD group and the other groups. The ASD group also showed significantly higher scores on the psychopathological measures than the control group, which supports data on psychopathological comorbidity in ASD (Höglund-Carlsson et al., 2013; Salazar et al., 2015). Likewise, children with ASD showed a significantly lower pragmatic competence. As far as sex differences are concerned, we found that boys had higher EDUTEA scores than girls, which is consistent with ASD screening questionnaires (Sun et al., 2014). Likewise, the prevalence of risk symptoms in boys was higher than in girls, which is in agreement with the higher prevalence of ASD and SCD between boys (Christensen et al., 2016; Kim et al., 2014).
This article presents the design and validation of a brief questionnaire for screening ASD/SCD in school settings. This is important because teachers have proven to be good informers about social communication and symptoms of restrictive or repetitive behavior (Morales et al., 2013) and because we still have many undiagnosed cases of ASD (for example, 50% of the ASD diagnoses in our sample had previously gone undetected).Taking into account all the results, the main strength of the study is the sample size and the fact that ASD was diagnosed on the basis of the results of gold standard tools (ADI-R and ADOS-2) and DSM-5 criteria in consensus between two clinicians. However, the study also has some limitations and several issues need to be addressed in the future. Although we have all the teachers’ reports on the EDUTEA, only around 50% of the families participated in the second phase so we could not individually assess all the subjects at risk. In this regard, participation was strongly influenced by the extent to which families were involved in school activities and each centre was involved with the project. Unfortunately, due to constraints of time and logistics, the cognitive performance of non-verbal children with autism could not be assessed. In the present study, very specific age groups were studied. The rationale for focusing on these two age groups was that the nuclear symptoms of ASD can be easily detected at school settings when children are between 3 and 5 years old, and that the disorder is assumed to be present and causing interferences (including academic problems) when the subjects are between 10 and 12 years old. We believe that future studies should be performed in a broader age range. Children from 3 to 12 years old should be screened and the performance of adolescents should also be assessed. We believe that specific analyses will have to be conducted to determine if a different cut-off point is needed for children with SCD. The performance of girls on the EDUTEA should be specifically addressed.
In conclusion, EDUTEA has proved to be a reliable questionnaire for screening the risk of ASD and SCD in the Spanish population in school settings. Because of its brevity and great predictive value, it could be useful to include the EDUTEA in future school ASD screening protocols.
This research was supported by the Ministry of Economy and Competitiveness of Spain and the European Regional Development Fund (ERDF; ref. PSI2015-64837-P). It was also supported by the Ministry of Education (grant FPU2013-01245). We acknowledge the support of the Catalan Department of Education. We are indebted to all participating teachers and families involved in the EPINED project.
