Therefore, following on from findings from the literature (Bioulac et al., 2012) which show that neuropsychological testing using virtual reality had better ecological validity than traditional CPTs, the aim of the current study was to compare the discriminant value of attentional variables provided by a traditional CPT test (TOVA; Greenberg, 1993) with those from a virtual reality test (Aula Nesplora; Climent et al., 2011. For this purpose, two different samples of participants were used, who were randomly assigned to the different assessment tools (traditional CPT and virtual CPT) so that the groups were equivalent in terms of age and gender. In addition, considering the causal heterogeneity in ADHD (Nigg, Willcutt, Doyle, & Sonuga-Barke, 2005) each group (the group assessed by the virtual reality test and the group assessed by the traditional CPT) was split into four groups: a Control group and three ADHD groups (corresponding to each of the three ADHD presentations).

Executive measures using Virtual Reality. Aula Nesplora (Climent et al., 2011) is a CPT based on a VR environment that reproduces the conditions of a classroom. Aula Nesplora evaluates attention, impulsivity, processing speed, and motor activity in children and adolescents between 6 and 16 years of age. The virtual environment is shown through 3D glasses (Head Mounted Display, HMD). Motion sensors and headphones are also included in order to make the task as realistic as possible. The participant takes the perspective of a student sitting at one of the desks looking at the blackboard. Head movements are registered by sensors located in the glasses and the software updates the angle of vision, giving the subject the feeling of actually being in a virtual classroom. The test consists of two tasks that are gradually explained by a virtual teacher. The first phase task is based on the “x-no” paradigm (traditionally known as “no-go”) in which the participant must press a button when they see (in the virtual blackboard presented by 3D glasses) or hear (through headphones) a stimulus, only if the stimulus is not “apple”. In the second phase, an “x” paradigm (or “go” task) is incorporated, with participants being asked to press a button whenever they see or hear the number “seven.” The variables provided by the instrument are no different from those of other CPTs, however, it completes this information, providing different measures by sensory modality (visual vs. auditory), presence/absence of distractors, and task type (go vs. no-go). The distribution of the distractors is different depending on the type of task (“no-go”, or “go”). During the first (no-go) task there are 9 distractors (two visual, three auditory, and four combined visual and auditory distractors). However, in the second (go) task there are 7 distractors (two visual, three auditory and one combined). In the present study the following measures were examined: omissions, response time, commissions, and variability according to task type, which represents the performance in the two halves of the test. Cronbach alpha for the test was .72.

Executive measures using traditional CPT. TOVA (Greenberg, 1993) is a CPT which provides objective measures of attention and inhibitory control normed by gender for ages 4 to 80. This test presents two simple images. The first shows the stimulus at the top of the screen and the second at the bottom. The participant is given a push-button that should only be pressed when the first image appears. The stimuli are presented for 100 milliseconds (ms) at a between-stimuli interval of 2,000 ms. The test lasts approximately 22.5 min with a 3-minute training session before testing. The TOVA incorporates both a ‘rare target’ and ‘response inhibition’ paradigm by presenting 22.5% targets and 77.5% non-targets in the first half of the test, and then 77.5% of targets and 22.5% of non-targets in the second half of the test. Once the test is finished, TOVA provides the following indicators: omissions (number of missed targets), response time (mean response latency), commissions (responses to non-targets), and variability (standard deviation of response times) and gives the performance in each variable splitting the scores of the test into four quartiles and two halves. It also indicates the number of multiple responses (number of stimuli to which the participant responded more than once) and anticipatory responses (very short latency responses). The TOVA provides an ADHD score using the formula of response time Z-score (Half 1) +D′Z score (half 2) + variability Z-score (total). The ADHD score is a comparison of the participant’s performances and that of a known ADHD sample. Alpha for this test was .82.

The authorities from participating schools and clinical centers signed to confirm their agreement to collaborate in the study. Prior to signing the informed consent for participating in the study, the children’s parents or guardians were informed of the aims of the project, what this experiment involved, and the benefits of taking part (parents would receive a report summarizing the main results from the neuropsychological tests administered). Once informed consents were signed, groups were formed according to the inclusion and exclusion criteria. The study was conducted in accordance with The Code of Ethics of the World Medical Association (Declaration of Helsinki), which reflects the ethical principles for research involving humans (Williams, 2008); approved and according to ethics committee guidelines (CEIC Ethics Committee of the Principality of Asturias; Approval No. CPMP/ICH/135/95. CODE: TDAH).

This study used an ex post facto descriptive–comparative design of four groups, three corresponding to the ADHD presentations, and a control group. Data analyses were conducted in two steps: first, the descriptive statistics for the variables under study were analyzed, paying special attention to skewness and kurtosis. Following the statistical recommendation for Behavioral Sciences (Gravetter & Wallnau, 2014), the maximum scores accepted for skewness and kurtosis were between ±2; second, two discriminant analyses were conducted to determine the specificity and sensitivity levels of each dependent variable (omissions, commissions, response time and variability -based on Virtual Reality vs. the Traditional CPT-) identifying the ADHD group and its presentation. The squared canonical correlation was included as an effect size estimation for the discriminant analyses. Following Cohen (1988), p. 79-80),  r2 between .01 and .09 is small, between .09 and .25 is medium, and above .25 is large.

Table 3 shows descriptive statistics of the dependent variables and results from two MANCOVAs. Only the variables that generated statistically significant differences among the groups were included as covariates (age in the case of Aula Nesplora, and IQ in the case of TOVA) (Table 1).

The first MANCOVA for the dependent variables from Aula Nesplora, taking age, F(8,160) = 1.53, p =  .148, η2 = .074, as a covariate, showed statistically significant differences between the four groups, λ = .506, F(24,429) = 4.93, p <  .001, η2 = .203. It is important to point out that high scores in Aula Nesplora variables are related to poor performance. Taking each variable separately, ANOVAs indicated that all variables from Aula Nesplora generated statistically significant differences between groups.

A second MANCOVA was carried out, using the dependent variables from TOVA and IQ, F(8, 162) = 1.04, p =  .405, η2 = .050, as a covariate. In this case, the results did not reveal statistically significant differences between the four groups, λ = .050, F(24, 465) = 1.18, p =  .249, η2 = .057. Using ANOVAs, only response time from the second half of the TOVA test gave statistically significant differences between the four groups, F(3, 171) = 3.65, p =  .014, η2 = .062. Table 3 shows differences between pairs of groups and post hoc comparisons.

Table 4 shows results from discriminant analyses, which were aimed at comparing the explanatory power of the attentional variables (omissions, commissions, response time and variability) provided by TOVA and those produced by Aula Nesplora. The attentional variables were added to the analysis differentiating the performance of each variable in the two halves of the tests. Function coefficients provide the resulting discriminant function.

Looking at the first model (variables from Aula Nesplora), omissions in the two halves and commissions in the first half of the test were statistically significant predictors of group membership. The squared canonical correlation of the discriminant function was used as an indicator of effect size. Its value was high for omissions ( r2 = .261) in the first half, medium for commissions ( r2 = .205) in the first half, and small for omissions ( r2 = .005) in the second half of the test. Omissions in the first half of the test also exhibited the highest standardized coefficient, being the most important variable identifying the ADHD presentations. This model was statistically significant, λ = .58, χ² (9) = 86.95, p <  .001), and classified 56.60% of the sample correctly (66.70% from the control group, 41.90% from the inattentive presentation, 44.10% from the impulsive-hyperactive presentation, and 57.40% from the combined presentation group). Finally, the highest absolute correlation between each variable and any discriminant function were found between omissions in the first half and the first discriminant function ( r = .91); between commissions in the first half and the second function (r = .99); and between omissions in the second half and the third function ( r = .69).

The second model (with TOVA variables) showed that only response time in the second half was a statistically significant predictor of group membership. The squared canonical correlation of the discriminant function was small for this variable ( r2 = .057). This model was statistically significant, λ = .943, χ² (3) = 9.97, p =  .019), although only a 33.70% of the sample was correctly classified (60.30% from the control group; 9.10% from the inattentive presentation; 10.60% from the impulsive-hyperactive presentation; and 50% from the combined presentation group).

On the whole, results indicated that the model which best classified the ADHD presentations was the model based on variables from Aula Nesplora test. While the identification percentages for the control group and the combined presentations of ADHD were similar in both models, it is the first model in which the explanatory power of the variables to predict inattentive and impulsive-hyperactive presentations is greater.

The results from the first model indicate the important role of omissions and commissions in identifying and discriminating between the three ADHD presentations as well as the group without ADHD. The discriminatory analyses for the TOVA test showed that response time was a significant predictor for identifying the combined presentation and the control group.

This study compared the discriminant value of attentional variables provided by a traditional CPT test (TOVA) with those from a virtual reality test (Aula Nesplora) to identify the ADHD presentations along with the presence or absence of ADHD symptomatology.

According to the analysis, the Aula Nesplora test showed better sensitivity and specificity than the TOVA test. The percentages of correctly identified children with the combined presentation of ADHD and children without ADHD were similar for both tests. However, the percentage of identification of inattentive and impulsive-hyperactive presentations was significantly higher using Aula Nesplora.

Some studies and systematic reviews have demonstrated the usefulness of traditional CPTs such as TOVA in differentiating children with ADHD from other diagnoses or healthy controls with better results than TOVA produced in the current study (Berger et al., 2017; González-Castro et al., 2013). For example, Berger et al. (2017) showed high accuracy in identifying participants aged between 7 and 12 with ADHD (with sensitivity and specificity rates of 85% or more). Previous research has shown that incorporating environmental distractors improved the test’s utility in differentiating ADHD from control children (Berger & Cassuto, 2014; Berger et al., 2017). Also, according to Slobodin, Cassuto, and Berger, (2015) the negative effect of environmental distractors on CPT performance of ADHD children did not diminish with age, suggesting that distractibility is a persistent deficit in ADHD. It could be one of the reasons for the lower power of TOVA compared to AULA, given that the TOVA test does not incorporate distractors.

Studies such as González-Castro et al. (2013) showed significant differences between the ADHD presentation types. The authors also indicated that variability and response time were the variables with greater explanatory power, which is similar to the results in the current study where the response time in the second half was the most significant variable in the TOVA test. This could be explained by the inner structure of the TOVA test, in which the first half used a “rare target” paradigm where only 22.5 % of the presented stimuli are targets and 77.5 % non-targets (Hall et al., 2016). Finally, the absence of differences between the types of presentation could be related to the fact that children are becoming more and more expert in and used to using computers. Thus, putting children in this kind of environment might advantage some students who are very familiar with technology.

In contrast, Aula Nesplora demonstrated good explanatory power in the identification of ADHD and ADHD presentations showing the additional usefulness of virtual reality CPTs in providing significant information for the differentiation of the ADHD presentations. Also, Negut et al. (2017) noted that the virtual classroom developed by Rizzo et al. (2000) and a traditional CPT discriminates between children with ADHD and developmentally typical children, with the virtual classroom being more powerful than traditional CPT. However, as those authors pointed out, that study is not without limitations, such as the fact that 70% of the ADHD sample had pharmacological support during the evaluation or scarce interaction between clinical status and test condition.

In the case of the Aula Nesplora, the most important variable to identify the ADHD presentations was omissions. This could be related to the incorporation of potential distractors in the virtual reality environment, aspect that is not present in traditional CPTs (Berger & Cassuto, 2014).

These findings are supported by previous studies showing that virtual reality tools represent a breakthrough in the diagnosis of ADHD, precisely because they allow control and ADHD groups to be compared in a realistic environment (Areces et al., 2016). Other advantages of these new measures are related to the additional information provided by the instrument. For example, instruments like Aula Nesplora incorporate measurements of head movement (registered by sensors located in the glasses) and the angle of vision or ocular movement. This improves the interpretation of the results given that we can discover whether the child failed to respond to the stimulus because they were absorbed or because they had a very high activity level. Despite the higher cost of this kind of system, it provides additional information that can prevent false positive or false negative diagnostic errors (Riccio & Reynolds, 2001).

In essence, at the present a single test cannot be used as a sole measure to diagnose ADHD. Although the most commonly-used and widely-accepted method of assessment is the clinician’s judgment nowadays (Hall et al., 2016), clinical assessments have vulnerabilities, such as those related to the use of interviews and questionnaires administered to family or teachers, who may have significant biases. These vulnerabilities may lead to under-diagnosis, over-diagnosis, and to under or over treatment of ADHD (Lange et al., 2014). Hall et al. (2016) carried out a systematic review to provide an overview of the evidence for commercially available CPTs. They included two studies in which the CPT was combined with other measures such as information provided by a teacher or parents and suggested that rating scales and TOVA are measuring distinct but important aspects of ADHD, and the addition of TOVA makes a valuable contribution to assessment of ADHD. Also, according to Vogt (2017), questionnaires are based on behavioral presentations and the correlation with neurocognitive tests is known to be low to moderate. In the future it would be interesting to compare the information provided by parents and teachers with performance in traditional and virtual CPTs.

In summary, given that both tests (TOVA and Aula Nesplora) have demonstrated their usefulness in identifying children with ADHD, it may be wise to incorporate this kind of tasks into current assessment protocols. This would allow clinicians and researchers to reach a more accurate diagnosis and adjust treatment (Rodríguez et al., 2016).

Finally, it is necessary to acknowledge some limitations of the present study that should be addressed in future studies. Firstly, additional studies with a wider sample size are needed to test whether the statistical power of the variables is similar to that found in this study. In addition, a direct comparison between both tests (Virtual Reality and Traditional CPTs) within the same sample would provide more information regarding the different benefits of each one in ADHD diagnosis.

In general, these results have some significant implications for clinical management of ADHD. It has been shown that the use of CPTs –regardless their differential characteristics- allows us to accurately identify those participants who may be suffering from the disorder, and to examine in depth the characteristics of each individual’s symptoms (e.g., predominantly inattentive presentation). Once a particular profile is established, this information will be particularly useful for tailoring specific interventions, adjusted to the child’s characteristics.