FOTOTRANS Study. Multicentre study on the validity of Fototest under clinical practice conditions

Abstract

Objective

Fototest is a short simple test, applicable to illiterate subjects and not influenced by educational level. The purpose of this study is to assess its validity for cognitive impairment (CI) and dementia (DEM) under routine clinical practice conditions.

Material and methods

A cross-sectional, multicentre and naturalistic study conducted in General Neurology clinics. The subjects were over 60 years old, with a previously established diagnosis of no cognitive impairment (NoCI), cognitive impairment without dementia (CInoDEM), or dementia (DEM). The discriminant validity was assessed using the area under the ROC curve (AUC), sensitivity (Sn), specificity (Sp), and probability coefficients. Partial correlations were calculated, adjusted for age, sex and education level with Eurotest, verbal fluency test (VFT) and Global Deterioration Scale (GDS) score to evaluate the concurrent and construct validity.

Results

Nineteen neurologists included a total of 589 subjects: 361 NoCI, 106 CINoDEM and 122 DEM. The discriminant value was better for DEM (AUC 0.94 ± 0.02) than for CI (0.86 ± 0.02). For DEM the best cut-off point was 26/27 (Sn = 0.88 [95% CI, 0.93-0.97], Sp = 0.87 [95% CI, 0.84–0.90]); and for CI, 28/29 (Sn = 0.71 [95% CI, 0.65–0.77], Sp = 0.84 [95% CI, 0.80–0.88]). Fototest showed a high and significant correlation with Eurotest (r = 0.70 ± 0.02), VFT (r = 0.68 ± 0.02) and the GDS score (r = −0.77 ± 0.02).

Conclusion

The Fototest showed adequate validity for CI and DEM in general neurology clinics in an extensive and wide geographical area.

Resumen

Objetivo

El Fototest es un test breve, fácil, aplicable a analfabetos y no influido por el nivel educativo. El objetivo de este estudio es evaluar su validez para deterioro cognitivo (DC) y demencia (DEM) en condiciones de práctica clínica habitual.

Material y métodos

Estudio transversal multicéntrico y naturalístico realizado en consultas de neurología general. Se incluyó a sujetos mayores de 60 años con diagnóstico previamente establecido de no deterioro cognitivo (NoDC), deterioro cognitivo sin demencia (DCsD) o demencia (DEM). La validez discriminativa se evaluó mediante el área bajo la curva ROC (aROC), sensibilidad (S), especificidad (E) y cocientes de probabilidad. Se calcularon correlaciones parciales ajustadas por edad, sexo y nivel educativo con Eurotest, fluidez verbal semántica (FVS) y el estadio GDS para evaluar la validez concurrente y de constructo.

Resultados

Diecinueve neurólogos incluyeron a un total de 589 sujetos: 361 NoDC, 106 DCsD y 122 DEM. La validez discriminativa es mejor para DEM (aROC 0,94 ± 0,02) que para DC (0,86 ± 0,02). Para DEM el mejor punto de corte es 26/27 (S = 0,88 [IC del 95%, 0,93-0,97], E = 0,87 [IC del 95%, 0,84-0,90]), y para DC, 28/29 (S = 0,71 [IC del 95%, 0,65-0,77], E = 0,84 [IC del 95%, 0,80-0,88]). El Fototest muestra una alta y significativa correlación con el Eurotest (r = 0,70 ± 0,02), la FVS (r = 0,68 ± 0,02) y el estadio GDS (r = –0,77 ± 0,02).

Conclusión

El Fototest muestra una adecuada validez para DC y DEM, en consultas generales de neurología de un extenso y variado ámbito geográfico.

Introduction

Fototest (www.fototest.es) is a cognitive test developed recently in our country, whose use is becoming increasingly widespread. Its distinguishing features are its brevity (it can be applied in less than 3 min), the possibility of use by illiterate patients and absence of paper and pencil tasks. These features make it especially suitable for people with a low educational level and for short consultation visits such as those in primary care.¹

Fototest consists of 3 tasks²; the initial task uses colour photographs and consists of naming 6 common objects shown in the foreground and corresponding to the categories “games”, “vehicles”, “fruits”, “musical instruments”, “items of clothing” and “kitchen utensils”; the frequency of use of each object varies between moderately frequent (“trumpet”) and frequent (“car”). Other variable factors are semantic field amplitude, from narrow (“kitchen utensils”) to extensive (“fruits”), and representativeness of each object with respect to the semantic field, including some items that are very typical (“spoon”) and others that are not as representative (“shoe”). A verbal fluency task is carried out upon completion of the task described above, consisting of providing names of people. Subjects are asked to say the names of persons of the opposite sex for half a minute. Next, they are asked to repeat the task with names of persons of the same sex for another half minute. It has been proven that this verbal fluency task is not influenced by educational variables.³ This verbal fluency task also acts as a distracting task. Once it has been completed, subjects are asked to remember the pictures named initially, for which they are given a time of 20 s. After this period, they are offered names of item categories as an aid, for only those items not remembered spontaneously. Therefore, this instrument includes several cognitive domains, language (naming), executive tasks (verbal fluency) and episodic memory, both free and aided by clues.

The results of Fototest, which are not influenced by the educational level of subjects,⁴ fall between a minimum of zero and an unlimited maximum, meaning that there is no ceiling effect. The distribution follows a normal function.⁴ Fototest has high internal consistency (Cronbach's alpha 0.94) and high test-retest (intraclass correlation coefficient [ICC] 0.89 ± 0.04) and interobserver reliability (ICC = 0.98 ± 0.01).⁴

Concurrent and discriminant construct validity has been evaluated in 2 studies: 1 case–control study with a total of 60 subjects⁵ and 1 cross-sectional study with 378 subjects.² Finally, a blind, prospective study conducted in primary care has shown that Fototest has greater effectiveness and lower cost than the Mini-Mental test,⁶ and similar efficacy to other instruments applicable to illiterate subjects, such as T@M and Eurotest, but more efficiency due to its lower cost and requiring half the time.^7,8 We recommend the cut-off points 26/27 for dementia (DEM) and 28/29 for cognitive impairment without dementia (CIwD).⁹

So far, all studies have been carried out by the group that developed Fototest, with no external verification of these results. Our goal now is to evaluate the diagnostic usefulness of Fototest in routine clinical practice conditions in a multicentre sample.

Material and methods

Design

FOTOTRANS was a phase II study for the evaluation of diagnostic tests.¹⁰ It was a cross-sectional multicentre naturalistic study, carried out in general neurology departments throughout Spain during the period between May and October 2006. The participation of researchers was voluntary and unpaid, and recruitment was conducted through the RedIris Neurology list¹¹ and the Neurology Study Group for Behaviour and Dementia of the Spanish Society of Neurology (GENCD-SEN).

Participants

The study included patients over 60 years, monitored continuously at the general neurology departments of the researchers involved, with previously established clinical and cognitive diagnoses. The only exclusion criteria were lack of consent and prior participation in the study.

Patients were classified as DEM (DSM-IV criteria¹²), CIwD (mild cognitive impairment criteria according to GENCD-SEN¹³) and no cognitive impairment (NoCI).

During consultation visits, each investigator selected the first subject on each day who met the inclusion criteria in a prospective, systematic and predefined manner. Only 1 subject was selected per day and researcher. The study required the inclusion of at least 25 subjects per investigator to ensure a minimum degree of consistency and validity of data from each of the researchers.

Procedure

Fototest was administered at the end of the consultation visit and its results did not modify the previous clinical or cognitive diagnosis. In addition, Eurotest^14,15 and a test of semantic verbal fluency (SVF) (naming animals during 1 min) were also administered.¹⁶ The cognitive diagnosis, which had been established previously, was based on the medical history, cognitive assessment and complementary studies conducted by each of the researchers.

We recorded the following variables: gender (male/female), age in years, literacy (literate/illiterate), educational level (none or incomplete primary, primary and secondary or higher education), cognitive diagnosis (DEM, CIwD, NoCI), possibility that the clinical diagnosis might affect cognitive performance (yes/no), use of drugs that might affect cognitive performance (yes/no), subjective complaints of memory loss (yes/no), results of short tests administered (Fototest, Eurotest, SVF) and global deterioration scale (GDS) stage.

Statistical analysis

A comparison of continuous variables between groups was performed using the ANOVA univariate test, and that of categorical variables using the chi-square test. The diagnostic usefulness was evaluated independently for NoCI vs. CI (CIwD + DEM) and DEM vs. NoDEM (CIwD + NoCI), through the area under the ROC curve (AUC), using the established cognitive diagnosis as a gold standard. We considered as the best cut-off point that which maximised the sum of sensitivity (Se) and specificity (Sp). We calculated specific likelihood ratios (LR) for various result ranges that maximised their interpretation, as well as their post-test probabilities for various prevalences (pre-test probabilities). The comparison of usefulness between tests was carried out through the comparison of AUC values derived from the same cases.¹⁷ Convergent validity was assessed by calculating the partial correlations between Fototest and other instruments employed, and that of the construct with the GDS stage. All these correlations were adjusted for age, gender and educational level. All parameters were estimated with their 95% confidence intervals (95% CI). Contrasts were bidirectional with an error α = 0.05. Calculations were performed using SPSS 11.5 (SPSS Inc., Chicago) and MedCalc 7.0.

Ethical and formal aspects

The FOTOTRANS study was approved by the research ethics committee of Hospital Universitario Virgen de las Nieves in Granada, and had the scientific backing of the Spanish and Andalusian societies of Neurology. All subjects or their caregivers were informed in detail and gave their consent to participate in the study. The study design and preparation of this manuscript conformed to the STARD recommendations for diagnostic test studies.¹⁸

Results

A total of 23 researchers began the study, of which 3 did not reach the minimum established set of 25 subjects and 1 applied Fototest incorrectly (fluidity of names for 1 min instead of half a minute). The 19 researchers finally considered included a total of 597 subjects, 8 of whom did not complete Fototest, thus resulting in a final sample of 589 subjects: 361 NoCI, 106 CIwD and 122 DEM (Fig. 1). Among the 361 NoCI subjects, 120 (33.2%) suffered a neurological process that could potentially induce cognitive impairment (cerebral vascular disease, epilepsy, Parkinson's disease, etc.), 76 (21.0%) were taking drugs that could negatively affect cognitive performance (tranquilisers, anticholinergics, etc.) and 93 (25.8%) reported subjective complaints of memory loss; 190 (52.6%) presented at least 1 of the above conditions and 62 (17.2%) reported more than 1.

Figure 1.

Flow diagram of the FOTOTRANS study. CIwD: cognitive impairment without dementia; DEM: dementia; NoCI: no cognitive impairment.

The socio-demographic characteristics of the sample and the results in the tests employed are summarised in Table 1. The overall sample had a mean age ± standard deviation of 72.69 ± 7.4 years, with a predominance of females (56.5%) and low educational level (7% of subjects were illiterate and 49.9% had no education or had not completed primary education). There were no significant differences in gender distribution between the diagnostic groups. Additionally, subjects with DEM were significantly older than those with CIwD and these in turn were older than those with NoCI (76.75 ± 6.0, 73.50 ± 6.8 and 71.09 ± 7.4 years, respectively; P < .001). Illiterate patients were more frequent in the DEM group (11.5%) than in the NoCI and CIwD groups (5.8% and 5.7%, respectively), although this difference did not reach statistical significance (P = .08). There was a significant difference in the test results for different groups with NoCI > CIwD > DEM (P < .001 for all comparisons). Distribution by GDS stage is shown in Fig. 2.

Table 1.

Socio-demographic characteristics and results by diagnostic groups.

	Total	NoCI	CIwD	DEM	P
Number of subjects	589	361	106	122
Age (years)	72.69 ± 7.4	71.09 ± 7.4	73.50 ± 6.8	76.75 ± 6.0	<.001
Gender (females)	333 (56.5)	205 (56.8)	58 (54.7)	70 (57.4)	.91
Illiterate	41 (7.0)	21 (5.8)	6 (5.7)	14 (11.5)	.08
Educational level
None/incomplete	294 (49.9)	180 (49.9)	44 (41.9)	70 (57.4)
Primary	192 (32.6)	115 (31.9)	41 (38.7)	36 (29.5)
Secondary or higher	103 (17.5)	66 (18.3)	21 (19.8)	16 (13.1)	.18
Fototest	29.63 ± 7.4 [589]	33.14 ± 4.8 [361]	28.89 ± 4.8 [106]	19.89 ± 6.8 [122]	<.001
Eurotest	22.98 ± 8.1 [548]	26.59 ± 4.9 [349]	22.41 ± 5.4 [101]	10.72 ± 7.5 [98]	<.001
SVF	12.17 ± 5.3 [578]	13.99 ± 4.7 [361]	12.08 ± 4.7 [103]	6.46 ± 3.5 [114]	<.001

CIwD: cognitive impairment without dementia; DEM: dementia; NoCI: no cognitive impairment; SVF: semantic verbal fluidity.

Values are presented as number of subjects (percentage) or mean ± standard deviation [number of subjects].

Figure 2.

Distribution of the sample by global deterioration scale (GDS) stage.

The results of Fototest for DEM are shown in Fig. 3A; the AUC for DEM was 0.94 (95% CI, 0.92 to 0.96), similar to Eurotest (0.95 [95% CI, 0.93 to 0.97], P = .54); both were higher than the SVF (0.90 [95% CI, 0.88 to 0.92], P < .05). For DEM, the best cut-off point was 26/27, with Se = 0.88 (95% CI, 0.81–0.94) and Sp = 0.87 (95% CI, 0.84–0.90). Table 2 shows the distribution by ranges of results and their specific LR. Fig. 3B shows the distribution of results for CI. The AUC of Fototest for CI was 0.86 (95% CI, 0.83 to 0.89), similar to Eurotest (0.84 [95% CI, 0.80–0.88], P = .71); both were higher than SVF (0.78 [95% CI, 0.74–0.82], P < .01). The cut-off point which maximised the sum of Sp and Se for CI was 28/29, for which Se = 0.71 (95% CI, 0.65–0.77) and Sp = 0.84 (95% CI, 0.80–0.88). Table 3 shows the distribution by ranges of results and their specific LR. The results of Fototest showed a high and significant correlation with Eurotest (r = 0.70 ± 0.02, P < .001) and SVF (r = 0.68 ± 0.02, P < .001) (Fig. 4), as well as a significant negative association with GDS stage (r = −0.77 ± 0.02, P < .001) (Fig. 5), which supported the convergent and construct validity of the instrument.

Figure 3.

Distribution of the Fototest scores by cognitive status. CI: cognitive impairment; DEM: dementia.

Table 2.

Distribution of results, specific likelihood ratios and post-test probability of dementia according to Fototest scores (grouped by ranges).

Fototest	DEM^a	No DEM^a	LR	Prevalence (pre-test probability)
				0.05	0.1	0.26^b	0.5
≤21	67 (0.55)	8 (0.02)	33.3	0.64^c	0.78	0.92	0.97
22–24	22 (0.18)	19 (0.04)	4.4	0.19	0.32	0.61	0.81
25–26	19 (0.16)	33 (0.07)	2.2	0.10	0.19	0.43	0.69
27–29	10 (0.08)	76 (0.16)	0.5	0.03	0.05	0.15	0.33
≥30	4 (0.03)	331 (0.71)	0.05	0.003	0.005	0.017	0.05
Total	122 (100)	467 (100)

DEM: dementia; No DEM: no dementia; LR: likelihood ratio.

a

No subjects (probability within column).

b

Prevalence of dementia in this study.

c

Post-test probability.

Table 3.

Distribution of results, specific likelihood ratios and post-test probability of dementia according to Fototest scores (grouped by ranges).

Fototest	CI^a	No CI^a	LR	Prevalence (pre-test probability)
				0.05	0.1	0.25	0.39^b	0.5
≤24	106 (0.465)	10 (0.028)	16.60	0.47^c	0.65	0.85	0.91	0.94
25–26	37 (0.162)	15 (0.042)	3.86	0.17	0.30	0.56	0.71	0.79
27–30	45 (0.197)	73 (0.202)	0.98	0.05	0.1	0.25	0.38	0.49
31–33	23 (0.101)	96 (0.266)	0.38	0.02	0.04	0.11	0.20	0.27
≥34	17 (0.074)	167 (0.463)	0.16	0.008	0.02	0.05	0.09	0.14
Total	228 (100)	361 (100)

CI: cognitive impairment; No CI: no cognitive impairment; LR: likelihood ratio.

a

No subjects (probability within column).

b

Prevalence of cognitive impairment in this study.

c

Post-test probability.

Figure 4.

Dispersion diagram of the Fototest scores with respect to Eurotest and verbal semantic fluidity (VSF).

Figure 5.

Fototest scores by GDS stage.

Discussion

The results of the FOTOTRANS study show that Fototest is a valid, useful instrument in the identification of DEM and CI in general neurology consultations under everyday clinical practice conditions. The discriminant validity was slightly lower for CI, a logical finding given the more difficult diagnosis of CI. This tends to occur in all instruments, whether abbreviated or full.

The overall discriminant value measured as the AUC of Fototest for DEM (0.95 ± 0.02) and CI (0.86 ± 0.02) was lower than that of T@M.¹⁹ The latter is a recent instrument evaluating various types of memory and specifically designed for the detection of Alzheimer's disease in the prodromal phase. In the only study conducted, T@M showed a very high unspecified dementia (UD) level for both Alzheimer's disease (AUC 0.99) and for amnestic CI (AUC 0.93). The reasons for this could be manifold. Firstly, the controls were healthy volunteers in the T@M study; those with subjective memory complaints were explicitly ruled out. These control subjects were probably a normal population. This would make them much easier to differentiate with respect to subjects with AD and DEM than our controls, who were from a clinical sample in which all were neurological patients and reported memory loss (>25%) or some circumstance (disease, drugs) that could compromise their cognitive performance (>50%) in a high proportion of cases. Secondly, T@M is a specific instrument for memory assessment. Therefore, the study paid attention to the diagnosis of Alzheimer's disease and amnestic cognitive impairment. In contrast, in our study the attention was on dementia and cognitive impairment in general, rather than on amnestic profile impairment. A recent study using the 2 instruments reported a similar diagnostic usefulness and associated cost. The only significant difference was that Fototest required less than 3 min to be administered whereas T@M needed almost 7 min.^7,8

There was no difference between the discriminant capacity of Fototest and that of Eurotest, an instrument of proven effectiveness and validity in subjects with a low level of education.²⁰ The Eurotest results for DEM in this study (AUC 0.95) were very similar to those found in another multicentre study²¹ conducted with this instrument (AUC 0.93), thus ensuring its strength and validity. While easy and short, Eurotest is more complex to implement and evaluate and takes longer to administer (about 7 min). Therefore, Fototest should be the first choice, since it has the same usefulness and is much easier and faster to apply.

Other brief instruments for CI that have been evaluated in our country are the adaptations of the MIS²² and MoCA²³ instruments. The results of the Spanish adaptation of MoCA,²⁴ a new abbreviated instrument that is becoming widespread and was specifically developed for the identification of CI, are very modest, especially for CI (DEM AUC 0.89 and CI AUC 0.69). In a prospective study in a clinical sample, the Spanish adaptation of MIS²⁵ with the cut-off point 4/5 showed an acceptable level of validity for the identification of DEM and CI (AUC 0.92 for both). The negative side is that MIS requires that subjects be able to read and write; consequently, it cannot be applied to illiterate subjects, and an additional tool must be available for such cases.

The results were also strong and consistent, and remained stable in various subanalyses. There were no significant differences between the results obtained in the southern regions, where the overall level of education was lower, and centres in the northern part of the country. Neither were the results altered when the sample excluded cases provided by researchers from hospitals in Granada or in all of Andalusia (data not shown).

The great strength of this study was its external validity, due not only to its nature as a multicentre study with a large sample size, but above all due to the clinical setting in which it was developed and the virtual absence of exclusion criteria, which allowed the incorporation of subjects systematically excluded from other studies. These features not only gave the study a naturalistic character, since the sample was a true reflection of the population in which the instrument should be applied, but also underestimated the usefulness of the instrument. Another strength of this study was the independence of the cognitive diagnosis with respect to the results of the instrument and the main researcher.

This study had the limitations inherent to phase II studies for the evaluation of diagnostic tests, primarily its cross-sectional character and non-random recruitment and allocation, which made it impossible to assess the true predictive validity. The application of the test in a non-blind manner by the researchers, who knew the diagnoses, was also a known source of bias that may have overestimated validity. Finally, the multiplicity of researchers could also lessen the validity of the cognitive diagnoses due to heterogeneity in the interpretation of criteria. However, they were all experienced professionals in the field of cognitive-behavioural neurology and, in addition, the diagnoses considered as gold standard were those the subjects already had in real life, with their corresponding social and healthcare implications, thus offering a high level of validity.²⁶

In conclusion, Fototest has confirmed, in an ample and varied clinical setting, its usefulness and discriminant validity for CI and DEM, as well as its concurrent and construct validity. This validity, along with its reliability, brevity, simplicity and applicability, makes it a perfect alternative for use in everyday clinical practice.

Financing

This study was partly funded by the Ministry of Health of the Junta de Andalucía (Exp. 441/06) and a research support grant from Janssen-Cilag Laboratories. None of these organisations participated in the design, data collection, analysis and interpretation, or in the elaboration of this study.

Conflict of interests

C. Carnero-Pardo is the creator of Fototest and Eurotest.