Validation of the Spanish-language version of the Montreal Cognitive Assessment as a screening test for cognitive impairment in multiple sclerosis

Abstract

Background

The neuropsychological batteries traditionally used for the assessment of cognitive impairment (CI) in patients with multiple sclerosis are complex tests requiring a long time to administer. Simpler tests are needed to detect cognitive impairment in daily clinical practice.

Objective

We aimed to evaluate the diagnostic validity and reliability of the Montreal Cognitive Assessment (MoCA) test as a screening tool for CI in patients with multiple sclerosis, as compared against the Brief Neuropsychological Battery.

Material and methods

We recruited 52 patients with multiple sclerosis (61.5% women; mean age [standard deviation]: 41.7 [11.5] years). We analysed the reliability (internal consistency, interobserver reliability, and test-retest reliability), construct validity (factor analysis, Pearson correlation coefficient, and coefficient of determination), and criterion validity (ROC curve, sensitivity, specificity, total agreement, positive and negative predictive values, positive and negative likelihood ratios, and Fagan nomogram) of the MoCA test in this population.

Results

The prevalence of CI was 21.2% according to findings from the Brief Neuropsychological Battery, and 25% according to the MoCA test. The MoCA test showed good internal consistency (Cronbach alpha, 0.822) and interobserver and test-retest reliability (intraclass correlation coefficient 0.80 and 0.96, respectively). The correlation coefficient between total Brief Neuropsychological Battery and MoCA test scores was 0.82. The optimal cut-off point on the ROC curve was 25-26, yielding 91% sensitivity and 93% specificity.

Conclusion

The MoCA test is a valid and reliable tool for screening for CI in patients with multiple sclerosis.

Resumen

Introducción

Las baterías neuropsicológicas empleadas tradicionalmente para el diagnóstico del deterioro cognitivo (DC) en la esclerosis múltiple son pruebas complejas que conllevan mucho tiempo. Se necesitan test más simples para detectar el DC en la práctica clínica diaria.

Objetivo

Evaluar la validez diagnóstica y la fiabilidad de la escala Montreal Cognitive Assessment (MoCA) como herramienta de cribado de DC en la esclerosis múltiple frente a la Batería Neuropsicológica Breve.

Material y métodos

Se seleccionaron 52 pacientes (61,5% mujeres, edad media [desviación estándar] 41,7 [11,5] años). Se analizaron la fiabilidad (consistencia interna, interobservador y test-retest) y la validez de constructo (análisis factorial, coeficiente de correlación de Pearson y coeficiente de determinación) y de criterio (curva ROC, sensibilidad, especificidad, acuerdo global, valores predictivos positivo y negativo, cocientes de probabilidad positivo y negativo y nomograma de Fagan).

Resultados

La prevalencia de DC fue del 21,2% según la Batería Neuropsicológica Breve y del 25% según el MoCA. El MoCA mostró buena consistencia interna (alfa de Cronbach 0,822) y buena fiabilidad interobservador y test-retest (coeficiente de correlación intraclase de 0,80 y 0,96, respectivamente). El coeficiente de correlación entre la puntuación total de la Batería Neuropsicológica Breve y el MoCA fue de 0,82. El punto óptimo de corte en la curva ROC fue 25-26, con una sensibilidad del 91% y una especificidad del 93%.

Conclusión

El MoCA es una herramienta de cribado válida y fiable para la detección de DC en pacientes con esclerosis múltiple.

Introduction

Cognitive impairment (CI) is estimated to affect between 45% and 65% of patients with multiple sclerosis (MS).^1,2 CI can present in patients with any form of MS, at any stage of the disease.³ While the symptoms and severity of CI are variable, the most frequently affected domains are learning, memory, attention, information processing speed, visuospatial abilities, and executive function.¹

Diagnosis of CI associated with MS is currently established with validated neuropsychological batteries, with the most common being Rao’s Brief Repeatable Battery of Neuropsychological Tests⁴ and the Minimal Assessment of Cognitive Function in Multiple Sclerosis.⁵ These batteries are complex, time-consuming, and must be administered by trained specialists, precluding their routine use in clinical practice. The Brief Neuropsychological Battery (BNB)⁶ is a validated, reduced version of the Brief Repeatable Battery of Neuropsychological Tests, and is quicker to complete and may be administered by any healthcare professional with appropriate training; nonetheless, the administration time for the BNB is approximately 20 minutes. In this context, the need arises for simple, rapid tests enabling objective detection of patients with possible CI in a matter of minutes, for subsequent referral for specialised neuropsychological assessment. Of the available cognitive screening tests, many of those normally used to screen for Alzheimer disease, such as the Mini–Mental State Examination and its variants, are not sensitive to MS-related cognitive alterations.⁷ Other instruments, including the Symbol Digit Modalities Test (SDMT),⁸ the Brief International Cognitive Assessment for Multiple Sclerosis,⁹ and the Multiple Sclerosis Neuropsychological Screening Questionnaire,¹⁰ have been used in patients with MS, with some limitations.¹¹ For example, the results of the latter test may be influenced by the patient’s mood, whereas the Brief International Cognitive Assessment for Multiple Sclerosis can be complex and requires a long time to administer (approximately 15 minutes).¹¹ Furthermore, while excellent sensitivity and specificity values have been reported for the SDMT, the test only provides partial information about cognitive performance in patients with MS.¹²

The Montreal Cognitive Assessment (MoCA) is a screening test requiring less than 10 minutes to administer. It has been validated in numerous languages, and displays good sensitivity and specificity in other diseases, such as mild CI and Alzheimer disease. The tool is also widely used to assess CI with dysexecutive alterations, as is the case for CI associated with Parkinson’s disease,¹³ for which it has also shown good sensitivity (75%) and specificity (82%). The few studies published on the use of the MoCA in patients with MS seem to support its value as a brief screening tool for the detection of CI in this patient group.¹⁴ However, until recently, no study had analysed the precision of the MoCA for detecting CI associated with MS, or established clearly defined cut-off points. The Portuguese-language version of the MoCA was recently shown to be psychometrically valid for screening patients with MS.¹⁵ The objective of this study is to validate the Spanish-language version of the MoCA as a screening tool to detect CI in patients with MS, compared against the BNB. We also aimed to analyse possible demographic and clinical factors associated with detection of CI with the BNB.

Material and methods

Study design

We conducted an observational cross-sectional study including consecutive patients with MS attended over a period of 4 months at a specialised demyelinating diseases clinic. The study design, database, and participants’ informed consent were assessed and approved by the hospital’s ethics committee.

Patients

We included 52 patients diagnosed with clinically isolated syndrome or MS according to the 2010 McDonald criteria¹⁶; all participants gave written informed consent to be included in the study. Inclusion criteria were as follows: 1) age older than 18 years; 2) absence of relapses or treatment with pulse corticosteroid therapy in the 12 weeks prior to study inclusion; and 3) sufficient hearing, vision, and physical condition to perform the assessment and attend interviews. The following exclusion criteria applied: 1) severe intercurrent or chronic systemic disease that might affect mental capacities; 2) neurological disease other than MS that may affect cognition (cerebrovascular disease, Parkinson’s disease, epilepsy, hydrocephalus, etc.); 3) first language other than Spanish; and 4) illiteracy.

Data were collected on demographic (age at study inclusion, level of schooling, sex) and clinical variables (age at diagnosis, disease progression time, initial type of relapse, clinical form of MS, disability according to the Expanded Disability Status Scale [EDSS],^17,18 treatments received, and form of treatment).

Cognitive assessment

The study aimed to validate the Spanish-language version of the MoCA scale. This test assesses visuospatial abilities by requiring patients to draw a clock (3 points) and copy a cube (one point). Naming is tested with 3 infrequent animals (3 points). Attention/concentration is evaluated with a forward and backward digit span (one point each), a concentration test (one point), and serial subtraction of the number 7 (one point). Language is assessed with the repetition of 2 syntactically complex phrases (2 points) and a test of verbal fluency (one point). Executive function and abstraction are tested in the verbal fluency test mentioned above, a short version of the Trail-Making Test part B (one point), and verbal abstraction tasks (2 points). Short-term memory is evaluated by delayed recall (after 5 minutes) of 5 words (5 points). Finally, the test assesses orientation with 4 questions targeting orientation in time and 2 testing orientation in space (6 points). The maximum score for the test is 30, and the cut-off point for CI is < 26.¹⁹ The test is freely available in various languages at www.mocatest.org.

The BNB was used as the reference test. The battery comprises 4 neuropsychological tests. It includes a test evaluating declarative and episodic memory, using 12 items from the 7-minute screen neurocognitive battery,²⁰ in which the items are read and semantic reinforcement is provided, with free and cued recall. Language is evaluated with tests of semantic fluency, phonemic fluency, and fluency with a missing letter. Finally, the SDMT and Paced Auditory Serial Addition Test are used to assess executive function and abstraction. In this version of the battery, the Paced Auditory Serial Addition Test is administered verbally and the patient is allowed to respond at their own pace, providing a precise reflection of information processing speed.⁶ We defined CI as scores 2 standard deviations below normative values in 2 subscales (memory, language, or executive function/abstraction) and/or in the total BNB score.

Quality of life and psychological evaluation

We screened for depression with the second edition of the Beck Depression Inventory, anxiety with the Hamilton Anxiety Rating Scale, and fatigue with the Modified Fatigue Impact Scale, and evaluated quality of life with the Functional Assessment of Multiple Sclerosis scale and the Multiple Sclerosis Quality of Life-54 scale.

Study protocol

During the initial consultation at the time of study inclusion, patients were given a booklet with the psychological and quality of life scales, which were always completed in the same order, and demographic and clinical data were collected. At the baseline consultation, patients returned the completed booklet and were administered the tests assessing CI (BNB and MoCA). Also during the baseline consultation, the first 12 patients were also rated for MoCA performance by an independent, blinded member of the neurology department in order to assess inter-rater reliability. While the patient completed the test, both neurologists collected data simultaneously, separated by a curtain. The patient used a single sheet of paper for the clock-drawing test, copying the cube, and to complete the Trail-Making Test; these tests were independently evaluated by each neurologist. Finally, 15 patients agreed to attend a third consultation after a period of 2-3 weeks, in order to assess the test-retest reliability of the MoCA (Fig. 1). None of these patients presented a relapse between the 2 assessments. In previous studies, test-retest evaluation has demonstrated good reliability without a significant practice effect.¹⁹ The flow chart in Fig. 1 shows the study process.

Figure 1.

Flow chart of the study process.

BNB: Brief Neuropsychological Battery; FAMS: Functional Assessment of Multiple Sclerosis Scale; HAM-A: Hamilton Anxiety Rating Scale; MFIS: Modified Fatigue Impact Scale; MoCA: Montreal Cognitive Assessment; MS: multiple sclerosis; MSQoL-54: Multiple Sclerosis Quality of Life-54.

Statistical analysis

We conducted a descriptive analysis of the clinical and demographic variables. Qualitative variables are reported as a distribution of frequencies, and quantitative variables are expressed as means and standard deviations or medians and interquartile ranges, for data not following a normal distribution.

To assess the reliability of the MoCA, we analysed internal consistency (Cronbach alpha statistic), inter-rater reliability, and test-retest reliability (Kappa index and intraclass correlation coefficient). The validity study of the MoCA assessed internal (factor analysis) and external construct validity (Pearson correlation coefficient and coefficient of determination) and criterion validity (determination of the area under the ROC curve, sensitivity, specificity, overall agreement, positive and negative predictive values, positive and negative likelihood ratios, and the Fagan nomogram).

We also conducted an analysis to identify sociodemographic and clinical variables that may be associated with diagnosis of CI according to BNB score. Quantitative variables were compared with the t test or with analysis of variance and the Bonferroni correction. Linear associations between quantitative variables were analysed with the Pearson correlation coefficient. The corresponding non-parametric tests (Mann-Whitney U test, Spearman correlation coefficient) were used to compare non–normally distributed data. A multivariate linear regression model was used to analyse variables presenting P-values below .05 in the univariate analysis in order to quantify effects on BNB score.

The SPSS software (version 20) was used for the statistical analysis. The threshold for statistical significance was set at P < .05.

Results

The sample included 32 women and 20 men, with a mean age (standard deviation) of 41.7 years (11.5). The flow chart in Fig. 1 shows the study process. Patients’ demographic and clinical characteristics are summarised in Table 1. The mean administration time was 8 minutes for the MoCA and 18 minutes for the BNB. The prevalence of CI in our sample was 21.2% (n = 11) according to the BNB and 25% (n = 13) according to the MoCA.

Table 1.

Baseline characteristics of the study population (n = 52).

Variable
Age in years, mean (SD)	41.7 (11.5)
Sex, n (%)
Women	32 (61.5)
Men	20 (38.5)
Years of schooling, mean (SD)	13.5 (3.6)
Level of education, n (%)
Basic	14 (26.9)
Intermediate	24 (46.2)
Higher education	14 (26.9)
Type of MS, n (%)
CIS	4 (7.7)
RRMS	42 (80.8)
SPMS	5 (9.6)
PPMS	1 (1.9)
MS progression time in years, median (IQR)	6 (4-13)
EDSS, median (IQR)	1.5 (1.0-2.0)
0-3 (mild disability), n (%)	44 (84.6)
3.5-6 (moderate disability), n (%)	8 (15.4)
6.5-10 (severe disability), n (%)	0 (0.0)
Disease-modifying treatment, n (%)	41 (79.8)
Annualised relapse rate	0.65

CIS: clinically isolated syndrome; EDSS: Expanded Disability Status Scale; IQR: interquartile range; MS: multiple sclerosis; PPMS: primary progressive MS; RRMS: relapsing-remitting MS; SD: standard deviation; SPMS: secondary progressive MS.

The total Cronbach alpha for the MoCA was 0.822. When different items were eliminated, this score decreased to values between 0.67 and 0.77 (Table 2). The correlation coefficient for each subscale and for the total score, as well as corrected correlation coefficients for the elimination of each subscale, are shown in Table 2.

Table 2.

Correlation between the different items on the Montreal Cognitive Assessment and total score.

	r	Corrected r	Corrected Cronbach alpha
Visuospatial abilities	0.75	0.61	0.68
Naming	0.72	0.66	0.71
Attention	0.78	0.65	0.67
Language	0.59	0.41	0.73
Abstraction/executive function	0.60	0.52	0.73
Short-term memory	0.72	0.44	0.77
Orientation	0.58	0.49	0.73

r: Pearson coefficient of correlation between item score and total score; corrected r: coefficient of correlation between the item and total MoCA score after elimination of that item.

The intraclass correlation coefficient was 0.80 (P < .001) for inter-rater reliability and 0.96 (P < .001) for test-retest reliability. The Kappa index was 1 in both cases.

Factor analysis revealed that the first 2 components accounted for 77.85% of variance, although due to clinical considerations we decided to analyse the 3-factor solution. Factor 1 included the visuospatial, naming, attention, and executive function subscales; factor 2 included the language and short-term memory subscales; and factor 3 included the orientation subscale. Correlations between variables and factors (factor loading) after oblique rotation are shown in Table 3.

Table 3.

Correlations between variables and factors (factor loading) after oblique rotation.

	Factor 1	Factor 2	Factor 3
	Visuospatial abilities, naming, attention, and abstraction/executive function	Language and short-term memory	Orientation
Visuospatial abilities	0.806	–0.062	0.151
Naming	0.871	–0.325	0.069
Attention	0.829	–0.040	–0.250
Language	0.405	0.789	0.003
Abstraction/executive function	0.732	–0.292	–0.475
Short-term memory	0.508	0.622	–0.108
Orientation	0.677	–0.082	0.631

Factor loading for components of each factor are shown in bold.

The coefficient of correlation between total scores on the MoCA and the BNB was 0.82 (P < .001), and the coefficient of determination was 0.676. Detailed analysis of correlations between total BNB score and MoCA subscale scores showed moderate correlation, with the strongest correlation being observed for the delayed recall subscale (r = 0.59) and the weakest for the abstraction subscale (r = 0.30).

Correlations between subscales on the BNB and on the MoCA, grouped by category (short-term memory, executive function, attention, and language) were moderate, but statistically significant in all cases (Table 4). The strongest correlations were between the subscales assessing language (r = 0.58) and attention and executive function (r = 0.55) (P < .001 in both cases).

Table 4.

Correlations between subscale scores on the Montreal Cognitive Assessment and the Brief Neuropsychological Battery (memory, language, executive function, and attention).

	BNB: short-term memory	BNB: language	BNB: abstraction/executive function	MoCA: short-term memory	MoCA: language	MoCA: abstraction/executive function
BNB: short-term memory	r	1
	P
BNB: language	r	0.353*	1
	P	.011
BNB: abstraction/executive function	r	0.336*	0.702**	1
	P	.016	< .001
MoCA: short-term memory	r	0.340*	0.516**	0.536**	1
	P	.015	< .001	< .001
MoCA: language	r	0.126	0.576**	0.540**	0.439**	1
	P	.378	< .001	< .001	.001
MoCA: executive function	r	0.133	0.506**	0.546**	0.353*	0.303*	1
	P	.350	< .001	< .001	.011	.031

BNB: Brief Neuropsychological Battery; MoCA: Montreal Cognitive Assessment; r: Pearson correlation coefficient; SDMT: Symbol Digit Modalities Test.

The highest and lowest statistically significant correlations are shown in bold.

*

Correlation significant at P < .05.

**

Correlation significant at P < .01.

The area under the ROC curve was 0.96 (P < .001). The MoCA cut-off points providing the highest sensitivity and specificity values were 25 and 26 (Table 5); therefore, the optimal cut-off point for detecting CI in patients with MS is < 26. The instrument displayed a sensitivity of 91% (95% confidence interval [95% CI], 62%-98%), specificity of 93% (95% CI, 81%-97%), and overall agreement of 92% (95% CI, 82%-97%). The positive predictive value was 77% (95% CI, 50%-92%) and the negative predictive value was 97% (95% CI, 87%-100%). The positive likelihood ratio was 12.42 (95% CI, 4.1-32.0) and the negative likelihood ratio was 0.09 (95% CI, 0.02-0.64). The Fagan nomogram is shown in Fig. 2. The diagnostic odds ratio was 137.8.

Table 5.

Validity, sensitivity, specificity, and cut-off points for the Montreal Cognitive Assessment as compared to diagnosis of cognitive impairment according to the Brief Neuropsychological Battery.

Cut-off point	Overall agreement	Kappa	Sensitivity (95% CI)	Specificity (95% CI)
< 24	0.90	0.70	72.73 (43.44-90.25)	95.12 (83.86-98.65)
< 25	0.92	0.78	90.91 (62.26-98.38)	92.68 (80.57-97.48)
< 26	0.92	0.78	90.91 (62.26-98.38)	92.68 (80.57-97.48)
< 27	0.75	0.45	90.91 (62.26-98.38)	70.73 (55.52-82.39)

95% CI: 95% confidence interval.

Figure 2.

a) ROC curve for total Montreal Cognitive Assessment score vs cognitive impairment according to the Brief Neuropsychological Battery. b) Fagan nomogram with likelihood ratios for the Montreal Cognitive Assessment. Blue line: positive likelihood ratio and post-test probability. Red line: negative likelihood ratio and post-test probability.

No significant correlations were identified between BNB score and patient age at diagnosis or at study inclusion. We also did not observe an association with sex. However, BNB scores did show an association with level of schooling: patients with basic education scored significantly lower than those with intermediate and university studies (P = .001). BNB scores showed a non-significant negative correlation with disease duration (rho = –0.21; P = .13). Patients with moderate disability (EDSS score 3.5-6) also scored lower on the BNB than those with moderate disability (EDSS scores 0-3) (P = .04). We were unable to reliably establish an association between CI and clinical form of MS, as the sample only included one patient with primary progressive and 5 patients with secondary progressive MS. In the linear regression model, years of schooling and degree of disability were identified as factors independently associated with BNB score. Regarding education, BNB scores increased by a mean of 5 points (95% CI, 2-8; P < .001) per year of schooling. Moderate disability was associated with an absolute difference of 33 points in mean BNB score (95% CI, 6-60; P = .017) when compared to mild disability. No significant association was observed between CI and degree of depression, anxiety, fatigue, or quality of life.

Discussion

Recent years have seen growing interest in the cognitive assessment of patients with MS, demonstrating the need for screening tools for use in clinical practice. In previous studies, the MoCA has shown good correlation with various neurocognitive batteries in screening for CI associated with MS.^14,15,21 However, only one study, using the Portuguese-language version of the MoCA,¹⁵ comprehensively examines the discriminative validity and diagnostic accuracy of the MoCA for detecting CI in patients with MS using a reference test for comparison. The aim of this study was to confirm the value of the Spanish-language version of the MoCA in screening for CI in patients with MS, in accordance with the relevant guidelines for this type of study (Standards for Reporting Diagnostic Accuracy Studies)²² and to compare it against a validated instrument. Our findings support the use of the Spanish-language version of the MoCA for this purpose, even in patients with mild disability. Furthermore, we have demonstrated that the MoCA is highly useful both for detecting CI in clinical practice and for the follow-up of patients with CI, even when administered by different raters, given the excellent inter-rater and test-retest reliability values.²³

In the study of internal consistency, the global Cronbach alpha was high, similar to the value obtained in the initial validation of the scale¹⁹ and the validations of the Portuguese- and Spanish-language versions,^15,24 exceeding the minimum value required for diagnosis or classification (0.80).²³ The correlation between individual subscale scores and corrected global score, the Cronbach alpha for each subscale, and the Cronbach alpha corrected for the elimination of each subscale demonstrate that the MoCA is a reliable instrument. All these results confirm that the MoCA accurately and consistently measures what it was intended to: CI associated with MS. Furthermore, the good convergent validity with the total BNB score established in the tests of content validity implies that both scales measure the same construct (CI), as has been demonstrated previously.¹⁵

As in other studies including patients with MS,^14,15,21 the MoCA has shown very good sensitivity without detriment to specificity; the positive and negative predictive values obtained were better than those observed for patients with Parkinson’s disease.¹³ Therefore, the MoCA seems to detect CI even better in patients with MS than in those with Parkinson’s disease. According to reference levels established by the Evidence-Based Medicine Working Group,²⁵ the positive and negative likelihood ratios of the MoCA are relevant and, together with the data obtained from the Fagan nomogram, allow us to conclude that the MoCA offers good discriminant capacity for detecting MS-associated CI. Furthermore, the area under the curve (0.96) indicates high diagnostic accuracy, better than that obtained in other studies (0.87).¹⁵ The MoCA cut-off points providing the best sensitivity and specificity coincide with that established in validation studies of the scale.¹⁹ These excellent diagnostic accuracy values, together with the shorter mean administration time compared to the BNB, make the MoCA a good method of screening for CI associated with MS.²⁶ Nonetheless, future studies comparing the MoCA with other brief screening instruments commonly used in MS, such as the SDMT, may be beneficial with a view to better defining its clinical use.

Results both from the BNB and from the MoCA established the prevalence of CI in our sample between 24% and 29%. While the different reported prevalence rates vary greatly, our results are consistent with the prevalence reported in patients with recent diagnosis of clinically isolated syndrome or relapsing-remitting MS with mild disability.²⁷ We found an association between level of schooling and CI; this is consistent with the results of recent longitudinal studies that found level of education to be a component of cognitive reserve and to protect against CI associated with MS.²⁸ However, we identified no significant differences related to sex, age at diagnosis, or age at study inclusion.

Our study presents certain limitations. Firstly, the BNB, rather than standard neuropsychological tests (Minimal Assessment of Cognitive Function in Multiple Sclerosis, Rao’s Brief Repeatable Battery of Neuropsychological Tests), was used as the reference test; nonetheless, a good correlation has been shown between BNB score and scores on the Brief Repeatable Battery of Neuropsychological Tests (r = 0.78; P <0.001).⁶ Secondly, given the characteristics of the centre where the study was conducted, our sample is representative of the population of patients with recent diagnosis of MS and includes few patients presenting severe disability; in any case, the good results obtained for this type of population clearly demonstrate the value of the MoCA as a screening tool for CI associated with MS at early stages of the disease.

In conclusion, the MoCA is a valid and reliable tool for screening for CI in patients with MS. The scale presents excellent diagnostic yield. CI as measured with the BNB is independently associated with level of schooling and degree of disability.

Conflicts of interest

The authors have no conflicts of interest to declare.

Acknowledgements

We are grateful to all our patients for their selfless participation in the study.