Comparative analysis between the MMSE and the RUDAS for dementia screening in low educated people in a Spanish psychogeriatric clinic

Abstract

Background and objectives

The Mini-Mental State Examination (MMSE) remains the most widely used test for the screening of dementia, but its limitations in low educated people are well known. This justified the development of new scales aimed at rooting out any socio-cultural bias, such as the Rowland Universal Dementia Assessment Scale (RUDAS). The aim of this paper is to compare the accuracy of the Spanish RUDAS and the MMSE for the diagnosis of dementia in a population with low level of education.

Methods

In a Psychogeriatric Unit a total of 97 outpatients were administered the RUDAS (by blinded examiners) and the MMSE (by an expert clinician, blind to the RUDAS results).

Results

35 of the 97 subjects received the diagnosis of dementia. The area under the ROC curve (AUC) for the RUDAS, 0.901 (IC 95% 0.84–0.96) was similar to MMSE AUC 0.889 (IC 95% 0.82–0.95). The ideal cut-off point for the RUDAS was 21/22 with 94.3% sensitivity and 72.6% specificity. The “best” cut point for the MMSE was 16/17, lower than the standard 23/24, with 85.7% sensitivity and 77.4% specificity.

The MMSE correlated with educational level (r = 0.432, p < 0.01), but the RUDAS did not (r = 0.087; n.s.).

Conclusions

The RUDAS was not only as accurate as the MMSE for the screening of dementia, but also, it was found to be free of biases associated withed the education level. Hence, the RUDAS seems to be a more adequate test for dementia screening in our cultural context than the MMSE. These results should be replicated in a primary care setting.

Introduction

Over the last few years, dementia research has received much attention because of the increased prevalence and evidence showing some efficacy of treatments in the early stages of the illness. Indeed, early detection is a key point in the management of dementia, which concerns a variety of professionals, especially those from primary care setting. Hence there is a need for the validation of simple, short and portable instruments, which can be easily administered in a reasonable time interval for the diagnosis of dementia.

Widely used instruments for assessing severity of dementia such as the Mini-mental State Examination (MMSE)¹ yielded too many false positives in undereducated population groups, the Galician elderly being a representative example.² Multiple previous reports addressed the socio-cultural biases of the MMSE^3–5 by establishing norms for specific population groups^6–8 or by selecting those items demonstrating better accuracy.⁹ Other strategies reduced the inter rater variation¹⁰ or designed algorithms in relation to other diagnostic tools.^4,11

Within this context, over the last few years some groups have designed new screening tests intended to eradicate these biases,^12,13 such as the Rowland Universal Dementia Assessment Scale (RUDAS).¹⁴ The RUDAS is an instrument developed in Australia to clarify dementia diagnosis in those with low level of education, thus taking into account a wide variety of linguistic and cultural variations.¹⁴

The RUDAS has been directly compared with the MMSE, with no relevant differences in their accuracy.¹⁵ The lack of biases due to educational level was demonstrated by the authors of the original test in its comparison with MMSE,¹⁵ although a validation study performed in India failed to replicate this result.^16,17 A systematic review and meta-analysis of studies comparing the psychometric properties of the RUDAS in different countries with different standards and cognitive tests confirmed the correlation between the scores of the RUDAS and the MMSE, i.e. they measured the same construct. Also, the RUDAS was found to be less affected by language and education level than the MMSE.¹⁸

The aim of this study was i) to investigate the accuracy of the Spanish version of RUDAS in comparison with the MMSE for diagnosis of dementia and ii) to test whether the RUDAS was associated with the education level, which is a classic bias, and therefore limitation, of the MMSE in our clinical setting. Based on the results from the authors of the original validationstudy,¹⁵ we hypothesized that i) there will no be differences in accuracy between both tests and ii) that the RUDAS will not correlated with the education level. This may have important implications on clinical practice, particularly in the primary care setting, where the RUDAS could be an alternative screening dementia tool, for the low educated elderly population living in our catchment area, Galicia, which is a region in the Northwest of Spain. The Galician population is widely scattered, two-thirds of Galician inhabitants live in the rural area 22% of the 2.7 million inhabitants are older than 65 years and the level of education amongst this elderly population remains still low. Most of the Galician elderly, who reside in rural areas, received a very irregular education: instead of regularly attending school until they were twelve or fourteen years old, they tended to miss school because of the long distances they had to cover and because they had to collaborate in farm work during poverty spells. These characteristics of our population and the limitations of the MMSE in our cultural context¹⁹ justified the need for adapting new instruments aimed to overcome these biases.

Method

Study participants

The sample comprised of outpatients receiving care at the Psychogeriatric Unit of the CHUS University Hospital (Santiago de Compostela, Spain). This publicly-funded service is covers a population of around 500,000 inhabitants residing in the area surrounding Santiago de Compostela. Patients over 65 years old were referred by general practitioners and other the Mental Health Services to receive specialized care by an multisciplinary team formed of two psychiatrists, a psychologist and a social worker.

Over the above RUDAS validation study, 97 patients who were administered the MMSE and RUDAS on the same day were recruited. Of note, 18 of the 115 subjects included in the original validation study were excluded since MMSE and RUDAS had not been administered at the same outpatient appointment. All the subjects were over 65 years and suffered from a variety of mental disorders, including frequent memory loss complaints. Informed consent was obtained from either the patients or their main carers. Patients with unconfirmed dementia at the initial assessment and those who were clinically unstable were also included.

Only patients with severe physical limitations (such as hemiparesia, severe hearing loss or blindness) were excluded. Participants consented to the test (or their relatives if the patient lacked capacity) and collaborated with the study procedure.

Description of the test

The RUDAS total scores range from 0 to 30, lower scores indicating more impaired cognition. In particular, six cognitive domains are assessed, including memory (up to four items from a grocery list to be recalled, each of which is scored from 0 to 2, hence 8/30), body orientation (up to 5 points over 30 from 8 items, i.e. over 5 correct items are rated 5 instead of 8), visuo-spatial praxis (which consists of drawing a cube, which can be evaluated from 0 to 3/30), motor praxis (by copying a hand exercise initially displayed by the examiner, which is rated up to 2/30), judgement (up to 4 points out of 30 can be given depending on the patient's answer to a question enquiring about how to cross the road, i.e. his/her road safety awareness) and language (namely, verbal fluency, by naming as many animals as possible within 1 min, with a maximum of 8/30). Only the word “tea” from the grocery list was changed for “coffee”, which is more common in our region. The RUDAS was validated in a sample of outpatients seen at a Psychogeriatric Unit showing good psychometric properties and being easy to administer to, and well accepted by, interviewee.^20,21

The MMSE version sued in this study had been validated in the Galician population²² as part of the diagnoses protocol of the Psychogeriatric Unit, which was an adaptation of the Portuguese validated version.²³ This was decided due to the excessive literality of most of the Spanish versions and the bilingualism of Galician population, in which Spanish and the native language, which is rather similar to Portuguese, are equally used. This version ranges from 0 to 30 and it includes the ‘minus 7’ substractions and world spelling.

Procedure

The RUDAS was administered by two medical students (RRR and JDLM) who were blind to the patient diagnosis. They were trained in the application of the instrument, including the support with written and audiovisual material provided by the authors. An expert clinician (RMA), who was blind to the results of the RUDAS, administered the MMSE as part of the service comprehensive diagnostic protocol which included several clinical interviews with the patient and their caregiver(s), with input from the multidisciplinary team in order to complete the diagnosis and organize the care plan. The assessment of the severity of dementia was measured by the Clinical Dementia Rating Scale (CDR)²⁴ and the Reisberg's Global Deterioration Scale (GDS).²⁵ A diagnosis of dementia was made according to the ICD-10 criteria. With regard to those with undiagnosed dementia at the initial assessment, it should be noted that, consistently with the psychogeriatric unit approach, these patients were followed up as part of the routine care (i.e., no special follow-up was delivered after the study inception).

Statistical analyses

Receiver operating characteristic (ROC) curve analysis compared the accuracy of the RUDAS and the MMSE regarding the diagnosis of dementia. The optimal cut-off point for the two instruments was determined, and sensitivity, specificity, positive and negative likelihood ratios were calculated at different cut-off points.

The association between the scores on the two instruments, the education level and the scores on the scales of severity of cognitive impairment was investigated using Spearman's correlation coefficient.

In order to test differences between subject groups, non-parametric tests were used such as Chi-square Mann–Whitney test and Wilcoxon's rank sum tested for comparisons between two groups and the Jonckheere–Terpstra test when comparisons involved three or more groups (assuming to have a natural order).

Data were recorded and analyzed using the SPSS v.13.0 statistical package.

Results

The RUDAS was easily administered and widely accepted within a time ranging from 7 to 20 min (mean = 12.6; median = 12.0; std = 2.9).

Subject characteristics

The demographic and clinical characteristics of the 97 subjects are summarized in Table 1. There was a preponderance of low education level, and 57.8% had not completed their primary education (less than 6 years of education). Thirty-five subjects (36.1%) were diagnosed with dementia and 10 individuals (10.3%) met criteria for mild cognitive impairment. No difference was found between demented and non-demented patients in education level (z = −0.8, p = 0.42) or gender (χ² = 0.12, p = 0.73). Women had a lower level of education (z = −2.63, p = 0.009).

Table 1.

Demographic and clinical characteristics of the whole sample and subsamples.

	Whole sample (n = 97)		Demented (n = 35)		No demented (n = 62)		Statistic	p value
	Mean ± SD	Range	Mean ± SD	Range	Mean ± SD	Range
Age	77.9 ± 5.9	66–92	80.2 ± 5.6	69–92	76.5 ± 5.6	66–90	t = 3.1	0.003
	N	%	N	%	N	%
Female	74	76.3	26	74.3	48	77.4	χ^2 = 0.12	0.73
Rural environment	65	67.0	22	62.9	43	69.4	χ^2 = 0.43	0.51
Education (years)
0 (Illiterate)	15	15.5	4	11.4	11	17.7	Z = −0.8	0.42
1–5	41	42.3	19	54.3	22	35.5
6–8	27	27.8	9	25.7	18	29
9–12	12	12.4	3	8.6	9	14.5
13 or higher	2	2.1	0	0	2	3.2
Memory complaints	66	68	27	87.1	39	62.9	χ^2 = 9.95	0.007
GDS
1 (no cognitive decline)	19	19.3	0	0	19	30.6	Z = −8.06	<0.0001
2 (very mild cognitive decline)	29	29.9	0	0	29	46.8
3 (mild cognitive decline)	17	17.5	4	11.4	13	21
4 (moderate cognitive decline)	12	12.4	11	31.4	1	1.6
5 (moderately severe cognitive decline)	15	15.5	15	42.9	0	0
6 (severe cognitive decline)	5	5.2	5	14.3	0	0
CDR
0 (no dementia)	39	40.2	0	0	39	62.9	Z = −8.53	<0.0001
0.5 (questionable dementia)	23	23.7	0	0	23	37.1
1 (mild dementia)	15	15.5	15	42.9	0	0
2 (moderate dementia)	17	17.5	17	48.6	0	0
3 (severe dementia)	3	3.1	3	8.6	0	0
Diagnosis ICD-10
F00 Alzheimer's disease	18	18.6	18	51.4	0	0
F01 vascular dementia	4	4.1	4	11.4	0	0
F00.2 dementia mixed type	10	10.3	10	28.6	0	0
F02 F03 other dementias	3	3.1	3	8.6	0	0
F06.7 mild cognitive disorder	10	10.3	0	0	10	16.1
F2 psychotic disorders	9	9.3	0	0	9	14.5
F3 mood disorders	24	24.7	0	0	24	38.7
F4 neurotic disorders	8	8.2	0	0	8	12.9
F6 personality disorders	6	6.2	0	0	6	9.7
No mental disorder	5	5.2	0	0	5	8.1
	Median	Q1–Q3	Median	Q1–Q3	Median	Q1–Q3
RUDAS	21	17–24.5	17	10–19	23	21–25	Z = −6.55	<0.0001
MMSE	17	14–22	13	9–16	20	17–24	Z = −6.35	<0.0001

There were no statistical differences in age, gender, education or main diagnosis between the 18 excluded subjects from the validation study and those included in the analysis (data available upon request).

ROC curves

The area under the ROC curve (AUC) for the RUDAS was 0.901 (CI 95% 0.84–0.96), hence similar to the MMSE AUC, which was 0.889 (CI 95% 0.82–0.95) (Fig. 1). The best cut-off point for the RUDAS was 21/22 with 94.3% sensitivity and 72.6% specificity. The best cut-off point for the MMSE was 16/17, thus much lower than the standard 23/24, with 85.7% sensitivity and 77.4% specificity. At the standard cut-off point the specificity of the MMSE was as low as 29% (Table 2).

Figure 1.

ROC curve for the RUDAS and MMSE in 97 subjects.

Table 2.

Sensitivity, specificity, positive LR and negative LR for MMSE and RUDAS at different cut-off points (n = 97).

		Cut-off point
		23/24^a	22/23	21/22	20/21	19/20	18/19	17/18	16/17^b
MMSE	Sensitivity/specificity	100/29	100/29	100/40.3	97.1/46.8	97.1/53.2	91.4/64.5	88.6/71	85.7/77.4
MMSE	+LR/−LR	1.4/0	1.4/0	1.7/0	1.8/0.06	2/0.05	2.6/0.13	3/0.16	3.8/0.18

		Cut-off point
		23/24	22/23^a	21/22^b	20/21	19/20	18/19
RUDAS	Sensitivity/Specificity	94.3/46.8	94.3/58	94.3/72.6	88.6/77.4	80/85.5	71.4/88.7
RUDAS	+LR/−LR	1.8/0.12	2.2/0.09	3.4/0.08	3.9/0.14	5.5/0.23	6.3/0.21

a

Standard cut-off points.

b

Selected cut-off points after the analysis of ROC curves.

Likelihood ratios (LR) at different cut-off points are also presented in Table 2. RUDAS scores less than 22 multiplied the pre-test odds of dementia by 3.4, while higher scores divided the pre-test odds by 5.5 (negative LR 0.18). MMSE scores less than 17 multiplied the pre-test odds of dementia by 3.8, while higher scores divided the pre-test odds by 12.5 (negative LR 0.08). At the published cut-off point for the MMSE (23/24), a positive test multiplied the pre-test odds for dementia by only 1.4, while for RUDAS considering cut-off point at 22/23, positive test multiplies pre-test odds by 2.2.

Correlations

The RUDAS and MMSE scores correlated significantly (r = 0.707, p < 0.0001). RUDAS scores were higher than MMSE scores in subjects with a low level of education and the opposite was true in subjects with higher levels of education. Thus, the mean and standard deviation of the difference between RUDAS–MMSE scores in the different educational level groups were: illiterate 5.5 ± 3.6; less than 6 years of education 3.4 ± 4.3; 6–8 years 0.7 ± 3.8; 9–12 years −0.33 ± 2.8 and more than 12–1.0 ± 0.0.

Both RUDAS and MMSE correlated with age, GDS and CDR (Table 3). The MMSE correlated with education level (r = 0.402, p < 0.0001), but the RUDAS did not (r = 0.088, p = 0.39).

Table 3.

Correlations (Spearman's coefficient) in 97 subjects.

	MMSE	Education	Age	GDS	CDR
RUDAS	0.707^**	0.088	−0.384^**	−0.722^**	−0.739^**
MMSE		0.402^**	−0.385^**	−0.719^**	−0.728^**
Education				−0.042	−0.134

**

p < 0.01.

Influence of age, education and gender

No significant gender differences were found for the RUDAS (20.65 vs. 19.80, p = 0.67) but a trend towards lower mean score in women was observed for the MMSE (19.65 vs. 16.95, p = 0.06).

The Jonckheere–Terpstra test was used for comparisons between low educated females (n = 49), low educated males (n = 7), highly educated females (n = 25) and highly educated males (n = 16). Significant differences were found for the MMSE (J–T statistic = 3.64, p < 0.0001), but not for the RUDAS (J–T statistic = 0.7, p = 0.49). Fig. 2 shows these results as a box-plot graphic.

Figure 2.

Comparison between RUDAS and MMSE results according to gender and education.

Effects of age, gender and education on the performance of the two screening tools were also evaluated using logistic regression (Table 4).

Table 4.

Regression coefficients (B) in linear models for scores of RUDAS and MMSE (n = 97).

Variable	RUDAS			MMSE
	B	95% CI	p-value	B	95% CI	p-value
Age	−0.359	(−0.547, −0.170)	<0.0001	−0.365	(−0.552, −0.179)	<0.0001
Female gender	1.053	(−1.521, 3.628)	0.811	−0.148	(−2.814, 2.517)	0.912
Schooling	0.768	(−0.245, 1.782)	0.136	2.475	(1.432, 3.518)	<0.0001

False positives

The rate of false positives was 27.4%, which equates to 17 subjects, of which 11 had borderline dementia (CDR = 0.5) and 13 received a score equal to or higher than 2 in the GDS. Within the subgroup of 6 subjects who did not show any sign of dementia (CDR = 0), four participants had low or zero scores at the memory item (data that was not consistent with clinical evaluation) and two scored low in praxias. The only subject within this 6-subject subgroup who did not suffer from any mental illness had a RUDAS score of 21.

In comparison with true negatives (subjects correctly classified as no dementia sufferers), false positives were significantly older than (mean age 78.9 vs. 75.6, p = 0.022), but there were no statistical differences regarding gender or schooling. False positives scored lower than true negatives in the RUDAS (median 19 vs. 25, p < 0.0001) and also in the MMSE (median 17 vs. 21, p = 0.0009).

Discussion

The RUDAS was easy to administer and well accepted by a sample of outpatients from our Psychogeriatric Unit, hence within a real-world clinical context. Throughout the duration of this research project, the two medical students who administered the RUDAS (RRR and JDLM) also administered the MMSE to these or other patients (data not included in this paper) and found the RUDAS to be easier to learn and administer than the MMSE.

The RUDAS is scored within a 0–30 point scale, allowing a direct comparison with the MMSE, which is the most widely used screening test for dementias. The RUDAS was translated into Spanish with minimal variations in its structure or in the wording of the questions, unlike the MMSE, which required an adaptation²⁶ and for which literal translations led to biases (for example, in one of the Spanish versions available, when the sentence “no ifs, ands or buts” is literally translated into Spanish). The MMSE had also culture-related biases due to its content, including tasks like spelling words or doing substractions by 7 s, which are items with poor relevance to some subgroups of people. Such a bias was not replicated for the RUDAS except for the item of judgement for which few patients cognitively intact gave an optimal answer. For the MMSE there is consensus regarding the need for specific adaptations in specific cultures. In this regard, the first Spanish version of the 35-item Mini-Mental State Examination had to be redesigned and re-validated as a 30-point-scored version.^26,27

The area under the ROC curve (AUC) for the RUDAS was similar to the AUC for the MMSE with regard to the diagnosis of dementia. Hence, the Spanish version of RUDAS appears to be at least as accurate as the MMSE for the screening of dementia. In our study, the Spanish version of RUDAS showed a higher sensitivity, but a lower specificity in comparison with the original test (72.6% vs. 98%) and the ideal cut-off point decreased from 22/23 to 21/22.¹⁴ These differences could be explained because of a) the smaller proportion of severe cases of dementia in our sample, and b) the origin of the sample, i.e., psychogeriatric outpatients suffering from a range of mental disorders, with frequent presentations with memory loss complaints and cognitive decline which does not meet criteria for dementia. In the case of the MMSE the change in the optimal cut-off point was even greater and it could not be explained exclusively because of the clinical origin of participants. In our sample, in contrast to the study by Iype et al.,¹⁶ which explained the lower specificity of the RUDAS on the basis of its association with the educational background, we postulate illiteracy as the putative factor underlying the lower specificity of the MMSE, while it may be unrelated to the RUDAS. Indeed, this decrease in scores on the MMSE was found in studies carried out in community samples in Galicia.^2,28

In our sample the scores on the MMSE correlated with education level, although such a correlation was not replicated for the RUDAS scores. While the MMSE appears to be influenced by gender, there were no gender differences in the RUDAS scores, although the education level may have acted as a confounder. When the sample was split up in groups by gender and education background significant differences emerged for the MMSE, which were not revealed for the RUDAS.

In our study, following a conservative procedure to classify dementia cases vs. non-cases, with a cut-off point for RUDAS of 21/22, an excellent sensitivity (94.3%) was found. However, this resulted in a rate of false positives of 27.4%, which may be clinically unacceptable when using the RUDAS for the screening of dementia. However, when analysing the clinical characteristics of the 17 false positives, most of them (11) were clinically diagnosed with potential dementia/mild cognitive impairment (CDR = 0.5). We suggest that considering these 11 subjects as a possible “case” of dementia does not represent a screening failure. Rather, these cases required a more thorough clinical assessment to rule out the onset of dementia and/or required longer-term follow-up. As a matter of fact, we have confirmed that of these 11 subjects, 3 patients developed dementia within the first year of follow-up and one more individual fulfilled dementia criteria in the second year. From this pragmatic clinical approach, the rate of “real” false positives drops to a mere 9.6% (6/62). In other words, specificity rises to 90.4%, positive LR rises to 9.8 and negative LR decreases to 0.06.

Moreover, we have sought to understand why 6 subjects with a CDR = 0 obtained a low score at the RUDAS in spite of the fact that they did not present with clinical manifestations of cognitive impairment. We can only hypothesize that for most of them their non-dementia/cognitive impairment psychiatric diagnosis may have influenced their cognitive performance at the time of being administered the test.

Strengths and limitations

The clinical context and the model of work of our Psychogeriatric Unit, where the subjects for this study were recruited, which includes a comprehensive assessment, provides reliability for the clinical diagnosis of dementia (ICD criteria), the gold standard in this study. Patients received input from the multidisciplinary team (a psychiatrist, a psychologist and a social worker) who arranged the care plans and prolonged the follow-up periods for as long as needed. This procedure reduced the rate of false positives and false negatives, regarding a clinical diagnosis of dementia (which was the gold standard in this study), which is of particular concern in cases with no severe cognitive impairment at first contact (potential false negatives otherwise).

Also, fewer cases of advanced dementia (score on CDR = 3) were included in comparison with the similar study of Storey et al. (8.6% vs. 31.8% of all dementias) and the prevalence of potential dementia (CDR = 0.5) in the total sample was much higher in our study (23/97 = 23.7% vs. 6/111 = 5.4%). Furthermore, our sample had a prevalence of dementia cases of 36.1% (35/97), while the prevalence in the above study was 56.8% (63/111), and therefore, closer to 50%, a level of prevalence at which screening instruments work better.²⁹ In other words, our study made the screening instruments subject to a more challenging test, which interestingly the RUDAS has successfully passed, yielding good to very good psychometric parameters. Indeed, we found a smaller difference in our sample between the median of the scores on the MMSE in demented (13, Q1: 9–Q3: 16) and in non-demented (20, 17–24) patients than in previous studies. This was also observed for the RUDAS (17, 10–19 vs. 23, 21–25).^15,16

However, this clinical context may be also a constraint when generalizing our results. Thus, most of the non-demented subjects were psychiatric outpatients. On the one hand, this means that the sample was likely to be representative. On the other hand, our findings may not generalize to the general population, particularly to those patients with cognitive impairment or dementia who only receive primary care. Specifically, participants in this study may have been recruited due to risks or complexities which do not apply to all patients with dementia.

Nevertheless, the Spanish RUDAS confirmed the lack of biases associated with education level or gender, thus behaving as a more adequate test for the screening of dementia in our population. These promising results add new evidence to the usefulness of RUDAS for screening dementia when the risk of socio-cultural biases is high, particularly in people with low education level. Future research in primary care and in the community settings is needed in order to replicate these results and establish the most optimal cut-off point of the RUDAS. Meanwhile, based on our results we would recommend the use of RUDAS over the MMSE for the screening of dementia in people with low levels of education such as the Galician population.

Description of authors’ roles

R. Mateos conceived and designed the study. He was responsible for the patients’ diagnostic process; carried out the MMSE, supervised data collection and data analysis and assisted with writing the article. R. Ramos-Ríos and J. López-Moríñigo collected data and administered the RUDAS to the whole sample. R. Ramos-Ríos drafted the paper and carried out the statistical analysis. All authors read and approved the paper.

Conflict of interest declaration

None.

Acknowledgments

For this study, R. Ramos-Ríos and J. López-Moríñigo, medical students at the time of collecting the data, received financial support by a Collaboration-Grant from the Spanish Ministry of Education during the 2003–2004 and 2004–2005 academic years, respectively.

Authors thank Jeff Rowland and Joella Storey for their support in the validation of the Spanish version of the RUDAS.