Test Your Memory is sensitive to cognitive change but lacks prospective validity

Abstract

Objective

To determine the prospective validity of Test Your Memory (TYM) and its sensitivity to change in cognitive state.

Type of study

Longitudinal prospective.

Methods

This longitudinal prospective study followed 71 patients with subjective cognitive symptoms and 48 with mild cognitive impairment for a mean time period of 35.2 ± 15 months. Subjects did not have dementia or depression at the beginning of follow-up and each participant was given the TYM at least two times. A psychometric threshold was established to determine presence of a cognitive deficit (z-score ≤ 1.5 on at least one cognitive domain) and the Disability Assessment for Dementia scale was used to ensure full functional ability. The criterion for deterioration was a change in the stage on the Global Deterioration Scale.

Results

Sixty-one patients remained cognitively stable and 58 worsened. There were no differences between them with respect to sex, educational attainment, the initial stage on the GDS, or the score on the first TYM. Subjects who worsened were older than those who did not. The TYM increased an average of 0.04 points per month in patients who remained stable or improved (95% CI, −0.01 to 0.08) and decreased an average of 0.14 points per month in those whose condition worsened (95% CI, −0.19 to −0.09). Subjects with mild cognitive impairment who worsened displayed a sharper loss of TYM points than did subjects with subjective cognitive symptoms.

Conclusions

While the TYM lacks prospective validity, it is sensitive to changes in cognitive state.

Resumen

Objetivo

Determinar la validez prospectiva del Test Your Memory (TYM) y su sensibilidad al cambio de estado cognitivo.

Tipo de estudio

Longitudinal prospectivo.

Métodos

Se siguió a 71 enfermos con síntomas cognitivos subjetivos y 48 con defecto cognitivo leve durante un periodo de tiempo medio de 35,2 ± 15 meses. Los sujetos no tenían demencia ni depresión al comienzo del seguimiento y a todos se le administró al menos dos veces el TYM. Se aplicó un criterio psicométrico para determinar la existencia de defecto cognitivo (≤ 1,5 puntos z en al menos un dominio cognitivo) y la escala Disability Assessment for Dementia para asegurar la indemnidad funcional. El criterio de empeoramiento fue el cambio de estadio en la Escala de Deterioro Global.

Resultados

Sesenta y un enfermos se mantuvieron cognitivamente estables y 58 empeoraron. No hubo diferencia entre ambos grupos con respecto al sexo, el nivel de instrucción, el estadio inicial de la GDS o la puntuación en el primer TYM. Los sujetos que empeoraron tenían más edad. El TYM aumentó un promedio de 0,04 puntos por mes en los pacientes estables o que mejoraron (IC95% −0,01 a 0,08) y disminuyó una media de 0,14 puntos por mes en los que empeoraron (IC95% −0,19 a −0,09). De los sujetos que empeoraron, en los que tenían defecto cognitivo leve la disminución de la puntuación del TYM fue más abrupta que en aquellos con síntomas cognitivos subjetivos.

Conclusiones

EL TYM carece de validez prospectiva pero es sensible al cambio de estado cognitivo.

Introduction

Subjective cognitive symptoms (SCS) are common among subjects older than 60. These symptoms also imply a higher probability of progressing to cognitive impairment or dementia.^1–3 Furthermore, some studies have reported an association between certain cognitive symptoms and performance on cognitive tests,^4,5 whereas other studies do not.⁶ In fact, ‘subjective memory complaints’ may be a confusing term that indicates either overestimation (patients with depression) or underestimation of actual difficulties (anosognosic patients).^7–9 To avoid confusion about the nature of the complaint, some researchers suggest removing subjective memory loss from among the diagnostic criteria for mild cognitive impairment (MCI), or else redefining it as a complaint reported by the patient and corroborated by a reliable informant.^7,10 Furthermore, the annual incidence rate of MCI in subjects older than 65 ranges between 5% and 8%¹¹; developing MCI has been linked to age and presence of vascular risk factors.¹² Regarding progression of MCI to dementia, the yearly conversion rate may be as high as 10%.¹³ Subjects with objective cognitive impairment and SCS are more likely to develop cognitive decline.¹⁴ However, using neuropsychological indicators is not an effective way of predicting MCI progression to dementia.¹⁵ Results might be improved by including a combination of cognitive, imaging, and biochemical markers in the algorithm.^13,16 In addition, up to one fourth of the patients who meet MCI criteria in the initial evaluation improve during follow-up. Therefore, their risk of exacerbation is no greater than in subjects with no cognitive impairment.¹⁷ Based on data extracted from the multi-centre longitudinal study The Alzheimer's Disease Neuroimaging Initiative, researchers developed an easy-to-calculate combined index to predict progression from MCI to Alzheimer disease.¹⁸ This index, however, incorporates indicators of functional deficit. As such, the algorithm may be intrinsically flawed since it applies a predictive rule which includes criteria for the dementia which it intends to predict, and which are incompatible with MCI.

In clinical practice, in addition to checking for objective symptoms in patients attending a neurology clinic due to suspected cognitive dysfunction, doctors must also predict whether each patient's condition will deteriorate, improve, or remain stable. Prognosis clearly depends on the disease that led to the consultation, but many of these patients do not receive a final diagnosis and will be categorised as having SCS or MCI with no apparent functional deficit. After completing the initial evaluations, prognosis remains uncertain in most cases. Clinical indicators or other biomarkers are therefore necessary to help predict outcomes with some certainty.¹⁹ Since some studies have observed that certain short cognitive tests or neuropsychological tests help predict cognitive decline in subjects with or without MCI,^16,18,20 our study aims to assess whether Test Your Memory (TYM)²¹ may be useful to this end. In other words, we wish to assess its prospective validity in subjects without dementia and depression and determine if TYM is sensitive to cognitive change.

Subjects, material, and methods

Subjects and procedure

We selected patients who attended our general neurology clinic due to symptoms possibly suggesting cognitive impairment and who were not diagnosed with dementia or depression at the time of the initial evaluation. Symptoms were not recorded in a structured manner but the most frequent were as follows: forgetting things from one moment to the next; difficulty following conversations, understanding instructions, or remembering shopping lists; and problems remembering recent events or scheduled appointments. Patients also presented unexpected episodes of disorientation in relatively familiar places. In line with the most recent criteria for MCI,¹⁰ SCS were considered to be those expressed by patients and confirmed by reliable informants. In our clinic, we suggest long-term follow-up on subjects expressing cognitive complaints even when no deficits can be confirmed initially. The reason is that subjective cognitive impairment may be present for several years before any objective evidence is apparent.^1,22 All subjects underwent neurocognitive assessment. Patients had their history taken and completed a neurological examination that included a study of the higher functions based on formal neuropsychometric tests repeated every 1 to 2 years. These tests, grouped by cognitive domains, included at least the following: attention (forward and backward digit span²³); language (semantic verbal fluency [animals/min²⁴] and Boston naming test²⁵); learning and episodic verbal memory (Free and Cued Selective Reminding Test:²⁶ free immediate recall [sum of recall attempts 1–3], immediate recall with category clues [sum of recall attempts 1–3], deferred free recall, and deferred recall with category cues); executive performance (phonemic verbal fluency [words beginning with ‘P’/min]²⁴, Trail Making Test parts A and B²³, and time to copy the Rey-Osterrieth complex figure²⁶); and visuospatial abilities (copy and immediate recall [3 min] of the Rey–Osterrieth complex figure²⁶). Mean z score values from the neurocognitive assessments were calculated to provide a representative value for each domain. To avoid excessive influence on domain scores by the outliers observed in some tests, we decided to assign minimum and maximum values of −3 and +3 z scores to any z scores falling outside that interval. Depending on patients’ symptoms and characteristics, additional tests may have been administered but not included in the cognitive domain scores. Subjects were also administered the 15-item version of the geriatric depression scale.²⁷ Relatives and household members completed the Neuropsychiatric Inventory Questionnaire²⁸ and Disability Assessment for Dementia²⁹ questionnaire, which has had a validated Spanish version since late 2013.³⁰ Furthermore, 88 subjects were administered the TYM twice, 27 three times, 3 four times, and one, 5 times to define progression of this parameter in the patient over time.

Presence of cognitive impairment was determined strictly according to neuropsychometric criteria (≤−1.5 SD in at least one cognitive domain). Normal functional capacity was determined based on interviews with relatives or household members and confirmed by a score higher than 92% on the Disability Assessment for Dementia scale.³⁰

Decline was defined as a change in the stage of the Global Deterioration Scale (GDS) during follow-up³¹: GDS 2→GDS 3 or higher, or GDS 3→GDS 4 or higher. We excluded subjects with disorders preventing them from filling out the TYM (i.e., poor visual acuity or debilitating tremor), those with focal brain alterations or a Hachinski ischaemic score above 3.³² Subjects with history of severe head trauma, polymedicated subjects (7 or more active drugs, excluding vitamins, mineral nutrients, microminerals, essential elements, and nutritional supplements), and patients with systemic psychiatric or somatic disorders that could affect their cognitive ability were also excluded. In line with the above, we did not include patients with a geriatric index of comorbidity greater than class II.³³

After the selection and classification procedure, each subject was included in one of 2 mutually exclusive groups: with SCS or with MCI. Patients in the SCS group exhibited no objective evidence of illness (symptoms were described by the patient, observed by a reliable informant, or both), and were in GDS stage 2.³¹ Patients in the other group were diagnosed with MCI¹⁰ and in GDS stage 3.

Statistical analysis

We tested hypotheses using bivariate analysis techniques adjusted for the level of measurement and the distribution of the study variables. Since results from multivariate analyses were not more revealing than those obtained with bivariate analyses, we have omitted them for clarity. We could not perform a survival analysis due to the irregular frequency of follow-up visits and the high rates of subjects lost to follow-up due to often unknown causes. In our calculations, we used TYM scores adjusted by education level²¹ as an ordinal variable. An alpha risk of 0.05 was assumed. Statistical analyses were performed using MedCal 14.12.0™.

Results

We monitored 119 patients (71 in GDS stage 2 and 48 in GDS stage 3) during a period of 12 to 60 months following the first consultation (x = 35.2 ± 15 months). During follow-up, 61 subjects remained cognitively stable or improved while 58 presented cognitive decline (Fig. 1). Patients experiencing decline and those who displayed no differences related to sex, years of schooling, baseline TYM score adjusted by education level, and months of follow-up. Patients whose condition worsened were older at the time of the first consultation than those whose condition did not (Table 1).

Figure 1.

Flow chart of subject distribution, diagnosis after follow-up, and main causes for exclusion.

Table 1.

Comparison between stable patients and those whose condition worsened.

	Stable (n = 61)	Declining (n = 58)	P-value
Age at first evaluation (x ± SD)	73.0 ± 7.8	76.4 ± 8.2	<.05^*
Sex
Women	33	34	ns^***
Men	28	24
Years of schooling	11.5 ± 4.4	13.5 ± 4.4	ns^*
GDS
GDS stage 2 (n = 71)	41	30	ns^***
GDS stage 3 (n = 48)	20	28
Baseline TYM, SCS	44.2 ± 3.9	44.0 ± 3.1	ns^***
Baseline TYM, MCI	39.5 ± 4.0	40.6 ± 2.5	ns^***
Baseline TYM, (SCS + MCI)	42.7 ± 4.5	42.3 ± 3.3	ns^***
Months of follow-up, SCS	36.7 ± 14.3	39.7 ± 15.6	ns^*
Months of follow-up, MCI	26.0 ± 13.6	34.4 ± 14.2	<.05^*
Months of follow-up, SCS + MCI	33.3 ± 14.8	37.1 ± 15.0	ns^*
Monthly improvement in TYM score, SCS	0.03 ± 0.15	−0.06 ± 0.13	<.01^***
Monthly improvement in TYM score, MCI	0.04 ± 0.25	−0.23 ± 0.20	<.001^***
Monthly improvement in TYM score, SCS + MCI (95% CI)	0.04 ± 0.19 (0.01: 0.8)	−14 ± 0.19 (−0.19: −0.09)	<.0001^**^/^*

GDS: Global Deterioration Scale; TYM: Test Your Memory; SCS: subjects with subjective cognitive symptoms; MCI: subjects with mild cognitive impairment.

P: statistical significance level.

*

t-test.

**

Mann–Whitney U test.

***

χ².

Broken down by subgroups, subjects with SCS had been in follow-up for longer periods (t-test, P < .05) and had noticeably higher scores on the TYM (44.2 ± 3.6) than subjects with MCI (40.2 ± 3.2) (Mann–Whitney U test, P < .0001). Although 58% of the patients in GDS stage 3 at baseline experienced decline, compared to only 42% of the subjects with GDS stage 2 at baseline, this association was not statistically significant (χ² 2.96; DF 1; P > .05). Of the 28 subjects with GDS stage 3 at baseline whose condition worsened, 21 progressed to Alzheimer disease. Of the 30 patients with GDS stage 2 at baseline who experienced decline, 19 met criteria for MCI at the end of follow-up and 9 met criteria for Alzheimer disease (Fig. 1). In the subgroup analysis, no differences in the TYM score at baseline were observed between patients whose condition worsened and patients who remained stable (Table 1).

Regarding changes in TYM scores throughout the follow-up period, cognitively stable subjects showed a slight increase in the score while subjects in decline lost a mean of 0.14 TYM points per month. The most pronounced drop was observed among patients who initially presented MCI (Table 1).

Discussion

One of the limitations of this longitudinal study is that patients were selected at a single general neurology clinic; therefore we do not know the characteristics of the theoretical population they might represent, the subject selection mechanisms (subjects requesting appointments vs those referred by a doctor), or how the situation might vary in other districts. Strictly speaking, our findings cannot be extrapolated. Furthermore, the selected patients visited because they or their close acquaintances were concerned about manifestations of cognitive symptoms. This being the case, we did not include patients with no cognitive complaints, whether or not any objective cognitive impairment could be confirmed. One debatable aspect of this study is the technique used to classify subjects. Classification is normally performed using information obtained almost exclusively from medical histories and interviews. However, since patients and raters were not blinded at any moment during follow-up, we opted to use a quantitative system, which has a lower risk of interpretation biases in our setting. We acknowledge that using such a system can yield subject groups that are not comparable to those obtained using less strict criteria, as Ganguli et al. have observed.³⁴ Regarding psychometric methods for classification, other debatable points are the definition of the procedure and the selection of the cut-off points applied. Some researchers consider a psychometric study to be abnormal when results on at least one test are lower than the selected cut-off point. Other authors use the same criterion applied to each cognitive domain,³⁵ which yields the same result. The main problem with this procedure is that no adjustments are made for multiple comparisons. In other words, the more tests are administered, the higher the probability of observing abnormal results by chance. We decided to use the most conservative criterion, described in the MYHAT study.³⁴ After grouping the tests by domains, we calculated mean domain scores. Deficit in a cognitive domain was understood to be likely if the mean of all test scores pertaining to that domain was below the cut-off point. When this procedure is used, lower values on one test can be compensated by normal or high values in others. Nevertheless, we believe that this approach is approximately equivalent to adjusting for multiple comparisons, and it also compensates for the fact that a subject can fail a single test for irrelevant reasons. Furthermore, we have not included outliers (z scores outside of −3 to 3) to prevent any single test from disproportionally affecting the domain score. In our opinion, this system is methodologically valid. Selecting different cut-off points to define probabilistic abnormality on a test will yield very different subject groups.³⁵ Our decision to place the cut-off point at ≤1.5 z scores is sufficiently inclusive. This criterion is practically identical to that used by the Florida Alzheimer's Disease Research Centre, and which yielded diagnostic groups that were significantly related to both APOE genotype and MRI ratings of medial temporal atrophy.³⁵

In this longitudinal study, TYM scores at the first consultation did not predict outcomes at 3 years in subjects with SCS or with MCI; on this basis, TYM lacks prospective validity. The high percentage of cognitive decline observed in our series (almost 17% per year) may be due to the negative selection bias typical of specialised clinics, or to the longer progression times in subjects who visit these clinics. In any case, this tendency suggests that cognitive symptoms and MCI, once depression has been ruled out, generally signify a poor prognosis, as other studies have also concluded.¹ One cross-sectional population-based study showed that higher numbers of cognitive complaints predicted lower scores on tests. It also reported that not all cognitive complaints and their objective counterparts are equally weighted. Some symptoms in that study, such as having trouble following a group conversation or navigating familiar surroundings, were stronger indicators of cognitive impairment than others (i.e., forgetting things from one moment to the next).⁴ Decline is likely to be easier to predict in subjects with more symptoms, especially if all symptoms are of a particular type. Since symptom characteristics were not recorded in a structured way, we cannot draw any conclusions on this subject. The cohort study by the University of Kentucky showed that more than half of all subjects reported SCS; this finding was associated with an odds ratio of 2.8 for developing MCI or dementia, compared to asymptomatic patients.¹

In any case, almost half the patients with SCS and about 60% of those who initially presented MCI will experience decline. We also expect half of the patients we examine due to these concerns to worsen within 3 years. However, the TYM score does not improve our predictive capability. If patients are older than 75, the odds are against them. Other researchers have found that female sex, certain vascular risk factors, and scores on selected objective tests predict cognitive impairment. After observing the rates of decay of TYM scores in subjects whose condition deteriorated, we concluded that scores in the SCS group decreased less sharply than scores in the MCI group (a mean loss of one point per year vs 3 points lost per year). However, increases were similar in subjects with SCS and MCI who remained stable or improved. Decline in subjects with MCI is quite pronounced. This coincides with observations by Johnson et al.³⁶ in subjects who deteriorated until they were diagnosed with Alzheimer disease. These subjects, after a prolonged period of slow decrease in performance, experience accelerated deterioration which results in functional deficit. In any case, our findings seem to suggest that although TYM lacks prospective validity, it is somewhat sensitive to cognitive change. However, we must recognise that its calibration is insufficient, as we have reported recently.³⁷

Funding

This study received no public or private funding.

Conflicts of interest

The authors have no conflicts of interest to declare.