Neuronorma Colombia: contributions and methodological characteristics

Abstract

Introduction

Availability of adequate normative data is essential when performing neuropsychological evaluation; good methodological quality of the studies that propose these data ensures that their conclusions are reliable and valid. We present the methodological characteristics of the Neuronorma Colombia Project in order to analyse its contributions and limitations.

Method

We present the characteristics of the normative sample, inclusion and exclusion criteria, statistical analysis, the procedure for obtaining normative data, and the instruments used.

Results

We present graphical profiles of patient performance, based on the Neuronorma Work Unit, to illustrate the interpretation of the results obtained when evaluating patients with the Neuronorma Colombia Battery.

Discussion and conclusions

Our study presents several methodological advantages, such as its multicentre, co-normalised design and the availability of the Neuronorma Work Unit, which allows the creation of graphical profiles of patient performance, a fundamental tool for diagnosis and research. We present the findings of subsequent research based on the proposed normative data, which demonstrate the value of the battery. The contribution of this study is discussed in the context of its immediate background.

Resumen

Introducción

Al realizar una evaluación neuropsicológica es necesario contar con datos normativos adecuados; la calidad metodológica de los estudios que los proponen garantiza que las conclusiones sean confiables y válidas. Se presentan las características metodológicas del Proyecto Neuronorma Colombia, con el fin de analizar sus aportes y limitaciones.

Método

Se exponen las características de la muestra normativa, criterios de inclusión y exclusión, análisis estadístico de los datos, procedimiento para la obtención de datos normativos y los instrumentos utilizados.

Resultados

Se presentan algunos perfiles gráficos de ejecución de casos clínicos, a partir de la Unidad de Trabajo Neuronorma, con el fin de ilustrar la interpretación de los resultados obtenidos al evaluar pacientes con la Batería Neuronorma Colombia.

Discusión y conclusiones

Existen ventajas metodológicas del estudio: su carácter multicéntrico y conormalizado, disponer de la Unidad de Trabajo Neuronorma, que permite obtener perfiles gráficos del desempeño del paciente, herramienta fundamental para el diagnóstico y la investigación. Se presentan los hallazgos de investigaciones posteriores realizadas a partir de los datos normativos propuestos, que son evidencia de la utilidad de la Batería. Se discute el aporte de este estudio, en el contexto de sus antecedentes inmediatos.

Introduction

Neuropsychological assessment is essential in the diagnosis of neurocognitive disorders and requires, among other things, appropriately adapted testing instruments.

In Colombia, studies have been performed to assess the psychometric properties of various cognitive screening instruments (Montreal Cognitive Assessment [MoCA] Test)^1–3 and to establish the influence of age and level of schooling on test performance (Mini–Mental State Examination [MMSE]),^4,5 as well as a validation study⁶ of the 15-item version of the Geriatric Depression Scale⁷ and a psychometric study of the Memory Complaint Scale.⁸ Regarding test batteries, the Consortium to Establish a Registry for Alzheimer’s Disease (CERAD) battery⁹ has been adapted and validated for the Colombian population, and a study was conducted to establish cut-off points for the neuropsychological assessment protocol of the Hospital Universitario San Ignacio memory clinic.¹⁰

The studies cited contribute cut-off points for screening tests,^1,3,8 include small samples, or use large samples but only address the properties of brief cognitive tests.⁹ Another weakness is that the samples used were gathered specifically for each test, and the studies thus do not control for variability in an individual’s performance across different tests. In the light of these limitations, there is a need to adapt instruments and obtain normative data through studies with larger samples, addressing the diagnosis and characterisation of cognitive profiles for different diseases.

Recent studies by Guardia-Olmos et al.¹¹ and Arango-Lasprilla et al.¹² report normative data for 10 neuropsychological tests that are widely used in Latin America. However, these studies present certain methodological limitations, which are discussed below.

The Neuronorma Colombia (NN.Co) project seeks to obtain normative data for the Colombian population aged over 50 years. It is an offshoot of the Spanish Neuronorma project, and was developed through collaboration between the Spanish research team led by Dr Jordi Peña-Casanova and the Colombian team led by Dr Patricia Montañés. The methodology followed was based on that of the Spanish project, which was in turn derived from the methodology used in the Mayo’s Older American Normative Studies (MOANS) project.^13–16 This article presents the methodological characteristics of NN.Co, which seek to address certain limitations identified in previous studies, and highlights their contributions and limitations in the scenario described.

Methods

Inclusion and exclusion criteria

Inclusion and exclusion criteria (Table 1) were adapted from those of the Spanish Neuronorma project.¹⁷ Participants gave written informed consent and had to score higher than the cut-off points established for Colombia in screening tests⁸; each participant was accompanied by an informant at consultations to complete these tests. We confirmed that participants were able to read and write, presented no physical limitations (or that any such limitations were corrected), and had presented stable health status for at least 3 months prior to inclusion.

Table 1.

Inclusion and exclusion criteria.

Inclusion criteria
Mini–Mental State Examination^5	> 24
Geriatric Depression Scale^7	< 6
Memory Complaints Scale^8	≤ 19
Modified Lawton Scale^18	No changes
Neuropsychiatric Inventory^19	No clinically significant psychiatric symptoms

Exclusion criteria
Unwillingness of the individual or their caregiver to participate in the study
Presence of a central nervous system disorder with potential cognitive repercussions
Presence of psychiatric disorders
Clinically significant systemic diseases (hypothyroidism, vitamin B12 deficiency, cardiovascular disease, type 1 diabetes, tertiary-stage syphilis, HIV infection, kidney failure, liver disease)
Current or previous alcohol or drug abuse
Hearing loss and significant amblyopia or vision loss

Procedure

The first phase, adaptation, involved reviewing and modifying the items of the tests included in the Neuronorma battery, selecting instruments, and establishing the order of application. These changes were defined by consensus between neuropsychologists and psychologists in the research team. Subsequently, to define the definitive protocol, a pilot study was conducted with 60 participants who met the selection criteria.^20,21

The second phase, standardisation, consisted in the unification of criteria for administering and scoring tests. Training videos and administration guidelines were developed for raters.

In the third phase, evaluation, inclusion and exclusion criteria were applied and the full battery was administered.

Each participant’s data were coded and stored using the Test Barcelona Workstation platform, provided by Dr Peña-Casanova. The platform generates a performance profile for each participant using the Spanish normative data.

Test battery

Screening tests were selected according to their psychometric properties and their use in the Colombian population, as mentioned in the section above on inclusion criteria. The NN.Co battery comprises 12 instruments (Table 2).

Table 2.

Tests included in the Neuronorma Colombia battery.

Process	Test
Language/naming	Boston Naming Test^24
Verbal memory	Free and Cued Selective Reminding Test^10,25
Language/fluency	Semantic (animals) and phonological (“P” words) verbal fluency tests^26,27
Inhibitory Control	Stroop Color and Word Test^28
Visuoconstructive skills	Rey-Osterrieth Complex Figure test (copy and recall)^29,30
Information processing speed	Symbol Digit Modalities Test^31
Sustained and alternating attention	Trail Making Test (parts A and B)^32
Planning	Tower of London test^33
Verbal comprehension	Token Test^34
Verbal working memory	Digit span (forward and backward; WAIS III)^35
Visual working memory	Corsi Block-tapping Test^36
Cognitive flexibility	Wisconsin Card Sorting Test^22

WAIS III: Wechsler Adult Intelligence Scale-III.

We excluded the Judgement of Line Orientation and Visual Objects and Space Perception Battery, and added the Wisconsin Card Sorting Test.²² The memory test selected was the adaptation of the Free and Cued Selective Reminding Test used in the Hospital Universitario San Ignacio memory clinic protocol.¹⁰ In the Rey-Osterrieth Complex Figure test, pencil colour was changed each minute to facilitate qualitative analysis of copy type and strategy.²³

Ethical considerations

All raters hold certifications in the protection of human research participants. The project was approved by the ethics committee of the human sciences faculty of Universidad Nacional de Colombia and the institutional committee of each participating institution. All participants freely consented to participate after being informed about the project, confidentiality, scope, and potential risks and benefits.

Normative data collection

Statistical analysis was performed using the IBM SPSS Statistics Base software (version 22.0). Normative data were obtained according to the protocol of the Spanish Neuronorma project,17 as follows:

• a)

  Overlapping intervals: normative groups were established around midpoints; thus, a single member of the normative sample may appear in more than one group. These groups present different midpoints but overlapping ranges for the variable in question.³⁷ This strategy softens the borderlines between age groups.

  Ten age groups were established, with an age range of 10 years (Table 3). When an individual patient’s performance is compared against the normative data, only one group should be used. For instance, a patient aged 53 years should be compared against normative group 1.

  Table 3.

  Normative groups (N = 438).

  Group	Midpoint age	Age range for midpoint	Age range for norms	Sample size
  1	55	50−56	50−60	142
  2	58	57−59	53−63	160
  3	61	60−62	56−66	176
  4	64	63−65	59−69	172
  5	67	66−68	62−72	171
  6	70	69−71	65−75	163
  7	73	72−74	68−78	146
  8	76	75−77	71−81	116
  9	79	78−80	74−84	84
  10	81+	81−90	77+	50

  Age distribution, midpoint, and age range for midpoint and norms were based on those used for the Spanish Neuronorma project.¹⁷
• b)

  Identification of the effects of age, sex, and level of schooling: these effects were determined using coefficients of correlation and determination between these sociodemographic variables and raw scores in the neuropsychological tests (Table 4). Corrections were applied for scores for which these coefficients are statistically significant.

  Table 4.

  Correlations (r) and shared variance (r²) between raw scores and age and level of schooling.

  	Age		Years of schooling
  Measurement	r	r^2	r	r^2
  BNT	–0.353*	0.12466	0.629*	0.39624
  Token Test	–0.277*	0.07673	0.465*	0.21655
  ROCF copy	–0.424*	0.17955	0.611*	0.37287
  ROCF T	0.430*	0.17131	–0.594*	0.3526
  FCSRT FR1	–0.285*	0.08135	0.283*	0.08006
  FCSRT TFR	–0.414*	0.17131	0.432*	0.18647
  FCSRT TR	–0.298*	0.08861	0.397*	0.15736
  FCSRT DFR	–0.389*	0.15171	0.388*	0.15036
  FCSRT TDR	–0.247*	0.06119	0.367*	0.1345
  ROCF 3MR	–0.378*	0.14285	0.485*	0.23566
  Verbal digit span forward	–0.174*	0.03	0.396*	0.156
  Verbal digit span backward	–0.232*	0.054	0.386*	0.149
  Visual digit span forward	–0.241*	0.058	0.355*	0.126
  Visual digit span backward	–0.298*	0.089	0.406*	0.165
  TMT-A	0.405*	0.164	–0.547*	0.299
  TMT-B	0.396*	0.156	–0.537*	0.288
  SDMT	–0.549*	0.302	0.695*	0.483
  Stroop test part A	–0.406*	0.165	0.507*	0.257
  Stroop test part B	–0.391*	0.153	0.445*	0.198
  Stroop test part C	–0.400*	0.16	0.407*	0.165
  TOL-SS	–0.062	0.004	0.235*	0.055
  TOL-EM	0.217*	0.047	–0.203*	0.041
  TOL-L	0.181*	0.033	–0.052	0.003
  TOL-ET	0.476*	0.227	–0.347*	0.12
  TOL-RT	0.485*	0.235	–0.340*	0.116
  WCST total correct	–0.108*	0.12	0.221*	0.049
  WCST correct categories	–0.095	0.009	0.258*	0.066
  WCST perseverative errors	0.1	0.01	–0.231*	0.054
  WCST non-perseverative errors	0.04	0.002	–0.064	0.004
  Semantic verbal fluency	–0.354*	0.125	0.496*	0.246
  Phonological verbal fluency	–0.295*	0.087	0.565*	0.319

  BNT: Boston Naming Test; DFR: delayed free recall; TDR: total delayed recall; FCSRT: Free and Cued Selective Reminding Test; FR1: free recall trial 1; r: Pearson correlation coefficient; r²: coefficient of determination; ROCF: Rey-Osterrieth complex figure test; ROCF 3MR: ROCF 3-minute recall; ROCF T: ROCF copy time; SDMT: Symbol Digit Modalities Test; TFR: total free recall; TMT: Trail Making Test; TOL: Tower of London test; TOL-EM: TOL extra movements; TOL-ET: TOL execution time; TOL-L: TOL latency; TOL-RT: TOL resolution time; TOL-SS: TOL success score; TR: total recall; WCST: Wisconsin Card Sorting Test.

  *

  P < .05.
• c)

  Construction of normative tables: raw scores are converted into scaled scores (X̄: 10; SD: 3); for each test, cumulative frequencies were obtained for raw scores and a percentile was assigned to each score, according to its location in the sample distribution. This procedure yields the normalised standard score adjusted for age (NSS_A).¹⁷

• d)

  Correction for level of schooling: this correction was calculated according to the formula proposed by Mungas et al.³⁸: NNS_AE = NSS_A – (β* [Educ – 11) (where NNS_AE is the normalised standard score adjusted for age and education, and Educ is the number of years of schooling), correcting scores where necessary for the different levels of education. This procedure adjusts scores based on the difference between the number of years of schooling of the test subject and the mean number of years of schooling in the sample (11 years). The regression coefficients obtained were used to correct scores for level of schooling.

Results

The normative sample included 438 individuals from different administrative regions of Colombia (Cundinamarca, Antioquia, Casanare, and Boyacá). Participants were gathered by convenience sampling in the population aged 50 to 90 years. The initial sample included 576 individuals; 138 were excluded after application of the inclusion and exclusion criteria, with the most frequent reasons being presence of depressive symptoms and significant subjective memory complaints. Excluded individuals were advised to seek medical attention. The sample was stratified according to the sociodemographic variables age, sex, and level of schooling. The latter variable was defined as the number of years of formal education.

Table 5 shows the distribution of the final sample according to these variables. Participants were recruited in an open call for participants advertised online and in public spaces.

Table 5.

Distribution of the sample.

	Total	Percentage
Sex
Women	296	67.6
Men	142	32.4
Age
50−54	56	12.8
55−59	70	16
60−64	87	19.9
65−69	75	17.1
70−74	73	16.7
75−79	49	11.2
≥ 80	28	6.4
Level of schooling
Low	99	22.6
Average	111	25.3
High	228	52.05

Normative data

The corrected normative data and their detailed interpretation and clinical applications can be consulted in the book Neuronorma Colombia: protocolo, normas, plataforma de perfiles neuropsicológicos y aplicaciones clínicas.³⁹ Purchase of the book enables access to the website https://www.humanas.unal.edu.co/neuronorma, where users are able to access such resources as the NN.Co Work Unit, representative videos of clinical cases, and a manual on the administration of the tests included in the battery.

Interpretation of patient profiles

The normative data obtained were used to construct the NN.Co Work Unit,⁴⁰ a tool developed in Microsoft Excel that automatically calculates scaled scores (X̄: 10; SD: 3) adjusted for age, sex, and level of schooling, and generates a graphical profile of the individual’s performance. The profile is based on those developed for the Test Barcelona Workstation platform as part of the Spanish Neuronorma project.¹⁷

The profile presents, in the following order, the subject’s performance in tests of working memory, attention, naming and verbal comprehension, visuoconstructive skills, visual and verbal memory, verbal fluency, and executive functions.

The graph is divided into 3 bands corresponding to scaled scores: the red band, at a scaled score of 5 points, indicates the beginning of the range of below-average scores (scores of 2–5); the yellow band (scaled score of 6) indicates low average values, and the green band indicates the average range (scales scaled scores of 7–13; mean, 10). The labels average, low average, and below average are based on those proposed by Guilmette et al.⁴¹ in a study seeking to establish consensus on the use of these qualitative descriptors.

These profiles favour the interpretation of results, enabling comparison of individual performance across domains and identification of differences in performance, preserved and impaired areas, patterns, and the magnitude of differences, on a single scale.

Clinical cases

Below, we present some examples of clinical profiles, using normative data from the NN.Co Work Unit.

Fig. 1 presents the profile of a 71-year-old man with 18 years of schooling, who was referred for neuropsychological assessment due to 4 years’ history of motor rigidity, difficulty initiating movement, progressive loss of strength, and reduced walking speed, as well as a considerable decrease in spontaneous speech and reading. Following onset of the motor symptoms, he was diagnosed with non-specific extrapyramidal disorder and prescribed levodopa, which did not improve symptoms. Symptoms had a significant impact on functional status, and he was fully dependent on a caregiver. Neuroimaging revealed chronic bifrontal subdural haematomas, with a compressive effect, and incipient periventricular leukoencephalopathy. Two surgical procedures were performed to drain the haematomas. The profile shown in the figure shows the results of neuropsychological assessment performed 2 months after the second procedure.

Figure 1.

Cognitive profile of a 71-year-old man with 18 years of schooling. FCSRT: Free and Cued Selective Reminding Test; ROCF: Rey-Osterrieth Complex Figure test; SDMT: Symbol-Digit Modalities Test; TMT: Trail Making Test; TOL-DX: Tower of London-Drexel; WCST: Wisconsin Card Sorting Test.

He presents preserved verbal and visual working memory, alternating attention, naming, phonological verbal fluency, and planning and mental flexibility, but scored below average on tests of sustained attention, visuoconstructive skills, visual and verbal memory, semantic verbal fluency, and verbal inhibitory control. In this patient, the objective of assessment was to characterise his progression after surgery; the results show those cognitive areas which were preserved and may serve as an anchor in a rehabilitation plan. Clinical progression prior to the surgical procedures was variable and unstable, with symptoms and the detection of vascular abnormalities prompting thorough follow-up, seeking to control and observe vascular signs and their relationship with marked cognitive and functional impairment.

Fig. 2 shows the profile of a 65-year-old woman with 16 years of schooling. Approximately 6 months prior to assessment, she had started noticing changes in her ability to recall recent events, reduced verbal fluency, occasional anomia, a poorer sense of orientation in space, and reduced gross motor skills resulting in falls, with no major repercussions. In addition to these symptoms, she presented low mood associated with vital crises and health concerns, particularly related to a longstanding diagnosis of type 2 diabetes mellitus. Her basic and instrumental functional independence were preserved.

Figure 2.

Cognitive profile of a 65-year-old woman with 16 years of schooling. FCSRT: Free and Cued Selective Reminding Test; ROCF: Rey-Osterrieth Complex Figure test; SDMT: Symbol-Digit Modalities Test; TMT: Trail Making Test; TOL-DX: Tower of London-Drexel; WCST: Wisconsin Card Sorting Test.

She scored 30/30 on the MMSE. With regard to the reported alterations, the caregiver reported better memory performance than the patient herself. Emotional changes were reported, including symptoms of anxiety, irritability, and headache, with a significant impact. She also presented symptoms of depression, dissatisfaction with her situation, despair, loss of energy and initiative, and a decrease in her normal activities. She scored 6 on the Geriatric Depression Scale.

Her neuropsychological profile shows preserved verbal working memory and visuospatial manipulation, focused and alternating attention, response speed, naming, semantic verbal fluency, planning and mental flexibility, visuoconstructive skills, verbal and visual memory, and inhibitory control. Scores were below average for measures of spatial working memory and phonological verbal fluency.

The objective of the assessment was to establish whether the changes perceived in the patient’s everyday life could be attributed to pathological ageing; ie, it sought to verify a potential diagnosis. The patient had history of concern about her health status, and symptoms of anxiety and depression; these aspects negatively impact attentional capacity. Her profile largely showed preserved cognitive and functional status, which may suggest that the perceived difficulties are explained by emotional factors, for which a management plan should be established.

Discussion

This article presents the methodology of the NN.Co project, the possibilities associated with use of the battery, and the products developed from its results.

Psychometric and normative studies are indispensable in enabling valid, reliable cognitive assessment processes. One factor justifying the need for this type of study is the variation associated with sociocultural characteristics. Despite significant work on the development of neuropsychological instruments in Latin America,⁴² many tools have been developed in culturally distant contexts (North America and Europe). One contribution of NN.Co is that it proposes a neuropsychological assessment battery adapted and standardised for application in the Colombian setting, focusing on cognitive assessment in elderly individuals.

Performance on these tests is strongly associated with the sociodemographic variables age, sex, and level of schooling.⁴³ In the NN.Co project, regression models were used to quantify this association.

A review of previous Colombian and Latin American normative studies showed that, despite the importance of these initiatives, the existing studies present various methodological limitations, in particular small sample sizes, inclusion of screening instruments only, and reporting only means and standard deviations, rather than standardised scores. The NN.Co project sought to correct these limitations.

It should be noted that, despite the existence of numerous similar initiatives in our region, such as the studies conducted in Colombia by Guardia-Olmos et al.¹¹ and Arango-Lasprilla and Rivera,¹² NN.Co is particularly relevant as it characterises the progression of cognition in normal ageing. Although the cited studies include a broad sample of participants aged 18–95 years, they categorise all individuals older than 77 years into a single group. In contrast, the NN.Co normative sample comprised exclusively individuals aged 50–90 years and was stratified according to age and level of schooling; this enables precise identification of differences in cognitive performance, particularly between the ages of 80 and 90 years. Furthermore, the previous studies only stratify their samples into 2 groups according to level of schooling (more or less than 12 years), and therefore may underestimate subtle differences associated with education.

We should also highlight the inclusion in the normative sample of individuals receiving pharmacological treatment for typical age-associated medical disorders. As a result of this, the study acknowledges common comorbidities of normal ageing, which may influence cognitive performance.

Another relevant consideration in neuropsychology and normative studies is the relationship between cognition and such sociocultural variables as education. Due to differences between education systems, results cannot be compared between countries, despite quantification by recording the number of years of formal schooling. This justifies the adaptations made to the protocol used in NN.Co, compared to the Spanish Neuronorma project. The effect of culture on cognitive performance has been identified as a significant source of bias and may occasionally modulate the effect of schooling and learning and problem-solving strategies. Such is the case for verbal learning, an area in which coding strategies tend to vary between cultures.⁴⁴

In the selection, use, and interpretation of normative data, the normative sample and the individuals being assessed must share the same sociodemographic characteristics. Ideally, normative data should not be used in patients whose ethnicity is not represented in the sample; if this is unavoidable, results should be interpreted with caution. This is necessary due to the relationship between race, ethnicity, and cognitive performance as a result of cultural difference, familiarity with instruments, linguistic differences, and the effect of the mother tongue.⁴⁵ As it is not always possible to ensure this compatibility, the rater must be aware of these limitations when interpreting results, taking into account the fact that the scope of the data is limited by the demographic characteristics of the normative sample.

Normative data were obtained through a process of co-normalisation, which further strengthens the validity of the study. In the clinical context, given the flexibility of the battery, it can be complemented with another fundamental source of validation in neuropsychology, clinical judgement.

Tests must be administered according to the same criteria as those applied in the normative study. In the clinical setting, administration may be subject to various contingencies. In these cases, an additional criterion must be considered in the interpretation, use, and selection of normative data: what is the nature of the domain or capacity being evaluated? Let us consider, for instance, the assessment of verbal memory, in which different tests use different criteria for cued recall after an interference period (20, 30, or 45 minutes, etc.). In this case, variation in the way instruments are administered should consider possible variations in comparison against the normative group if no adjustment is made for forgetting rate and the periods of greater vulnerability to information loss.

Additional studies conducted as part of the NN.Co project demonstrate its usefulness. For instance, a longitudinal study including 68 subjects from the original normative sample represents an initial effort to determine a reliable change index in clinical samples.⁴⁶ The creation and use of the NN.Co Work Unit has made available databases for the characterisation of clinical profiles in such diseases as mild neurocognitive disorder⁴⁷ and depressive symptoms in elderly patients.⁴⁸ Another study addressed the psychometric properties of such instruments as the Boston Naming Test based on the normative data obtained, reporting an internal consistency of 0.90.⁴⁹

Further validation studies are needed for different diseases. This was a focus of the book Neuronorma Colombia: protocolo, normas, plataforma de perfiles neuropsicológicos y aplicaciones clínicas,³⁹ developed by a group of leading Colombian neuropsychologists.

One of the limitations of the original study is the need to include more women in the sample, as well as more ethnicities and population groups (indigenous communities and rural populations from other regions). The size of the longitudinal study sample should also be increased in order to improve the validity of the results. Comparison against other data and normative samples may also be valuable in establishing reliable clinical and diagnostic criteria. Finally, analysis of qualitative information from the tests may enrich our clinical interpretation.

Funding

This study has received no specific funding from any public, commercial, or non-profit organisation.

Conflicts of interest

None.

Acknowledgements

The authors wish to thank students from the Clinical and Cognitive Neuropsychology research group of Universidad Nacional de Colombia, and the raters in the other regions: Dr Ingrid Nayibe Díaz, Dr David Montoya, and Dr Ana Milena Gaviria.