Validation of the American-Spanish Oxford Cognitive Screen (OCS-Sp): Normative data and psychometric properties in acute stroke patients

Abstract

Introduction

Cognitive impairment is a typical sequel and a solid long-term disability predictor that can be screened at early stages post-stroke. However, most routinely used cognitive screening tools were designed to detect dementia, which differs significantly from post-stroke cognitive impairment, including focal cognitive deficits. The Oxford Cognitive Screen (OCS), a cognitive bedside screening tool specifically designed for acute stroke, provides a good alternative for clinical practice.

Aim

This study aims at validating an American-Spanish version of the OCS (OCS-Sp) in healthy participants and acute stroke patients.

Methods

The original version of the OCS was linguistically and culturally adapted into American Spanish. A total of 152 volunteers were recruited, 87 healthy controls and 65 acute stroke patients. Normative data analysis for determining cut-off scores and psychometric validation and reliability analyses in the stroke cohort were completed.

Results

Following a linear regression model demonstrating age, gender, and particularly years of education affecting the performance of the OCS-Sp, the cut-off scores obtained for all subtests were adjusted by these demographic variables. Logistic regression classification analyses revealed that all subtests could discriminate between patients and healthy volunteers. No differences in performance between versions A and B of the test (p > 0.05) were found. The test–retest reliability results in patients showed high agreement between the scores obtained at both time points.

Conclusions

The OCS-Sp obtained similar psychometric scores to the original English version, demonstrating its validity and reliability as an instrument to assess cognitive impairments in American Spanish-speaking acute stroke patients.

Resumen

Introducción

Las alteraciones cognitivas son una secuela común y un predictor de discapacidad a largo plazo en pacientes post accidente cerebrovascular (ACV). Sin embargo, las herramientas de screening cognitivo más utilizadas fueron inicialmente diseñadas para detectar demencia, lo cual difiere significativamente de las alteraciones cognitivas post-ACV. El Oxford Cognitive Screen (OCS), una herramienta de screening cognitivo específicamente diseñada para la etapa aguda del ACV, representa una buena alternativa para la práctica clínica.

Objetivo

El objetivo de este estudio es validar la versión hispanoamericana del OCS (OCS-Sp) en voluntarios sanos y pacientes post-ACV en etapa aguda.

Métodos

La versión original del OCS fue lingüística y culturalmente adaptada al español americano. Se reclutaron 152 voluntarios, incluyendo 87 controles y 65 pacientes post-ACV en etapa aguda.

Resultados

El modelo de regresión lineal mostró que las variables edad, género y particularmente, años de educación afectan el rendimiento en OCS-Sp, los puntajes de corte obtenidos fueron ajustados según estas variables demográficas. El análisis de clasificación de regresión logística reveló que todos los subtest pueden distinguir entre pacientes y controles. No hubo diferencias entre las versiones A y B del test (p >0,05) lo que ayuda a evitar efectos de aprendizaje en sesiones de reevaluación. La fiabilidad test-retest demostró un alto acuerdo entre los puntajes obtenidos en ambos tiempos.

Conclusiones

El OCS-Sp obtuvo puntajes psicométricos similares a la versión original, demostrando su validez y fiabilidad como un instrumento para evaluar las alteraciones cognitivas en pacientes post-ACV en etapa aguda, hablantes del español-americano.

Introduction

Stroke is one of the most common causes of death and disability-adjusted life years (DALY's) worldwide.^1–3 Over 13 million people suffer from a stroke annually, and 1 in 4 people will experience it in their lifetime.^2,3 The incidence of first stroke events in Chile is 140.1 per 100,000 inhabitants.⁴ After a stroke, cognitive impairment is commonly present. It decreases the quality of life in these patients and strongly predicts long-term disability.^5–9 Thus, early detection and assessment are crucial to favor the design of appropriate intervention programs.^7,10,11 A prompt evaluation facilitates the early start of therapy, which may alleviate the degree of dependency of the patients on daily life activities and the risk of depression.^7,12

Currently, most clinicians use the Mini-mental State Examination (MMSE)¹³ and the Montreal Cognitive Assessment (MoCA)¹⁴ to identify cognitive impairments in acute stroke patients. Although these screening tools have an evidence base in assessing healthy aging, current evidence for their use in post-stroke cognitive impairment is far more limited.^15,16 The MMSE and the MOCA are limited in assessing common post-stroke domain-specific impairments, including aphasia, visual loss, visuospatial inattention (neglect), apraxia, reading and writing problems. Additionally, performance on these cognitive screening tools can be confounded by these frequent stroke-specific co-occurring problems.^17–19 MoCA subtests require substantial verbal abilities, and aphasic patients will fail non-language domain tests (e.g., memory) because of language impairments.^13–16 Similarly, patients can fail subtests because they neglect one side of the page (e.g., in the trail-making test).¹⁷ Other extensive neuropsychological batteries that can obtain accurate evaluations of the cognitive domains affected after a stroke are inadequate for administration during the acute phase since they are highly time-consuming, cause fatigue in patients,^20–22 and require highly trained professional neuropsychologists who would not be able to see every patient.

The Oxford Cognitive Screen (OCS) is a bedside, short, easy-to-administer test designed for stroke patients.²³ Unlike MMSE and MoCA, the OCS is designed to offer domain-specific results, allows finger-pointing responses to minimize bias from aphasia, and incorporates assessment of apraxia and neglect.¹⁶ The OCS provides an overview of five cognitive domains with specific subtests organized in blocks (language, memory, attention-executive functions, praxis, and number cognition) that consider language and attention adaptations for common deficits of stroke patients and upper limb motor weakness.²³ Though only a brief screen, the OCS subtests were shown to correlate to known neuropsychological domain-specific lesion areas.²⁴ Besides, the OCS contains two complete parallel sets of stimuli grouped into 11 subtests with the same structure (version A and B) conceived to avoid potential learning effects during retest sessions.²³

In most American Spanish-speaking countries, clinicians cannot access adequately validated tools for assessing cognitive functions in stroke patients.^13,14 Licensed by Oxford University Innovations Health outcomes, which holds the OCS copyright, the OCS has been adapted to and normed for different languages. These include Italian,²⁵ Russian,²⁶ Cantonese,²⁷ Danish,¹⁹ Dutch,²⁸ Chinese Putonghua,²⁹ Brazilian and European Portuguese,^30,31 evidencing the need for validated cognitive screening tools for stroke patients worldwide. The present study aims to validate the American-Spanish version of the OCS in healthy participants and acute stroke patients.

Methods

The study was approved by the Medical Ethical Committee of the Pontificia Universidad Católica de Chile (Reference number: 18121007). All participants were volunteers and signed an informed consent form before inclusion.

Participants

Eighty-seven healthy participants (control group) and 65 stroke patients (patient group) native Spanish-speaking adults (>18 years old) participated in the study. The inclusion criteria for the control group included the absence of cognitive impairment indicated by an MMSE score higher than or equal to 24/30, no presence of neurological and/or psychiatric disorders, and the absence of sensory impairment preventing the assessment from being completed. Patients were recruited at the Hospital Clínico UC-Christus and the Complejo Asistencial Dr. Sótero del Río. The inclusion criteria for the patient group included participants at the acute stage post-stroke (within the first three weeks post-stroke) with the capacity to keep their attention for at least 15 min and to give consent for themselves. Based on the neurological report, patients diagnosed with transient ischemic attack and cognitive impairment before the stroke were excluded.

Out of 65 stroke patients, data on the educational level of 9 participants was missing; one patient was under 60 years old. The demographic data of all participants recruited in the study is summarized in Table 1. Across groups, no differences were found in age and handedness. Given that the incidence of stroke is higher in adults older than 60 years, we reported demographic variables of interest (age and education) in our sample according to this age range.³² No differences in years of formal schooling were found between the control and the patient group for participants older than 60.

Table 1.

Demographic information.

	Control group(n = 87)	Patient group(n = 65)	p-Value
Age, n, mean (SD)			0.22
<60 years	32, 47.38 (8.67)	17, 46.12 (9.20)
≥60 years	55, 70.65 (7.55)	48, 72.75 (7.63)
Gender, n (%)			0.01*
Female	60 (68.97%)	31 (47.69%)
Male	27 (31.03%)	34 (52.31%)
Handedness, n (%)			0.37
Right-handed	77 (88.50%)	58 (89.23%)
Left-handed	7 (8.05%)	4 (6.15%)
Ambidextrous	3 (3.45%)	0 (0%)
Unknown	0 (0%)	3 (4.61%)
Type of stroke, n (%)
Ischemic	–	51 (78.4%)
Hemorrhage	–	14 (21.5%)
Hemisphere, n (%)
Left	–	25 (38.4%)
Right	–	31 (47.6%)
Bilateral	–	3 (4.6%)
Unknown	–	6 (9.2%)

	Control group ≥ 60 years(n = 55)	Patient group ≥ 60 years(n = 48)	p-Value
Education, n (%)			0.20
≤12 years	32 (58.1%)	29 (60.4%)
>12 years	23 (41.9%)	11 (22.9%)
Unknown	0 (0%)	8 (16.6%)

SD = standard deviation. p-Values were obtained according to Fisher's exact test for categorical variables.

*

p < 0.05.

We presented age and education variables in participants above 60 years, given the incidence of stroke in people older than sixty.

Regarding the clinical profile of the patient group, ischemic strokes represented 78.4% of the sample. Half of the patient group (47.6%) had a right hemispheric stroke, and 4.6% had a bilateral stroke. The hemispheric location of 9.2% of the lesions was unknown. The average days between the stroke and the first evaluation with OCS-Sp were 8.8 days (SD ± 7.1).

Administration of the OCS-Sp versions A and B

Across groups, 31 healthy participants and 27 patients were evaluated with both versions of OCS-Sp. They were all randomly and equally assigned to each version of the test, with a maximum of 8 days between both evaluations. The version of the first evaluation (A or B) was randomly assigned, and the second evaluation was accomplished with the opposite version of the first. Consequently, each group of participants with two assessments (control and patients) had a subgroup first evaluated with version A and later with version B (A-B) and another subgroup with the opposite order (B-A).

Besides, 56 healthy participants were assessed with only one version of the test (35 with version A and 21 with version B). Thirty-eight patients were assessed only once (19 with version A and 19 with version B) (Table 2).

Table 2.

Participants evaluated with versions A and B of OCS-Sp.

	Version A & B	Only version A	Only version B	Total
Control group	31	35	21	87
Patient group	27	19	19	65

Adaptation of the American Spanish Oxford Cognitive Screen (OCS-Sp)

First, two independent professional translators translated the original OCS (versions A and B) into Spanish. The final consensus version was further reviewed by a speech and language therapist and a clinical linguist (IVC Laswche = 0.87). The test presented some linguistic challenges for its application in an American-Spanish environment. Hence, some subtests were linguistically and culturally adapted: picture naming, semantics, sentence reading, orientation, verbal memory, and episodic memory (see Appendix 1). We used neutral Spanish items, non-specific to the Chilean context, to ensure they are generalizable to other American Spanish-speaking countries.

Versions A and B of the English OCS contained 14 different drawings used as stimuli in three subtests (picture naming, semantics, and episodic memory). These stimuli are used in both versions, either as target items or as distractors. The OCS-Sp maintained 13 stimuli from the English version. The image for ‘filing cabinet’ was discarded due to the low frequency and variability of the word across Spanish-speaking countries. We added two new stimuli to build the two versions of OCS-Sp (scissors and hen) to keep the syllabic length and semantic categories criteria. In the OCS-Sp, all stimuli were drawn again following the original style of the English OCS.

Due to linguistic differences between English and Spanish, the sentence reading subtest had to be adjusted and was done following the same rationale as the Italian version.²⁵ Given that Spanish has a significantly lower number of irregular words, new sentences fitting the requirements of the original version were created. The new sentences contained words with a low-frequency syllabic structure and high neighborhood density words to detect superficial dyslexia and neglect dyslexia, as in the following example from version B of the OCS-Sp: ‘El viejo transportista sirvió de inspiración ante la claustrofobia de sus compañeros en el transatlántico’ (‘The old carrier served as an inspiration of his companions on the ocean liner’ in its English translation). Although the original questions of the orientation subtest remain the same (city, moment of the day, date), some answers were changed. Afternoon substitutes for Midday as a specific moment of the day. Due to the changes in picture naming, semantics, and sentence reading subtests, the verbal and episodic memory section also had to be adjusted.

OCS-Sp

Similarly to the original version, the OCS-Sp includes short-high frequency words, vertical layouts, and multimodal presentations to facilitate the comprehension of patients with potential language deficits. It also includes multiple-choice options in some subtests, with no penalty for their use. The score in this type of subtest is one point for each correct answer, regardless of the response modality (free response or using multiple-choice options). The administration takes about 15–20 min, and the results are included in a simple summary figure based on the patient's performance and the cut-off scores. This figure is clinician-friendly, facilitating communication among the rehabilitation team and allowing them to identify the primary cognitive impairments, as well as the unaffected cognitive domains, without accessing the entire medical file of the patients.²³ The description and the scoring of each subtest of the OCS-Sp are provided in Table 3.

Table 3.

Description of OCS-Sp subtests.

OCS-Sp subtests	Description	Scoring
Picture naming	A total of 4 images are presented separately, and the participant is asked to name them.	1 point/correct response (max = 4)
Semantics	A total of 3 images are presented simultaneously, and participants must recognize 3 categories.	1 point/correct response (max = 3)
Orientation	Participants are asked in what city they are, the time of the day, month, and year. If participants cannot respond, multiple-choice options (MCO) are provided.	1 point/correct response (max = 4). No penalty for using MCO.
Visual field	A confrontation test evaluates the 4 quadrants of the visual field. The participant is asked to point at the hand that is moving.	1 point/quadrant (max = 4)
Sentence reading	Participants are asked to read a sentence.	1 point/word read correctly (max = 15)
Number writing	Participants are asked to write 3 different numbers.	1 point/correct response (max = 3)
Calculation	Participants are asked to resolve 4 simple mental arithmetic exercises (sum and subtraction). If participants are unable to response, MCO are provided.	1 point/correct response (max = 4). No penalty for using MCO.
Total hearts	These 3 items are evaluated in a heart cancelation subtest, in which a horizontal page with complete and incomplete drawings of hearts (150 items) is presented to the patient. The incomplete hearts have a gap on the left (50 items) or right (50 items) side. Participants are asked to cross out all the full hearts within 3 minutes. Neglect is evaluated on this subtest by object and space asymmetries.	Total hearts: 1 point/full heart cross out (max = 50). Object asymmetry: Subtract the number of hearts with the right gap erroneously crossed out from the hearts with the left gap erroneously crossed out. Space asymmetry: Subtract the number of full hearts on the right side of the page from the full hearts on the left side of the page.
Object asymmetry
Space asymmetry
Imitation	Participants are asked to imitate hand movements (2 sequences) and finger movements (2 gestures).	3 points/correct and precise gesture (max = 12)
Verbal memory	First, the participant is asked to recall the sentence from the ‘sentence reading’ subtest. If the participant does not recall, MCO is presented to facilitate the recognition of the 4 target words.	Recall: 1 point/correct response. (max = 4)Recognition: 1 point/correct response. (max = 4)
Episodic memory	Participants are asked about subtests completed earlier on (4 multiple-choice questions).	1 point/correct response (max = 4)
Executive score	This subtest involves 2 simple and 1 complex trail. In the simple subtests: (1) participants are asked to connect circles among triangle distractors and (2) to connect triangles among circle distractors. The complex subtest consists of connecting alternate triangles and circles. In the 3 subtests, the participant connects the largest to the smallest item.	Executive score: Subtract the correct connections on the complex subtest (max = 13) from the sum of the triangles (max = 6) and the circles (max = 6).

Statistical analysis

Given that a preliminary Kolmogorov–Smirnov test revealed that data had a non-normal distribution, non-parametric tests were used in the initial analysis. Subsequently, linear regression analyses were used to assess the relationship between the scores and the independent variables of gender, age, and years of education. We expressed age and years of education as continuous variables and gender as categorical. We obtain scores for each subtest of the OCS-Sp corrected with linear regression analysis for the potential bias of omitting these sociodemographic variables. These corrected scores were used in logistic regression models to accurately classify healthy controls and patients. From this classification model, the estimation of the area under the Receiver Operating Characteristic Curve (ROC curve) was calculated. The optimal cut-off value was based on a balance of sensitivity and specificity grounded on the Youden index. Regardless of the administration order in the two groups, differences between both test versions (A and B) were determined through a Wilcoxon Signed-Rank test. A Wilcoxon Signed-Rank test was also used to evaluate the potential influence of the order of administration in which both versions were presented to the volunteers (A-B or B-A). Test–retest reliability was calculated using the intraclass correlation coefficient (ICC) for the patient group. The analysis was made using R's psychometrics and statistical packages (version 4.0.2). Statistical significance was established at p-value < 0.05.

Results

Effects of gender, age, and education on the OCS-Sp subtests

A total of 143 participants (87 healthy volunteers and 56 patients) were included in the analysis (due to missing data on the level of education of 9 patients). The variable gender had an effect on the total heart subtest (p < 0.05). In contrast, the variable age showed an effect on imitation, verbal memory (recognition), and executive subtests (p < 0.05 in each one). Years of education had an effect on eight subtests of the OCS-Sp: semantics (p < 0.05), number writing (p < 0.01), calculation (p < 0.01), verbal memory recall (p < 0.01), and episodic memory subtest (p < 0.05). The effect of the variables on each of the OCS-Sp subtests is reported in Table 4.

Table 4.

Linear regression models for each subtest of OCS-Sp.

Control and patient group (n = 143)
OCS-Sp subtests	Intercept (S.E.)	Male^a (S.E.)	Age (S.E.)	Years of education (S.E.)	R^2
Picture naming	3.98** (0.30)	−0.08 (0.09)	−0.00 (0.00)	0.01 (0.01)	0.05
Semantics	2.94** (0.18)	−0.07 (0.05)	−0.00 (0.00)	0.01* (0.01)	0.07
Orientation	3.96** (0.26)	−0.02 (0.08)	−0.00 (0.00)	0.00 (0.01)	0.01
Visual field	4.10** (0.23)	−0.08 (0.07)	−0.00 (0.00)	0.00 (0.01)	0.02
Sentence reading	15.38** (1.28)	−0.32 (0.39)	−0.02 (0.01)	0.04 (0.05)	0.04
Number writing	2.02** (0.42)	0.00 (0.13)	0.00 (0.00)	0.04** (0.02)	0.05
Calculation	2.06** (0.56)	−0.02 (0.17)	0.00 (0.01)	0.08** (0.02)	0.12
Total hearts	43.71** (5.90)	−4.63* (1.80)	−0.09 (0.07)	0.45* (0.21)	0.11
Object asymmetry	0.70 (1.04)	0.15 (0.32)	0.01 (0.01)	−0.07 (0.04)	0.06
Space asymmetry	2.05 (1.50)	0.82 (0.46)	0.01 (0.02)	−0.07 (0.05)	0.04
Imitation	12.30** (1.19)	−0.59 (0.36)	−0.03* (0.01)	0.05 (0.04)	0.09
Verbal memory (recall)	0.26 (0.64)	−0.30 (0.19)	−0.00 (0.01)	0.10** (0.02)	0.14
Verbal memory (recognition)	3.57** (0.52)	−0.02 (0.16)	−0.01* (0.01)	0.02 (0.02)	0.06
Episodic memory	3.40** (0.45)	0.16 (0.14)	−0.01 (0.01)	0.03* (0.02)	0.07
Executive score	−1.00 (1.53)	0.20 (0.47)	0.04* (0.02)	−0.05 (0.06)	0.05

S.E: standard error.

a

The reference is female. R²: R-squared.

*

p < 0.05.

**

p < 0.01.

Given the effect of demographic variables, we adjusted the participants’ scores in our logistic regression classification model to control their effect on the performance of each subtest of the OCS-Sp (see Appendix 2). The results showed that all OCS-Sp subtests classify healthy controls and stroke patients. Odds ratios under 1 were obtained for most subtests, indicating that the participant is less likely to belong to the patient group as the accuracy score increases. In the asymmetries and executive score subtests, odds ratios were above 1, given higher values denoting worse performance here.

Diagnostic accuracy and optimal cut-offs of the OCS-Sp

The ROC curves and the optimal cut-off for each subtest were calculated with data from 143 participants. The area under the curve (AUC) measures the model's ability to discriminate between participants who experience the outcome of interest and those who do not. Values close to 0.6 are considered acceptable to discriminate the presence of the outcome of interest, while values around 0.7 are considered excellent.³³ We found AUC above 0.6 for most subtests, except for episodic memory (Fig. 1).

Figure 1.

ROC curve for each subtest of OCS-Sp.

The sensitivity and specificity of the cut-off scores of the OCS-Sp are presented in Table 5. These cut-off scores optimize the best classification of patients and healthy participants (higher correct classifications for both groups). Below the cut-off indicates impairment for all subtests except for object and space asymmetry and the executive score. In the asymmetry subtests, the cut-offs are based on the absolute value of the obtained score. Then, the neglected side (right/left) is given by the sign of the obtained score (positive scores indicate left neglect, and negative scores denote right neglect).

Table 5.

Optimal cut-offs for each subtest of the OCS-Sp.

Control and patient group (n = 143)
OCS-Sp subtests	Sensitivity	Specificity	AUC	Cut-off
Picture naming	58.93	58.62	0.65	<4
Semantics	58.93	58.62	0.66	<3
Orientation	57.14	57.47	0.61	<4
Visual field	64.29	64.37	0.65	<4
Sentence reading	55.36	55.17	0.63	<15
Number writing	64.29	64.37	0.67	<3
Calculation	62.50	62.07	0.66	<3
Total hearts	64.29	64.37	0.66	<41
Object asymmetry	55.36	55.17	0.64	>0^a
Space asymmetry	66.07	65.51	0.66	>2^a
Imitation	55.36	55.17	0.63	<10
Verbal memory (recall)	58.93	59.77	0.67	<2
Verbal memory (recognition)	57.14	56.32	0.60	<3
Episodic memory	53.57	54.02	0.58	<4
Executive score	57.14	56.32	0.64	>0

AUC: area under the curve.

a

Cut off based on the absolute value of the obtained score.

Analysis of versions A and B of the OCS-Sp

In the control group with two assessments (n = 31), regardless of the administration order, the evaluations with version A were compared with those with version B through the Wilcoxon test. No significant differences between the scores of both versions were found for the control group in any subtest (p > 0.05). The same was held for patients with two assessments (n = 27), except for the significant differences in the episodic memory subtest (p = 0.006). Additionally, it was evaluated whether the order of administration (A-B or B-A) influenced the scores obtained. For the control and patient groups with two assessments was compared the ‘A-B subgroup’ (n = 13) with the ‘B-A subgroup’ (n = 14). The administration order did not result in significant differences (p < 0.05) for the control and the patient group. Consequently, versions A and B of the OCS-Sp behaved similarly (see Appendix 3).

Performance of the patient group

A few participants in the patient group could not do some of the subtests (obtained zero points). Across tasks, 2/65 had problems with the picture naming subtest, 1/65 with semantics, 1/65 with orientation, 2/65 with sentence reading, 9/65 with number writing, 7/65 with calculation, 1/65 with total hearts, 1/65 with imitation, 38/65 with verbal memory (recall), 2/65 with verbal memory (recognition), and 1/65 with episodic memory. We also reported the incidence of impairments in our acute stroke population divided into quartiles and provided the percentages of impaired patients based on the cut-off scores (see Appendix 4). Number writing and verbal memory (recall) were challenging subtests for acute stroke patients, as most participants were categorized as ‘severe’ (lowest quartile). By contrast, picture naming and orientation subtests were better preserved (61.5% and 75% of the patients were in the highest quartile, respectively).

Test–retest reliability

Test–retest reliability was determined based on the acute stroke patients who completed both versions of the OCS-Sp (n = 27). Versions A and B were randomly assessed between 3 to 8 days post-stroke, with an average of 3.4 days (SD = 0.5) between the test and the retest session. The obtained values show moderate to good reliability between the two evaluation sessions, with ICC values ≥0.5 for most subtests (Table 6).

Table 6.

Test–retest reliability in the patient group.

Patient group (n = 27)
OCS-Sp subtests	Mean T1^a (SD)	Mean T2^b (SD)	ICC	CI 95%
Picture naming	3.63 (1.01)	3.59 (0.747)	0.617	0.146–0.827
Semantics	2.78 (0.64)	2.67 (0.734)	0.726	0.402–0.875
Orientation	3.85 (0.36)	3.78 (0.577)	0.037	−1.177–0.567
Visual field	3.85 (0.53)	3.78 (0.69)	0.680	0.295–0.854
Sentence reading	12.81 (3.98)	12.89 (4.18)	0.922	0.828–0.964
Number writing	2.26 (1.20)	2.11 (1.15)	0.788	0.536–0.903
Calculation	2.56 (1.48)	2.67 (1.04)	0.766	0.484–0.894
Total hearts	32.67 (14.83)	33.81 (14.32)	0.960	0.914–0.982
Object asymmetry	1.93 (3.15)	1.74 (2.58)	0.927	0.840–0.966
Space asymmetry	3.48 (3.59)	2.89 (3.25)	0.669	0.280–0.850
Imitation	9.89 (2.56)	9.19 (3.31)	0.630	0.203–0.830
Verbal memory (recall)	0.52 (0.64)	0.59 (0.89)	0.549	0.135–0.765
Verbal memory (recognition)	2.52 (1.19)	2.33 (1.04)	0.815	0.600–0.915
Episodic memory	3.07 (1.17)	3.07 (1.14)	0.450	−0.238–0.752
Sequences: mixed	7.15 (4.97)	7.74 (4.04)	0.846	0.662–0.930
Executive score	1.96 (3.14)	2.22 (3.59)	0.504	0.049–0.742

SD: standard deviation; ICC: intraclass coefficient correlation; CI 95%: confidence interval 95%.

a

Values from the first assessment (independent of the version of the OCS-Sp).

b

Values from the second assessment (independent of the version of the OCS-Sp).

Discussion

This study presented the American-Spanish adaptation of the Oxford Cognitive Screen (OCS-Sp), assessed the role of demographic variables (age, gender, and years of education) in the performance of patients and controls on the individual OCS-Sp subtests, and provided specific adjusted cut-off scores.

Our analysis indicated that demographic variables had a differential effect on the performance of some of the OCS-Sp subtests. Age affected imitation, recognition memory, and executive subtests. Gender had a selective influence only detectable on the total heart subtest. Education, however, had a widespread effect on several subtests (semantics, number writing, calculation, verbal memory (recall), and episodic memory), thus highlighting the importance of controlling for educational background, especially in countries where there are still considerable differences among the general population.

The cut-off scores obtained from the Spanish-speaking sample were calculated by adjusting for the demographic variables affecting subtest performance. Most subtests showed an AUC above 0.6, except for episodic memory. This domain is assessed before finalizing the application of the OCS, requiring the participants to recognize a set of stimuli presented during the evaluation by giving four multiple-choice questions (each one comprising four options). This low AUC could be explained by the design of the task to evaluate episodic memory, which may not include task complexity. Therefore, we could not obtain enough variability in the performance of the sample studied. As highlighted by Frederick & Speed,³⁴ forced-choice testing is susceptible to malingering detection, so researchers should implement differing difficulties in the trials.^35,36 However, our scores align with those obtained from the English original²³ and the other existing authorized validated versions.^19,25,26,28 To facilitate the screening test application in clinical practice, we provided already adjusted cut-off scores so that clinicians do not need to perform extra calculations to adjust for demographic variables or check different score tables.

To control for learning effects on test–retest assessments, we collected data from the same cohort of participants. Our results replicated those obtained in the original OCS,²³ with stable scores for both sessions. The difference between test and retest sessions (independently of the version test) was explored by calculating the ICCs. Our patient group showed moderate to high reliability in most of the subtests of the OCS-Sp. Lower ICC values were obtained for orientation, episodic memory, and executive score subtests. We attribute these results to potential spontaneous recovery³⁷ and likely practice effects, especially noted in orientation and executive subtests, given a delay of only a few days between the test and retest sessions. Despite minor asymmetries, the results mentioned above confirmed the high agreement of the test–retest and the high temporal stability of the OCS-Sp. The availability of parallel versions and test–retest reliability is of utmost importance for neurorehabilitation, allowing measurement of the effectiveness of therapy during rehabilitation and ruling out the interference of learning effects.³⁸

When specifically evaluating the performance in the episodic memory subtest, we observed that out of the 27 patients (randomly evaluated with versions A and B), 20 scored the same or showed a difference of 1 point in this subtest in both versions test. A total of 5 patients showed a difference of 2 points, and the majority scored higher in the second evaluation independently of the version presented. However, two patients scored 0 points in one version and 3 points in the other version; one of them got a higher score in the second evaluation, while the other performed worse. These results on the performance impact the analysis of the comparison of versions and the test–retest reliability. We attributed these differences to subtest-specific factors. In addition, some of them could perform better in incidental memory due to an anticipatory attitude to the subtest in the second evaluation a few days later than the first evaluation.³⁹ Additionally, we observed that those who scored worst in the second evaluation (3 patients) also had a restrictive prognosis. The neural correlates of episodic memory recognition impairment (evaluated with the OCS) have been associated with cortical damage in the left insula, left central operculum, and planum polare cortices.²⁴ Therefore, future validation studies of the OCS in other languages should keep considering the location and severity of the stroke to interpret the overall results of the OCS.

The current study had some limitations. The first limitation was the reduced sample size due to the global coronavirus disease pandemic. Extending the number of observations may help further refine demographic variables’ impact over the subtests of the OCS-Sp. To compensate for potential effects, we provided adjusted cut-offs. The AUC of our cut-off scores is considered acceptable; however, further studies evaluate a cut-off score merged by cognitive domain. Second, although clinical linguists and speech and language therapists from Spain, Colombia, Argentina, and Perú contributed to the linguistic adaptation of the test, all the participants included in the study were tested in Chile. Future studies should evaluate the OCS-Sp in other Spanish-speaking countries to create a more representative sample and re-evaluate our cut-offs. Another limitation is the verbal memory measure, in which most patients had difficulties performing through free recall, while only two patients had difficulties by recognition. The original version described that verbal memory recall was too challenging for their participants and recommended giving an individual prompt to remember the sentences.²³ Our study applies the prompt; however, it appears it was also difficult for our patients. Due to this limitation, the verbal recall OCS-Sp subtest cannot be thoroughly compared with other screening tests which assess verbal memory.¹⁷ Those other screenings, such as MoCA and MMSE, assess verbal memory by freely recalling single high-frequency words.^13,14 The OCS-Sp is assessed by recalling low-frequency syllabic structure and high neighborhood density words that are part of a sentence. Finally, regarding inclusion criteria, we only assessed our healthy participants with the MMSE to detect cognitive impairments. We did not evaluate the patients since the MMSE had limited capacity to evaluate common cognitive impairments in the stroke population,^13,15,16 adding to the short time available to evaluate in the acute stage. However, a recent study compared the OCS with the MMSE in clinical populations.¹⁸ The results showed that while 35% had an impaired performance according to the MMSE, 92% were impaired in at least one OCS domain. Moreover, all the patients who were impaired with the MMSE showed impaired performance in the OCS. This study revealed a higher detection rate of the incidence of cognitive impairment with the OCS and the absence of false negatives compared with the MMSE.¹⁸

Conclusion

The culturally and linguistically adapted American Spanish OCS (OCS-Sp) stands as a valid and reliable instrument to briefly assess domain-specific cognitive impairments in stroke patients during the acute phase as a first-line cognitive screening tool. However, following the initial screen, at a later stage post-stroke, a further neuropsychological assessment may be required to determine the nature of any persisting deficits or to detect more subtle impairments.

Conflict of interest

None.

Acknowledgments

The authors acknowledge speech-language therapists Elizabeth Mulato, Vania Iturra, and Pamela Donoso. Silvia Martínez-Ferreiro also acknowledges the support of the AADI project (FEDER – Europe & Région Occitanie, FSE 2014-2020 N°2019-A03105-52) and the Ramón y Cajal contract (RYC2020-028927-I). Nele Demeyere (Advanced Fellowship NIHR302224) is funded by the National Institute for Health Research (NIHR). The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR, NHS, or the UK Department of Health and Social Care. Finally, the authors thank the Collaboration of Aphasia Trialists funded by COST and The Tavistock Trust for Aphasia for providing methodological expertise relating to aphasia test design.