Assessment of individual clinical outcomes: Regarding an electroconvulsive therapy

Abstract

Evaluation of therapeutic results and of the efficacy and effectiveness of treatments is an area of interest both for clinicians and researchers. In general, randomized controlled trial designs have been used as the methodology of choice in which intergroup comparisons are made having a minimum of participants in each arm of treatment. However, these procedures are seldom used in daily clinical practice. Despite this fact, the evaluation of treatment results for a specific patient is important for the clinician in order to address if therapeutic goals have been accomplished both in terms of statistical significance and clinical meaningfulness. The methodology based on Reliable Change Index (RCI – Jacobson and Truax)¹ provides an estimate these two criteria. The goal of this article is to propose a procedure to apply the methodology with a single case study of a woman diagnosed with major depression and treated with Electroconvulsive Therapy (ECT).

Resumen

La valoración de resultados terapéuticos y la evaluación de la eficacia y efectividad de los tratamientos es un área interés para clínicos e investigadores. Usualmente se ha utilizado la metodología basada en diseños controlados aleatorios y la comparación intergrupos con un mínimo de participantes incluidos en cada brazo de tratamiento. No obstante, estos procedimientos raramente pueden ser aplicados en la práctica clínica habitual, pero sí es importante para el clínico evaluar si los resultados del tratamiento propuesto para una persona específica cumplen con los objetivos terapéuticos, tanto en su significación estadística como por su relevancia clínica. La metodología basada en el índice de cambio fiable (Jacobson y Truax)¹ permite estimar esta doble consideración y el objetivo del artículo es proponer el procedimiento de aplicación a través del estudio de un caso único diagnosticado de depresión mayor tratado con Terapia Electro Convulsiva (TEC).

Introduction

Evaluation is at the heart of the clinical process. It is a key element for diagnosis, treatment planning and evaluation of treatment effects. Obtaining the expected results implies both success of the intervention, and therapeutic discharge, while failure derives in a decision making process regarding change or removal of prescribed treatment or the assumption of new therapeutic recommendations. In any case, information regarding clinical outcomes of an intervention is of utmost importance for health providers.²

Specifically, evaluation of therapeutic efficacy based on scientific evidence relies on a methodology that uses comparisons of groups that are exposed to different interventions; one of which is the object of evaluation, the other with a proven therapeutic value. The basic design is known as random clinical trial, a procedure that defines a step-by-step protocol in order to avoid the effect of bias and preserves casualty of intervention on the therapeutic effect. Group variability control and statistic use allows for decision making regarding the cause (chance or intervention) of the observed changes.³

However, in day-to-day clinical practice, health providers rarely count on big groups to make comparisons and thus, more often than not, their decisions are based on specific cases. The habitual procedure is to consider change shown by the individual, analyzing variation of the state or trait of interest, normally assessed through biological, symptomatic or orectic indexes. For instance, in mental health, therapy has as a goal to change states (i.e. depressive signs and symptoms) or traits, a personality disposition (i.e. neuroticism). In order to do so, states or traits are assessed with standardized instruments that allow for the scoring of the case in relation to a normative group. Therefore, if a person is evaluated with the same instrument before and after treatment, the difference in scores can offer a quantitative index of change than can be incremental, decremental or equal. Besides, scores can be analyzed with reference to a normative sample, allowing for the ranking of the scores of the person in a functional or dysfunctional range. Scales usually provide information about cutoff points to facilitate clinical decisions.

However, compared with the methodology used for scientific decision making regarding efficacy of treatment modalities, decision making in individual cases lacks tools that could inform if the observed change is both clinically meaningful and statistically significant. We need to know if the change has a sufficient magnitude so as to be considered an important clinical outcome and rule out chance.

The goal of this paper, based on a clinical case treated with Electro Convulsive Therapy (ECT), is to offer an answer using the Reliable Change Index,^1,4 a formula that can help in clinical and statistically significant decision making in individual cases.

Clinical case

The patient is a 45-year-old married woman, mother of two children and teacher on a sick leave of work. She was admitted in the Psychiatric Unit of a general hospital, being the fifth hospitalization. She was referred by a psychiatric due to severe depressive symptoms for the past several weeks in the context of a Bipolar Disorder.

The patient showed depressive symptoms, especially inability to enjoy daily activities, more exacerbated in the morning, severe asthenia, psychomotor slow-down, tensional headaches, attention and concentration impairments, and nondelirious thoughts of inadequacy, abandonment, disability and guilt.

Her daily functioning is characterized by being self-demanding, presenting herself as a very respectful, polite and well-mannered person, even rigid in response to a firm sense of duty. Her values showed a strong commitment to her responsibilities around her role as a good mother, daughter, wife and worker. She shows obsessive traits (order and control) and a notable denial of hostility. She refers a very organized daily routine that holds her together.

Her medical record indicates hypothyroidism treated with hormones. With reference to psychiatric antecedents, she relates several affective problems since youth, initially of a depressive sing that would remit with Benzodiazepine, even spontaneously. Three years ago she presented a manic episode with psychotic symptoms, congruent and incongruent (paranoid) with her mood. She refers no psychiatric family history.

The patient states that her first depressive episode goes back 6 years when her father died. It was treated with antidepressants, a medication she continues to use today. Initially she responds to IRSA antidepressants showing a quick and effective response with complete recovery. Since then, at least 5 depressive episodes have occurred with a poorer response to treatment, slower recovery without reaching a true euthymic state and maintaining subclinical depressive symptoms that interfere with functioning and quality of life. Medication was changed several times prescribing dual drugs (Venlafaxina) and later a combination of antidepressants with maximal dose (Venlafaxina and Mirtazapina) and incorporating other mood-stabilizing drugs (lithium salt or valproic acid). The intensity and functional disability caused by these symptoms required inpatient treatment. She also attends cognitive psychotherapy on a weekly basis showing good adherence in the past five years.

After several pharmacological interventions without therapeutic results (the latest Clorimipramine 225 mg/d, suspended due to no response and severe secondary effects) Electro Convulsive Therapy is recommended. Twelve bilateral sessions are programmed, three per week on alternate days. A very favorable response was observed. During interviews after the third session she is euthymic, with a significant improvement in all symptoms. Finally, given the positive response, ten sessions were administered and maintenance sessions where recommended at discharge following Martínez-Amoros, Cardoner, Galvez and Urretavizcaya's suggestions.⁵

When the patient was admitted for ECT treatment, she participated in a clinical interview to assess frequency and intensity of depressive symptoms and personality traits. Three standardized instruments were used in the Spanish adapted version: Hamilton Depression Scale,⁶ Beck Depression Inventory,⁷ and NEO-PI-R⁸ a widely used personality test that measure six traits: Neuroticism, Extraversion, Openness to experience, Affability, Conscientiousness. The same protocol and instruments were used at the end of the second week. Table 1 shows her scores in the three scales at pretest and posttest.

Table 1.

Clinical and statistical significant change in NEO-PI-R factors, DBI and HAM-D.

	Pretest	Posttest	Clinical cutoff point	Clinical change	Change difference	RCI	Statistically significant change^*	Interpretation
NEO PI-R
Neuroticism	63	58	35/65	No	−5	−0.913	No	No change (NC)
Extraversion	37	37	35/65	No	0	0.000	No	NC
Openness	37	37	35/65	No	0	0.000	No	NC
Agreeableness	58	57	35/65	No	−1	−0.183	No	NC
Conscientiousness	53	47	35/65	No	−6	−1.096	No	NC
Anxiety	63	58	35/65	No	−5	−0.913	No	NC
Hostility	43	56	35/65	No	+13	2.373	Yes	Significant change
Depression	68	51	35/65	Yes	−17	−3.104	Yes	Meaningful clinical change
Self-consciousness	68	60	35/65	Yes	−8	−1.461	No	Nonsignificant change
Impulsiveness	49	53	35/65	No	+4	0.730	No	NC
Vulnerability to stress	73	66	35/65	No	−7	−1.278	No	NC
Warmth	37	43	35/65	No	+6	1.096	No	NC
Gregariousness	56	42	35/65	No	−14	−2.556	Yes	Significant change
Assertiveness	29	37	35/65	Yes	+8	1.461	No	Nonsignificant change
Activity	42	47	35/65	No	+7	1.278	No	NC
Excitement seeking	43	50	35/65	No	+7	1.278	No	NC
Positive emotions	37	37	35/65	No	0	0.000	No	NC
Fantasy	34	37	35/65	Yes	+3	0.548	No	Nonsignificant change
Esthetics	47	47	35/65	No	0	0.000	No	NC
Feelings	43	46	35/65	No	+3	0.548	No	NC
Actions	37	40	35/65	No	+3	0.548	No	NC
Ideas	42	40	35/65	No	−2	−0.365	No	NC
Values	43	40	35/65	No	−3	−0.548	No	NC
Trust	57	57	35/65	No	0	0.000	No	NC
Straightforwardness	73	60	35/65	Yes	−13	−2.373	Yes	Meaningful clinical change
Altruism	58	47	35/65	No	−11	−2.008	Yes	Significant change
Compliance	63	57	35/65	No	−6	−1.096	No	NC
Modesty	50	57	35/65	No	+7	1.278	No	NC
Tender mindedness	31	44	35/65	Yes	+13	2.373	Yes	Meaningful clinical change
Competence	57	44	35/65	No	−13	−2.373	Yes	Significant change
Order	53	49	35/65	No	−4	−0.730	No	NC
Dutifulness	57	43	35/65	No	−14	−2.556	Yes	Significant Change
Achievement striving	53	50	35/65	No	−3	−0.548	No	NC
Self-discipline	34	47	35/65	Yes	+13	2.373	Yes	Meaningful clinical change
Deliberation	60	53	35/65	No	−7	−1.278	No	NC
Social desirability	63	46	35/65	No	−17	−3.104	Yes	Significant change
Symptomatic scales for depression			L, M, G^#
BDI	30	8	10, 19, 30	Yes	−22	−6.987	Yes	Meaningful clinical change
HAM-D	28	12	8, 14, 19	Yes	−16	−12.531	Yes	Meaningful clinical change

#

Cutoff points for DBI and HAM-D: M, mild depression; MD, moderate; G, severe.

*

RCI values >1.96 or ≥1.96 are statistically significant, p < .05. The higher the RCI value the lower the probability observed change is due to chance.

Informed consent was obtained from the patient both for the assessment and the treatment wit the ECT.

Evaluation methodology: reliable change index

Reliable change indexes are a set of statistical procedures used in many fields of psychological assessment to find if the scores of a given individual in a psychometric test have changed compared to a previous assessment with the same instrument.⁴ Reliable change techniques were initially described by Jacobson and Truax¹ and were designed to help in the decision making of significant therapeutic change in patients who had experience a disorder or a clinical intervention.

Jacobson and Truax¹ presented a definition of a clinically significant change in therapy. In order to obtain this index, standardized scales with norms and cutoff points to define functional and dysfunctional ranges are needed. In this case, the Spanish norms of the scales were used: Hamilton Depression Scale⁶ with a .82 Cronbach's Alpha, mean 5.8, and a SD of 2.2; Beck Depression Inventory⁷ with a .83 Cronbach's Alpha, mean 5.4, and a SD of 5.4, and the NEO PI-R.⁸ Cutoff points for the three instruments are presented in Table 1.

Additionally, these authors and others (see Iraurgi⁴ for a review) suggested that the most effective way to estimate if an individual score has reliably and significantly changed is to express the magnitude of the effect. To do so, the magnitude of the change (difference between pretest and posttest) is related to a measurement estimation error calculated on the basis of the reliability of the instrument used and also from the variability of the scores in the normative group. This estimate is the Reliable Change Index (RCI) and the calculations needed are as follows:

• -

  Step 1. Calculate the standard error of the mean: S_E = S_x·(1 − r_xx) ½, or S_E = S_x·√(1 − r_xx)

• -

  Step 2. Calculate the standard error of the difference: SE_dif = (2S_E²)^1/2, or SE_dif = S_E·√2

• -

  Step 3. Calculate the reliable change score: RCI = (x₂ − x₁)/SE_dif

where S_x is the standard deviation of scores of the test in the normative group; r_xx is the reliability coefficient of the test; x₂ is the posttest score and x₁ is the pretest score obtained. The resulting RCI is expressed using standardized z scores: values ≤−1.96 or ≥1.96 are statistically significant with a probability below 5% (p ≤ .05). Table 1 shows the RCI for the scales and subscales of the three instruments used with this patient, resulting from the algorithm described and using the reliability indexes and SDs of the scales. An application using Excel was design for the calculations.

Finally, Jacobson and Truax¹ define a clinical significant change as one that on the one hand, reaches the cutoff point in terms level of functioning and, on the other, the magnitude is statistically significant. Fulfillment, or not, of one or both criteria results in the following categories: no change (both criteria unfulfilled), nonsignificant change (above the clinical cutoff score but without statistical significance); significant change (a statistically significant result without surpassing the clinical cutoff point) and clinically significant change (both criteria are fulfilled). Table 1 shows the category reached in this case for the scales and subscales of the instruments used.

In order to illustrate the results take as an example the Hamilton Depression Scale (last row in Table 1). The patient obtained, before the ECT treatment a score of 28 far above the cutoff point for severe depression (CP = 19). Therefore, the patient presented high intensity and frequency of symptoms that corresponded to a 99 percentile in the norms. Following the intervention, the HAM-D score was 12 somewhere between the cutoff point for mild depression (CP = 8) and moderate depression (CP = 14) though she still reached a percentile of 95. Magnitude of change is 16 points associated with a RCI of −12.5, a value indicating a statistically very significant change (associate p value is below .0001). However, in this case, the scores have not reached the cutoff point indicating functionality (PC 8) but level of symptoms has been reduced indicating an improvement in intensity and frequency of symptoms that has not occurred by chance. Therefore, this is a clinically meaningful and statistically significant change.

Discussion

A total of 38 variables were used to evaluate change in a depressed patient treated with ECT. Six variables were statistically and clinically meaningful, three of them clearly associated with the goals of the intervention: the HAM-D, the DBI and the depression facet of the NEO-PI-R.

There were no changes in the five NEO-PI-R factors, an expected finding given the underlying assumptions regarding its character as broad personality traits and stability. Trait measures are sensitive to long term changes. In the same lines, change was not identified in 18 of the NEO-PI-R facets but it is noteworthy to mention some statistically significant changes observed and other not. For instance, in the self-consciousness facet, the patient moves form a 68 pretest score to a score of 60 at posttest, a nonsignificant change. A dysfunctional score is reduced to a functional level, and this could be interpreted as an improvement. However, such an improvement could respond to chance given the fact that the magnitude of change is in the expected range of measurement error. Therefore, having RCI helps us in evaluating change and cautiously interprets it.

On the other hand, we also found results that can be meaningful from a clinical perspective even though criteria strictly based on norms could lead us to disregard them. This is the case of results in the hostility facet (change difference of 13, RCI = 2.37) or Gregariousness (CD −14, RCI 2.55). In both cases, RCI values indicated a statistically significant change with a p < .008, this is, in 8 out of 1000 cases we will err when considering that the magnitude of change was due to chance. However, pretest and posttest scores are in the range of functionality and would not be interpreted. Despite this fact, an increase of 13 points in Hostility and a reduction of 14 points in gregariousness may be susceptible of clinical interpretation and viewed as changes attributable to the effect of the intervention.

Finally, consider another score that can have clinical implications: vulnerability. A 73 pretest score changes to 66 at posttest in this patient. Both scores are in the dysfunctionality range and the RCI value (−1.28) does not grant the conclusion of a statistically significant change which leads to no change. However, had the pretest score be of 76, with a 10 point change, the RCI value would have been −2.10, meaning a significant change though not meaningful from a clinical perspective given that the posttest score is in the functionality range. However, such a result will be of interest for the clinicians informing that the change observed is in consonance with the goals of symptom reduction and therefore it would be another criterion to guide future prescriptions.

Conclusion

Data presented in the clinical case discussed show how the reliable change index can be of interest for health providers in guiding decision making regarding observed health results in individual cases. A sound tool is available that goes beyond subjective judgments and offers data to support decisions, controlling for the chance effect in expected results and changes. Besides, it offers a judgment system bases in two levels: clinical and statistical, which allow for the consideration of other results that might have substantial clinical implications. Specifically, in an E.C.T. case it could be a useful tool to asses the efficacy of a continuation and maintenance strategy, as recommend by Martínez-Amoros et al.⁵

Conflict of interest

The authors declare no conflict of interest.