Systematic Client Feedback: A naturalistic pilot study

Abstract

Introduction

Routine outcome monitoring or systematic client feedback (SCF) continues to garner empirical support. The Partners for Change Outcome Management System (PCOMS) is one application of SCF with significant research support but no studies have been conducted in Spain. This investigation describes the effects of PCOMS in routine practice via a comparison to published PCOMS studies.

Material and methods

In a retrospective observational study, PCOMS was implemented with 42 clients treated in routine psychotherapy. Outcomes were measured by the Outcome Rating Scale (ORS) and descriptively compared to PCOMS studies included in a meta-analysis and a PCOMS benchmarking study regarding reliable and/or clinically significant change, no change, and deterioration.

Results

The current study achieved similar rates of reliable and clinically significant change (73.8%) to the benchmarking study (65.6%) and the RCTs (67.5%). Regarding no change rates, the current study (23.8%) mirrored results of the RCTs (25.6%) and was better than the benchmarking study (35.4%). The current study incurred a low 2.4% deterioration rate compared to a 10.7% and 7.0% rate of the benchmarking study and RCTs, respectively.

Conclusions

While not without flaws, this pilot study offers some evidence that the improved outcomes associated with PCOMS may also occur in psychotherapy settings in Spain.

Resumen

Introducción

La monitorización sistemática de resultados o recolección sistemática del feedback del cliente (SFC) continúa acumulando apoyo empírico. El Partners for Change Outcome Management System (PCOMS) es una aplicación de la SFC cuya eficacia no ha sido investigada en España. Este estudio describe los resultados tras implementar PCOMS en un contexto de práctica clínica rutinaria, y los compara con los estudios de referencia disponibles en la literatura especializada.

Material y métodos

En un estudio observacional retrospectivo, PCOMS se implementó con 42 clientes tratados en psicoterapia rutinaria. Los resultados se midieron mediante la Escala de Calificación de Resultados (SRO) y descriptivamente en comparación con los estudios de PCOMS incluidos en un metaanálisis y un estudio de evaluación comparativa de PCOMS con respecto a cambios confiables y/o clínicamente significativos, sin cambios y deterioro.

Resultados

Se encontraron tasas similares de cambio fiable y cambios clínicamente significativos (73.85) a las obtenidas en el estudio de referencia (65.6%) y ECA (67.5%). En cuanto a las tasas de ausencia de cambios, el presente estudio (23.8%) mostró los mismos resultados que los ECA (25.6%) y mejores que los del estudio de referencia (35.4%). La tasa de deterioro (2.4%) fue inferior al 10.7 y al 7.0% obtenidos en el estudio de referencia y en los ECA, respectivamente.

Conclusiones

Con algunas limitaciones que se describen, este estudio piloto ofrece evidencia preliminar de que la mejora de resultados asociada a PCOMS puede darse también en población española.

Introduction

Over a decade ago, the American Psychological Association (APA) Taskforce on Evidence Based Treatments (2006) recommended that clinicians routinely collect and utilize client-report data to inform treatment. Since then, growing research support of Routine Outcome Monitoring (ROM) or Systematic Client Feedback (SCF) (e.g., Delgadillo et al., 2018; She et al., 2018) has not only empirically validated the Task Force recommendation but also increased its use in varied psychotherapy settings. SCF refers to the continuous monitoring of client perceptions of progress throughout therapy and a real-time comparison with an expected treatment response (ETR) to gauge client progress and signal when change is not occurring as predicted (Duncan & Reese, 2015). With this alert, clinicians and clients have an opportunity to shift focus, revisit goals, or alter interventions before deterioration or dropout (Duncan, 2014; Gimeno-Peón, Barrio-Nespereira, & Prado-Abril, 2018; Prado-Abril, Sánchez-Reales, & Inchausti, 2017).

The need for SCF is based on findings that many clients quit therapy prematurely (Swift & Greenberg, 2012) or have negative outcomes (Reese, Duncan, Bohanske, Owen, & Minami, 2014). The need is also reinforced by the poor outcomes reported in real world settings (e.g., Hansen, Lambert, & Forman, 2002). Client feedback systems aim to identify treatment failures before they occur, allowing time for clinicians to restore therapy to a positive trajectory. More than simply monitoring and collecting data (Kendrick et al., 2016), practitioners must use the client feedback to identify and solve the obstacles of the psychotherapeutic process to improve treatment effectiveness (Delgadillo et al., 2018; Duncan, 2014; Gimeno-Peón et al., 2018).

Several feedback systems have emerged (Castonguay, Barkham, Lutz, & McAleavey, 2013), but only two have randomized clinical trial (RCT) support and have been designated as evidence based practices. First is the Outcome Questionnaire 45.2 (OQ; Lambert, 2015) System. Lambert is the pioneer of SCF, evolving outcome measurement to a “real-time” feedback process with a proven track record of improving outcomes (Lambert & Shimokawa, 2011). The other SCF intervention is the Partners for Change Outcome Management System (PCOMS; Anker, Duncan, & Sparks, 2009). Emerging from clinical practice and designed with the front-line clinician in mind, PCOMS employs two, four item scales, one focusing on outcome given at the beginning of the session (the Outcome Rating Scale [ORS]; Miller, Duncan, Brown, Sparks, & Claud, 2003) and the other assessing the therapeutic alliance administered toward the end of the encounter (the Session Rating Scale [SRS]; Duncan et al., 2003). PCOMS directly involves clinicians and clients in an ongoing process of measuring and discussing both progress and the alliance, the first system to do so. Both PCOMS and the OQ System are a-theoretical, can be integrated into any approach, and can be used across client diagnoses and from a transdiagnostic approximation (e.g., González-Blanch et al., 2018).

PCOMS evolved from “client directed, outcome-informed” clinical work (Duncan & Miller, 2000). Building on common factors research (Lambert, 1986), Duncan, Solovey, and Rusk (1992) proposed a “client directed” psychotherapy that privileged client perspectives regarding the causes and solutions of problems (the client's theory of change) and depended on client resources in any intervention effort. “Outcome-informed” (Duncan & Miller, 2000) began with the use of Lambert's OQ and the Working Alliance Inventory (Tracey & Kokotovic, 1989). From these origins, and concerns regarding the feasibility of longer measures, the Outcome Rating Scale and Session Rating Scale were co-developed (Miller, Duncan, & Johnson, 2002; Miller & Duncan, 2000). Thereafter, based on 2 years of private practice as well as multiple team supervisions in a family therapy community clinic, Duncan created the clinical process of using the ORS and SRS, detailed first in Duncan and Sparks (2002). The need for an operationally defined process for conducting RCTs that spoke only to the clinical use of the ORS and SRS resulted in the name, “Partners for Change Outcomes Management System” or PCOMS (Anker et al., 2009; Duncan & Reese, 2015).

There are seven RCTs that support the efficacy of PCOMS over treatment as usual (TAU) in individual (Brattland et al., 2018; Reese, Norsworthy, & Rowlands, 2009; She et al., 2018), couple (Anker et al., 2009; Reese, Toland, Slone, & Norsworthy, 2010), and group (Schuman, Slone, Reese, & Duncan, 2015; Slone, Reese, Mathews-Duvall, & Kodet, 2015) therapy. In addition, benchmarking studies have found outcomes comparable to RCTs in a public outpatient setting that implemented PCOMS (Reese et al., 2014) as well as with a psychiatric inpatient unit (Reese et al., 2018).

Dissemination of SCF has also been enhanced by the development of new technologies for data collection and analysis in everyday clinical settings (Prado-Abril, 2016). Widespread data collection enables the creation of algorithms for normative trajectories that guide both clinician and client in all decisions that affect treatment (Delgadillo et al., 2017), thus creating more consistent quality of care across clients. For example, technological application of PCOMS across the US and 20 other countries has resulted in over 1.5 million administrations of the ORS and SRS in the data base and predictive algorithms (Duncan & Sparks, 2018).

The empirical support for PCOMS is promising but questions remain about cross-cultural applications. For example, are these findings generalizable to clinical practice in Spain? Pragmatic studies offer a way to evaluate effectiveness of any given intervention (e.g., Weinfurt et al., 2017). Unlike clinical trials, pragmatic studies assess effectiveness in naturalistic contexts, prioritizing external and ecological validity over internal validity. Pragmatic studies, often called practice oriented research (POR; Fernández-Álvarez & Castonguay, 2018), enable the evaluation of target populations in target settings and promote collaboration between clinicians and researchers (Weinfurt et al., 2017).

To this end, this pilot study explored the effects of PCOMS in routine practice via a descriptive comparison of the results with a published benchmarking study of PCOMS (Reese et al., 2014) and three PCOMS RCTs included in the Lambert and Shimokawa meta-analysis (2011). We hypothesized that outcomes generated in this setting by this provider would favorably compare with these published studies of PCOMS regarding reliable and/or clinically significant change, no change, and deterioration.

Method

Participants

The sample was recruited from the clinic of a clinical psychologist (AGP) with seven years of clinical experience who practices a systemic integrative theoretical orientation. Eligible candidates were those treated between February 2016 and June 2018, older than 16 years, who completed the PCOMS scales throughout all sessions, and who attended a minimum of two sessions.

Out of 75 candidates, 42 individuals (19 males and 23 females) met the selection criteria. The mean age of participants was 33.41 years (SD = 11.17, range 17–49). Reasons for service included: anxiety disorders (31.0%); mood disorders (23.8%); and the remaining 45.2% consisted of couple problems, family issues, vulnerable personality traits, and substance misuse, etc.). Table 1 shows the socio-demographic and clinical characteristics of the sample.

Table 1.

Sample characteristics and status at most recent session.

	N = 42
Socio-demographic characteristics
Females	23 (54.7%)
Mean age (SD)	33.41 (11.17)
Singles	29 (69%)
Labor assets	25 (59.5%)
Education
Secondary	4 (9.5%)
High School-Vocational	16 (38%)
University	18 (43%)
Clinical characteristics
Diagnoses
Anxiety disorders	13 (31%)
Mood disorders	10 (23.8%)
Other problems of clinical relevance	19 (45.2%)
Status at most recent session
Continue in psychotherapy	11 (26.2%)
Premature termination	14 (33.3%)
Planned discharge	17 (40.5%)
Average of psychotherapeutic sessions (SD)	7.5 (6.8)

Instruments

The Outcome Rating Scale (ORS; Miller et al., 2003). The first part of the PCOMS feedback intervention is the administration of the ORS at the beginning of each session and the discussion of client progress. The ORS (see Fig. 1) is a Visual Analog Scale consisting of four 10 centimeter lines, corresponding to four domains (individual, interpersonal, social, and overall), allowing for the client's idiosyncratic rendering of his or her life circumstance. Clients place a mark on each line to represent their perception of their functioning in each domain if using a paper and pencil version or touch or click on an iPad or other device if using a web based application. A centimeter ruler is used to measure the distance to the nearest millimeter from the left end of the scale to the client's mark on each of the four lines, or it is automatically scored by the web system. The scores range from 0 to 40, with lower scores signaling higher distress. The web version of the PCOMS (betteroutcomesnow.com) was used in this study.

Fig. 1.

The Outcome Rating Scale (ORS) and Session Rating Scale (SRS). Copyright 2000, 2002, respectively by B. L. Duncan and S. D. Miller. For examination only. Free working copies for individual use in 28 languages are available at https://www.betteroutcomesnow.com/.

Multiple psychometric studies and RCTs have validated the reliability and validity of the English ORS (Duncan & Reese, 2015), and hundreds of thousands of administrations have determined cutoff and reliable change norms (Duncan & Sparks, 2018; Jacobson & Truax, 1991). The reliable change index of the ORS is 6 points; the clinical cutoff is 25. Scores under 25 reflect the distress level of persons seeking psychotherapy. Clinically significant change is denoted by a change of 6 points and the crossing of the cutoff at termination. The Spanish version of the ORS used in the current study has demonstrated good internal consistency (α = .96) and strong concurrent validity (r = .70) with the Clinical Outcomes in Routine Evaluation – Outcome Measure (CORE-OM; Moggia, Niño-Robles, Miller, & Feixas, 2018), and is consistent with other psychometric studies of the English-based ORS.

The Session Rating Scale (Duncan et al., 2003). The second component of the PCOMS feedback intervention is the administration of the SRS toward the end of each session and the discussion of client perceptions about the alliance. The SRS is also a four-item visual analog scale based on Bordin's (1979) classic delineation of the components of the alliance: the relational bond and the degree of agreement between the client and therapist about the goals and tasks of therapy (see Fig. 1). Clients place a mark or touch on a 10-cm line nearest the pole that best describes their felt experience with their therapist. The client's marks on the four items are either measured by a centimeter ruler and totaled or automatically scored for a total ranging from 0 to 40.

The English SRS also has evidence of generating reliable and valid scores in both psychometric and RCT investigations (Duncan & Reese, 2015). Regarding the cutoff score for the SRS, a conservative estimate derived for clinical purposes with descriptive statistics (score at which the majority of clients are above) is 36 (Duncan & Sparks, 2018). The Spanish SRS has shown similar properties to the English SRS (α = .94; Moggia, Niño-Robles, Feixas, & Miller, 2017).

Procedure

The clinician incorporated the clinical process of PCOMS (Duncan & Sparks, 2018) in his routine practice (based in a brief non-manualized systemic-integrative approach). The client's progress in therapy and perspective of the alliance was monitored and discussed in each session allowing treatment to be individually tailored to the needs of the client (Duncan, 2014). Fig. 2 depicts the web application of PCOMS that plots the client scores against the expected treatment response, allowing ongoing discussion with the client about the benefit, or lack thereof, of services. Such discussions promote psychotherapy as a vehicle for change and open a collaborative space for shared decision-making about all aspects of treatment.

Fig. 2.

The betteroutcomesnow.com graph with Outcome Rating Scale (ORS) scores, Session Rating Scale (SRS) scores, and the expected treatment response (ETR). Progress meter shows client to be less than 50% of ETR, suggesting a conversation about changing therapeutic directions.

Data analyses

An inspection of Table 2 reveals the current study descriptively compared to the Reese et al. (2014) benchmarking investigation conducted in the US (N = 5168) and the RCTs (Anker et al., 2009; Reese et al., 2009; N = 558) conducted in Norway and the US, respectively, used in the Lambert and Shimokawa (2011) meta-analysis. As depicted in Table 2, the three studies were grouped into four categories according to the magnitude of the differences in the ORS scores between the first and the last session (Bauer, Lambert, & Nielsen, 2004). The first category, reliable change, included those cases that obtained a pre-post difference of 6 points or higher (reliable index of change of the ORS). The second category, clinically significant change, reflects a pre-post difference in the ORS scores of 6 points and, at post-test, the client transcended the clinical cutoff of the ORS (25 points). The third category (no change) refers to those clients who neither worsen nor improve in a significant way; i.e., improvements or deteriorations that do not reach 6 points of difference. Finally, the category of deterioration includes those whose ORS scores decrease 6 or more points with respect to the first session.

Table 2.

Descriptive statistics and outcome classifications on the Outcome Rating Scale by study.

	N = 42^aMean or # (SD or %)	N = 5168^b	n = 299^c
ORS score at first session	13.59 (8.03)	19.38 (8.17)	18.26 (8.19)
ORS score at last session	25.48 (9.63)	25.18 (9.05)	27.65 (9.39)
Pre-post ORS scores difference	11.88	5.8	9.39
Effect size^d	1.48	.71	1.15
Reliable change	14 (33.3)	1173 (22.7)	N/A
Clinically significant change	17 (40.5)	1618 (42.9)	N/A
RCSC	31 (73.8)	2791 (65.6)	203 (67.5)
No change	10 (23.8)	1830 (35.4)	77 (25.6)
Deterioration	1 (2.4)	555 (10.7)	21 (7.0)

Note: # = number of clients in category; ORS = Outcome Rating Scale; N/A = not available; RCSC = reliable and/or clinically significant change.

a

Current study.

b

Reese et al. (2014); N = 5168 but available for achieving clinically significant change n = 3774.

c

Anker et al. (2009) and Reese et al. (2009); N = 558 but n = 299 in feedback condition.

d

Pretreatment–posttreatment effect sizes were calculated by dividing the mean difference by the pretreatment standard deviation.

We also tracked client status at the most recent session: the percentage of clients who were still in active treatment (continue in psychotherapy); those who prematurely terminated (unilateral resolution of the process); and those who had a planned discharge (consensual resolution of the process).

Results

The 42 clients in the current study received 7.5 psychotherapy sessions (SD = 6.8, range 2–32): 26.2% remained in active treatment, 33.3% terminated therapy prematurely, and the remaining 40.5% were discharged by consensual decision (see Table 1). Table 2 provides descriptive statistics and compares the outcome categories of the three studies. The first session mean of the current study was 13.59, indicating significant distress. The mean at the end of the treatment was 25.48, a gain of 11.88 points, and slightly above the clinical cutoff. Clients, on average, achieved clinically significant change as measured by the ORS; i.e., they changed more than the reliable change index of the ORS (6 points) and transcended the clinical cutoff (25). Regarding the ORS outcome categories, 40.5% achieved clinically significant change, 33.3% changed reliably, 23.8% experienced no change, and 2.4% of the clients deteriorated. Adding the first two categories, 73.8% or 31 participants demonstrated reliable and/or clinical significant improvements. Comparison with the other studies demonstrated more similarities than differences.

Results compared with another published PCOMS studies

Comparing these results to the RCTs included in the Lambert and Shimokawa (2011) meta-analyses demonstrated that this study produced similar results. Clients, on average, in those studies changed 9.39 points and crossed the clinical cutoff. Results exceeded those attained in the benchmarking investigation in public behavioral health with impoverished clients (Reese et al., 2014) regarding average change (5.8 points), although clients here also crossed the clinical cutoff. The first session means, however, were noticeably different. The current study first session ORS mean (13.59) was substantially lower (indicating higher distress) than both the benchmarking first session mean (19.38) and the RCT first session mean (18.26). The small size of the sample is perhaps the best explanation but more research is needed with clinical populations in Spain.

Regarding the outcome categories, the three data sets look very similar with rates of reliable and clinically significant change of 73.8% (current study), 65.6% (benchmarking study), and 67.5% (the RCTs). These findings corroborate the effects reported in other PCOMS trials (see Duncan & Reese, 2015). Similarly, meta-analyses of psychotherapy efficacy typically find between 56.7% and 67.2% improvement; in contrast, naturalistic studies without SCF have reported improvement as low as 20% (Hansen et al., 2002). Regarding the no change rates, the current study (23.8%) mirrored results of the RCTs (25.6%) and was better that the benchmarking study (35.4%). The biggest difference among the outcome categories occurred in deterioration rates. The current study incurred a very low 2.4% deterioration rate compared to a 10.7% and 7.0% rate of the benchmarking study and RCTs, respectively. Deterioration rates are often reported to be 8–10% (Cuijpers, Reijnders, Karyotaki, de Wit, & Ebert, 2018; Prado-Abril et al., 2017). Again, the small sample size is likely the best explanation.

Discussion

This study examined the effects of a systematic client feedback system in routine psychotherapy practice. An analysis of reliable and clinically significant change revealed that 73.8% of participants benefitted from treatment. Clients, on average, improved by 11.88 ORS points and crossed the clinical cutoff of 25, thereby achieving clinically significant change. It is notable that clients who ended treatment prematurely are included in the analysis.

A higher rate of premature termination (33.3%) occurred compared to the rate reported (19.7%) in a recent meta-analysis (Swift & Greenberg, 2012). Depending on the type of analyses and how dropouts are defined, however, the variability of premature termination rates can range from 20 to 60% (Swift & Greenberg, 2012). A closer case-by-case look of the current study revealed that 6 (42.8%) of the 14 clients who had prematurely terminated had a pre-post improvement of 6 or more points in the last session. Duncan (2014) asserts that so called “drop outs” should only be counted if a reliable change has not occurred, arguing that the termination may be unplanned from the clinician's perspective but it isn’t “premature” from the client's point of view. Finally, the private healthcare context may also have influenced the rate in the current study. Services are unfunded and expediency is highly valued – clients want rapid intervention and quick results. More research is needed of premature termination that is consistent in definition, complete in analysis, and that includes the client's perspective.

As noted, PCOMS is not only a monitoring system to inform the therapist but also requires discussion and collaboration with clients (Sparks & Duncan, 2018). Such a process creates a higher demand on the therapist to incorporate the feedback which requires specialized training, credentials, and clinical expertise (Gimeno-Peón et al., 2018; Inchausti, Prado-Abril, Sánchez-Reales, Vilagrà-Ruíz, & Fonseca-Pedrero, 2018; Prado-Abril et al., 2017). Successful implementation also requires an organizational commitment to data collection, timely identification of not-on-track clients, and dissemination of the data to clinicians and supervisors, as well ongoing attention to adherence and data integrity (Duncan, 2014).

The results of this study are promising but should be interpreted with caution given the limitations. First, the study includes only a single clinician and therefore no control of therapist effects (Prado-Abril et al., 2017). There was also no baseline information with this clinician nor a control group to establish whether PCOMS had any relationship to the reported results. Part of these limitations are due to the fact that the sample was obtained during a limited period of time and in the context of the private practice of a single clinician. The objective was to show the effectiveness of psychotherapy when integrating SCF, comparing the results with benchmark studies. Future research should include a control group that allows to clarify to what extent PCOMS can improve outcomes in routine practice of clinical psychologists from Spain. The use of only one outcome measure in the pre–post treatment analysis also limits the conclusions that may be drawn.

The empirical support for SCF continues to grow. Until this study, it was unknown whether the feedback effect could be replicated in Spain. It was also unknown to what degree cultural differences might influence the effect. While preliminary and not without flaws, this pilot study offers some evidence that the documented improved outcomes associated with PCOMS may also occur in psychotherapy settings in Spain.

Ethics approval and consent to participate

The study was conducted in compliance with local regulations and internationally established principles of the Declaration of Helsinki (64th World Medical Association General Assemble, Fortaleza, Brazil, 2013).

Funding

The authors have no funding to report.

Conflict of interest

Barry L. Duncan is a co-holder of the copyright of the Partners for Change Outcome Management System (PCOMS) instruments. The measures are free for individual use but Duncan receives royalties from licenses issued to groups and organizations. In addition, the web-based application of PCOMS, BetterOutcomesNow.com, is a commercial product and he receives profits based on sales.