Intra-rater reliability and feasibility of the HexCom advanced disease complexity assessment model

Abstract

Aim

To study the intra-rater reliability and feasibility of the HexCom complexity assessment model by analyzing internal consistency, intra-rater reliability and response time.

Design

Test–retest study with a selection of 11 clinical situations that cover the full scope of situations assessed by the HexCom model and which are responded to individually.

Location

Home care, primary care, hospital and sociosanitary care. Two specialized palliative home care teams (PADES).

Participants

A total of 20 professionals comprising 10 experts in palliative home care (PADES) and 10 professionals from general palliative care (primary care, hospital and sociosanitary care). These professionals came from the fields of family medicine (5), internal medicine (2), geriatrics (2), nursing (9), psychology (1) and social work (1).

Main measurements

Cronbach's alpha, weighted kappa, response time.

Results

Cronbach's alpha of 0.91 for HexCom-Red and 0.87 for HexCom-Clin. Intra-rater reliability ranging from good to very good for HexCom-Red (kappa: 0.78–1) and from moderate to very good for HexCom-Clin (kappa: 0.58–0.91). Average response time of 0:57 for HexCom-Red and 3:80 min for HexComClin.

Conclusions

HexCom-Red and HexCom-Clin are reliable tools and feasible for use by all professionals involved in both general and specialized palliative care at different levels.

Resumen

Objetivo

Estudiar la fiabilidad intraobservador y la viabilidad del modelo de valoración de la complejidad HexCom®, a través del análisis de la consistencia interna, la concordancia intraobservador y el tiempo de respuesta.

Diseño

Estudio test-retest con un panel de 11 situaciones clínicas que incluyen todo el abanico de situaciones valoradas por el modelo HexCom® y que son respuestas individualmente.

Emplazamiento

Atención domiciliaria, primaria, hospitalaria y sociosanitaria. Dos equipos de atención domiciliaria paliativa especializada (PADES).

Participantes

Veinte profesionales, 10 expertos en atención domiciliaria paliativa (PADES) y 10 de atención paliativa generalista (atención primaria, hospitalaria y sociosanitaria). Incluye medicina de familia (5), medicina interna (2) y geriatría (2), enfermería (9), psicología (1) y trabajo social (1).

Medidas principales

Alfa de Cronbach, Kappa ponderado, tiempo de respuesta.

Resultados

Alfa de Cronbach de 0,91 por HexCom-Red® y de 0,87 por HexCom-Clin®. Fiabilidad intraobservador entre buena y muy buena por HexCom-Red® (Kappa: 0,78-1) y entre moderada y muy buena por HexCom-Clin® (Kappa: 0,58-0,91). Tiempo de respuesta de 0:57 min de media por el HexCom-Red y de 3:80 min por HexCom-Clin®.

Conclusiones

Los instrumentos HexCom-Red® y HexCom-Clin® son fiables y su uso factible para todas las profesiones implicadas en la atención paliativa, tanto generalista como especializada y de los diferentes niveles asistenciales.

Introduction

End-of-life circumstances differ from one person to the next, as do a person's needs and the complexity of care to be managed by professionals.¹ General palliative care is provided by the community's primary care and hospital professionals while specialized, more resource-intensive palliative care is focused on offering support to situations characterized by greater care complexity.² In other words, it is the level of complexity that determines whether support is required from a team of specialists.

That being so, a care model for patients with advanced disease and/or at the end of life was developed in Catalonia; it is known as the Hexagon of Complexity (HexCom) and is based on the patient's needs and strengths, as well as their care environment.³ It stems from the work of a group of experts coordinated by La Societat Catalano-Balear de Cures Pal·liatives in collaboration with La Societat Catalana de Medicina Familiar i Comunitària under the guidance of the Sociosanitary Master Plan of the Department of Health of the Government of Catalonia.⁴ The model includes six domains of need: clinical, psychological/emotional, social/family, spiritual, ethical and death-related. For each domain, the level of complexity can be low, moderate or high. Various instruments have been produced from this model, including HexCom-Red, which has six items focused on facilitating management/referral between care levels, and HexCom-Clin, which has 18 items for clinical application by specialized teams. In a recent systematic review, HexCom-Clin was deemed one of the most comprehensive complexity assessment tools for palliative care on the international stage.⁵ The two instruments share domains and subdomains, which means the 18 subdomains of HexCom-Clin are covered by the six areas of both tools.

A Delphi study was recently published on the face validity of the model⁶ and included the participation of 14 experts from both primary care and hospitals and both general and specialized palliative care. The results showed a high level of agreement, with a content-item validity rate exceeding 0.92, thus confirming that the model is considered useful for a wide range of health professionals for recording complexity in a coordinated fashion in sociosanitary practice. An earlier study by Esteban using HexCom-Red on a population of 500 patients demonstrated excellent consistency across the intensity of intervention provided by the home care support team (PADES) and primary care (kappa 0.92), as well as across intervention from PADES and hospital specialists (kappa 0.83). Consistency remained at this level and was homogenous when analyzed based on clinical typology (oncology or otherwise) and according to patient prognosis (complex chronic patients, advanced disease or end of life).⁷ Similarly, studies using HexCom-Clin support the predictive, construct and discriminant validity of the model.^8,9 It has thus been corroborated that there exists an association between complexity and place of death, greater complexity and younger patients and cancer patients, and greater complexity and patients with cancer and neurological disorders as they approached death.

One fundamental aspect of the psychometric validation of the instruments in clinical use is completion time and consistency across responses (intra-rater reliability).¹⁰ Neither of these issues has been studied for the HexCom model. Due to the clinical instability of palliative patients, who undergo clinical changes and rapid deterioration, we must often scrutinize written or filmed cases¹¹ and simply accept the limitations inherent to this methodology.

The aim of this study is to evaluate the intra-rater reliability and feasibility of the HexCom complexity assessment model through internal consistency, intra-rater reliability and response time.

Material and methods

Study design: Test–retest study with a selection of 11 clinical situations that cover the full scope of situations assessed by the model. These situations were created by the principal investigator (Annex) and focused on three patient typologies: cancer (three clinical situations), advanced disease due to organ failure (four), and advanced dementia (four). Complexity was assessed on an individual basis immediately after reading each case, and response time was recorded.

Multicenter study: Granollers PADES and Reus PADES.

Participants and setting: Study population: Two populations of currently active professionals who had prior experience with the model and were representative of final users. Group One: Experts in palliative home care working on a PADES team who had participated in prospective multicenter study that used HexCom-Clin. Participants: Four doctors (two specialized in family medicine, one internist and one geriatrician), four nurses, one psychologist and one social worker. Base: Granollers PADES. Group Two: Professionals not specialized in palliative care working in primary care, home care, hospital or sociosanitary care who had participated in the intra-rater reliability study that used HexCom-Red. Participants: Five doctors (three specialized in family medicine, one internist and one geriatrician), five nurses (two from hospital, one from primary care, and one from case management). Base: Reus PADES.

Variables: Items from the HexCom-Clin and HexCom-Red instruments and completion time (in minutes and seconds, timed).

Procedure: A random list was created of combinations of three non-repeating situations, which were later assigned to each of the 10 raters following the list. All 10 raters read the three cases twice over a period of 15 days. Thirty assessments were collected per group, for a total of 60 tests; the process was repeated for the retests (for a total of 120 assessments).

Statistical analysis: Internal consistency was analyzed using Cronbach's alpha and intra-rater reliability was measured using the weighted kappa. The latter was interpreted according to Landis and Koch's criteria¹²: slight (<0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80) and almost perfect (0.81–1.00). The significance level was p ≤ 0.05. The statistics package software Stata for Windows, version 16.0, was used for all analyses.

Sample size: Considering an intraclass kappa agreement of no less than 0:42 (H0), with a significance level of 5% in a one-sided comparison, with a power of 80% and expecting a minimum rate of complexity of 50% for each item, 30 test-retest assessments were needed to determine how significant a true kappa of 0.80 (H1) was.

Timeline: June–September 2019: The clinical situations were written up and the participating professionals were selected. In December 2019, the two test–retest assessments were conducted 15 days apart.

Results

Internal consistency was high for both instruments (Cronbach's alpha: 0.91 HexCom-Red; 0.87 HexCom-Clin). Intra-rater reliability (Table 1) ranged from good to very good for HexCom-Red (kappa: from 0.786 to 1) and from moderate to very good for HexCom-Clin (kappa: from 0.582 to 0.918). In the case of HexCom-Red, all items showed very good consistency (kappa > 0.80) except for the clinical domain, which was good (kappa: 0.78). When agreement was analyzed by domain, it was found to be good for HexCom-Clin in the psychological/emotional, social/family, ethical and death-related domains (kappa: from 0.65 to 0.73), moderate in the clinical domain (kappa: 0.58) and very good in the spiritual domain (kappa: 0.91). As for the subdomains of HexCom-Clin (Table 2), agreement was good for most items (kappa: from 0.64 to 0.78) except for clinical–physical, psychological–emotional, social/family-emotional and -practical, ethical-decisions and -DHD and death-related grief, which were moderate (kappa: from 0.40 to 0.58). As for the resources section, agreement was moderate for most items, and good for microsystem-patient (kappa: 0.74) and very good for mesosystem-family links (kappa: 0.80) and strength-financial (kappa: 0.84).

Table 1.

Intra-rater reliability of complexity domains (weighted kappa).

	HexCom-Red		HexCom-Clin
Domain	Weighted kappa (95% CI)	p	Weighted Kappa (95% CI)	p
Clinical	0.786 (0.581–0.992)	0.000	0.582 (0.341–0.823)	0.001
Use the “Insert Citation” button to add citations to this document.Psychological	0.960 (0.884–1.000)	0.000	0.710 (0.341–0.823)	0.000
Spiritual	0.966 (0.902–1.000)	0.000	0.918 (0.837–0.998)	0.000
Social/family	1.000 (1.000–1.000)	0.000	0.657 (0.399–0.914)	0.000
Ethical	0.921 (0.814–1.000)	0.000	0.734 (0.571–0.897)	0.000
Death-related	0.963 (0.892–1.000)	0.000	0.724 (0.539–0.908)	0.000

Table 2.

Intra-rater reliability of HexCom-Clin complexity subdomains.

Domain of need	Weighted kappa (95% CI)
Clinical physical	0.533 (0.270–0.795)
Clinical therapeutic	0.679 (0.468–0.889)
Psychological personality	0.657 (0.426–0.888)
Psychological emotional	0.428 (0.164–0.694)
Spiritual meaning	0.780 (0.647–0.913)
Spiritual connection	0.809 (0.662–0.957)
Spiritual transcendence	0.873 (0.769–0.977)
Social/family relationships	0.776 (0.610–0.942)
Social/family emotional	0.471 (0.204–0.737)
Social/family practical	0.526 (0.273–0.780)
Social/family external	0.878 (0.744–1.012)
Social/family financial	0.866 (0.750–0.983)
Ethical information	0.675 (0.462–0.888)
Ethical clinical decisions	0.522 (0.340–0.704)
Ethical desire to hasten death	0.462 (0.173–0.750)
Death-related place	0.747 (0.543–0.950)
Death-related situation last days	0.656 (0.440–0.871)
Death-related grief	0.571 (0.355–0.788)
Domain of resources
Strength patient	0.746 (0.523–0.969)
Strength family caretaker	0.432 (0.115–0.749)
Family bonds	0.806 (0.647–0.966)
Professional bonds	0.643 (0.437–0.849)
Strength financial	0.840 (0.688–0.993)
No pending matters	0.581 (0.381–0.783)
Outlook progressive course	0.503 (0.240–0.766)
Team	0.492 (0.144–0.839)
Macrosystem	0.406 (0.187–0.625)
Balance	0.650 (0.370–0.929)

Feasibility: Mean response time in minutes and seconds was 0:57 for HexCom-Red and 3:8 for HexCom-Clin. Response time was significantly lower for the retest (Table 3).

Table 3.

Feasibility response time (mean and standard deviation).

	Test	Retest	Difference	P
HexCom-Red	0.57 (0.28)	0.37 (0.17)	0.20 (0.24)	0.000
HexCom-Clin	3.8 (1.3)	3.0 (1.1)	0.7 (1.0)	0.003

Discussion

The results of this study demonstrate good internal consistency for both instruments, very good intra-rater reliability for HexCom-Red and moderate to good intra-rater reliability for HexCom-Clin, as well as excellent feasibility for both HexCom-Red, which took participants less than one minute to complete, and HexCom-Clin, with a response time of less than four minutes.

There are some limitations inherent to the study design. Among feasibility analysis parameters, test-retest feasibility is a significant psychometric property. Due to the clinical instability of palliative patients, who undergo rapid deterioration, we must often study written or filmed cases.¹¹ Living through a situation in real life, watching a video of it, or reading about it on paper constitute radically different experiences. This fact represents an insurmountable limitation of this type of study. The methodology of our study is like that used to assess the reliability of the Japanese version of Support Team Assessment Schedule (STAS-J), in which 16 nurses and one doctor working in a palliative care unit rated 10 hypothetical cases on two separate occasions.¹³ The interval between assessments is another limitation of the results: we followed the system used to validate the Spanish version of the Supportive and Palliative Care Indicators tool for low-income settings (SPICT™), in which the retest was conducted 10 days after the test.¹⁴ One strong point of our study is that it included professionals from all fields and levels involved in both general and specialized palliative care.

Our results are comparable to those found in the literature and can be considered good when compared with other instruments commonly used in palliative care. In the case of NECPAL CCOMS-ICO©, in a study with two doctors and two nurses, Rodríguez-Calero identified moderate agreement (kappa 0.50),¹⁵ while in the Chilean version, Troncoso described it as ranging from 0.632 and 1.0.¹⁶ As for the Integrated Palliative care Outcome Scale (IPOS), good test-retest reliability has been reported for the subscales physical symptoms (intra-class correlation coefficient [ICC] = 0.77) and emotional symptoms (ICC = 0.72), and moderate for the subscale communication and practical issues (ICC = 0.53).¹⁷ With respect to the Palliative Care Outcomes Collaboration Symptom Assessment Scale (PCOC SAS), Daveson found a weighted kappa of 0.6 (CI 95% 0.54–0.63) for the pain item.¹⁸ Similarly, for the Japanese version of the STAS-J, which was also analyzed using hypothetical cases, a weighted kappa of 0.64–0.85 was reported.¹³

We also discern similarities when comparing our results for response time with those found in the literature. For example, a response time of 2.3–2.4 min was described for the Supportive and Palliative Care Indicators Tool for a low-income Setting (SPICT-LIS).¹⁹ In the case of the Palliative Care Needs Assessment Tool (PC-NAT),²⁰ completion time ranged from 5 to 10 min Participants took around 5 min to complete the NECPAL CCOMS-ICO©, although this figure varies considerably from one study to the next, ranging from 1.9–7.72 minutes in a study by Rodríguez-Calero¹⁵ and 6.7 ± 4.01 minutes in a study by Troncoso.¹⁶ Much like us, Troncoso also found that response time improved with training, decreasing from 6.7 min to 3.9 min the second time around (p ≤ 0.001).

It should be noted that rater subjectivity and, especially, non-linearity, interdependence and changes over time play a significant role in assessing complexity at the end of life.^21,22 In these conditions, professionals responded on an intuitive basis, which is not as easily activated in the context of fictitious written cases as it is in the context of real care. As Gigerenzer puts it, the response to complexity is simplicity, and a truly efficient healthcare response requires mastering the art of focusing on what's important and forgetting the rest.^23,24 In a situation where information is lacking and uncertainty abounds, simple diagnostic methods tend to be the best option.²⁵ In that regard, HexCom-Red falls in line with the growing appearance of tools that allow various professionals in the system to share their vision and personalize patient care through multi-dimensional assessment within a limited time frame and which can be regularly administered by a single professional in any area of the health system.^26,27

The internal consistency (Cronbach's Alpha) and stability of responses (intra-rater reliability) identified in our study, alongside the face validity results,⁶ endorse the use of the HexCom model in clinical practice. Nevertheless, as Carvajal noted, validating an instrument is a continuous and dynamic process that acquires greater consistency as more psychometric properties are measured across different cultures, with different populations and subjects.¹⁰ We have thus sought approval from the IDIAPJGol Clinical Research Ethics Committee for the formal study “Validation of an advanced disease complexity assessment model (HexCom)” (code 19/215-P), which is to be based on real patients rather than written cases and includes assessment of concurrent validation.⁵ This study, however, is currently postponed due to the COVID-19 pandemic. If carrying out fieldwork becomes unfeasible, a set of videotaped role-play simulated situations would be used to test the HexCom tool. These would include the necessary variables to make those scenarios comparable to real-life situations. The analysis of videotaped performances often yields some interpretative variability on the part of the observers. Nevertheless, these allow anticipating the benefits and difficulties of implementing the instrument before it is used in real healthcare settings.

In conclusion, HexCom-Red and HexCom-Clin are reliable instruments and feasible for use by all the professions involved in both general and specialized palliative care at different levels.

What is currently known about the topic

• •

  The criterion for access to specialized palliative care must be the care complexity of patient needs.

• •

  Of the various models designed to assess complexity, the HexCom model is one of the best rated internationally.

• •

  HexCom-Red was designed to manage and refer cases whereas HexCom-Clin is intended for clinical care and research.

What this study contributes

• •

  It demonstrates the reliability of the model, with good results for internal consistency and intra-rater reliability.

• •

  It shows the feasibility of the model, with a response time of less than one minute for HexCom-Red and four minutes for HexCom-Clin.

Funding

This study obtained partial funding from the University Institute for Primary Care Research IDIAP JGol (4R17/053) and a research grant for the principal investigator (XB).

Conflict of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

We would like to thank all the study participants for sharing their time and knowledge with us. We would also like to thank the members of the Granollers and Reus PADES teams. Lastly, we would like to show our appreciation to Txus Micó for his secretarial help.