Critical appraisal of the Colombian clinical practice guide for the prevention, diagnosis and treatment of suicidal ideation and/or behaviour (adoption)

Abstract

Introduction

Suicide is a complex, global public health problem. The Colombian clinical practice guideline provides relevant input for its prevention, diagnosis and treatment. The objective was to evaluate the methodological quality, credibility and applicability of the Colombian clinical practice guideline for suicidal behaviour.

Methods

An academic group of 12 evaluators was established to assess the guide and its recommendations in a standardised way, using the AGREE-II and AGREE-REX instruments. The evaluations were given in the range of 0.0–1.0 with 0.7 as a cut-off point for appropriate quality.

Results

The global assessment of the AGREE-II was greater than 0.7 in the dimensions: "scope and objective" (0.86), "clarity of presentation" (0.89), "applicability" (0.73) and "editorial independence" (0.89). The lowest scores were for “participation of those involved” (0.67) and “rigour in preparation” (0.69). With the AGREE-REX, the results in all dimensions were below 0.70, which indicates lower quality and suitability for use.

Conclusions

The adoption process of the Colombian guideline for suicidal behaviour was a rigorous methodological process, while the practice recommendations were valued as of low applicability due to low support in local evidence. It is necessary to strengthen the generation and synthesis of evidence at the national level to give greater support and applicability to the practice recommendations.

Resumen

Introducción

El suicidio es un complejo problema de salud pública global. La guía colombiana de práctica clínica es un insumo relevante para su prevención, diagnóstico y tratamiento. El objetivo es evaluar la calidad metodológica, la credibilidad y la aplicabilidad de la guía de práctica clínica colombiana sobre la conducta suicida.

Métodos

Un grupo académico de 12 evaluadores se estandarizó para la valoración de la guía y sus recomendaciones mediante los instrumentos AGREE-II y AGREE-REX. Las valoraciones se dieron en el intervalo de 0,0–1,0, con 0,7 como punto de corte para apropiada calidad.

Resultados

La valoración global del AGREE-II fue > 0,7 en las dimensiones «alcance y objetivo» (0,86), «claridad de la presentación» (0,89), «aplicabilidad» (0,73) e «independencia editorial» (0,89). Los menores puntajes fueron para «participación de los implicados» (0,67) y «rigor en la elaboración» (0,69). Con el AGREE-REX los resultados en todas las dimensiones estuvieron por debajo de 0,70, lo cual indica menores calidad e idoneidad de uso.

Conclusiones

El proceso de adopción de la guía colombiana sobre conducta suicida fue un proceso riguroso en lo metodológico, en tanto que las recomendaciones de práctica se valoraron como de baja aplicabilidad por el escaso sustento en evidencia local. Se requiere fortalecer la generación y la síntesis de evidencia en el país para dar más soporte y aplicabilidad a las recomendaciones de práctica.

Introduction

Suicide currently explains the death of 800,000 people a year, which represents 1.4% of all deaths and 3–5% of deaths at ages 15–35 years. More deaths occur from suicide than from malaria, breast cancer, wars, or homicide. For every completed suicide, there are more than 20 suicide attempts, which explains why suicide is considered a public health priority.¹

Reducing suicide mortality has been prioritised by the World Health Organization as a global goal included among the sustainable development objectives.² Since 2014, suicide rates in Colombia have shown an upward trend, after 12 years of lower rates. In the 2014–2017 period, the highest rates were reported in males aged over 64 years and in females aged between 15 and 29 years. According to the National Institute of Legal Medicine and Forensic Sciences, 2,696 suicides were recorded in 2018 (suicide rate, 5.93/100.000 inhabitants).³

Clinical practice guidelines (CPG) are recommendations systematically developed to help decision-making between health professionals and patients regarding health care in specific clinical circumstances.⁴ The quality of the guidelines is defined as the confidence that potential biases in their development have been adequately resolved and the recommendations have internal and external validity and apply to practice.⁵

A CPG can be adopted (received without any adjustments), adapted (modified and adapted to the new context, resulting in a new CPG), or created de novo (from new questions). Different aspects determine the choice of approach for a CPG: adoption and adaptation seek to use limited resources more efficiently by leveraging existing efforts and considering factors specific to the context, while the development of a new CPG involves developing new recommendations not included in other guidelines.⁶

In Colombia, there is a CPG on suicidal behaviour, which, when adopted, involved accepting the recommendations of other guidelines and securing the authorisation of the developer group. The regulations in force in the country establish that health service providers must ensure processes and procedures aimed at guaranteeing patient safety and risk management through the adoption of CPGs, preferably those issued by the Ministry of Health. Health and Social Protection. To date, there has not been an appraisal of the CPG independent of the developer group that contributes to contextualising its methodological quality and its applicability to the Colombian context and provides critical elements of judgement for its users. Thus, the objective of this study is to critically appraise the CPG on the prevention, diagnosis, and treatment of suicidal ideation and/or behaviour (adoption) in Colombia from the perspective of a group of clinical users and under the guidelines of the AGREE collaboration.⁷

Methods

Background

The group of appraisers, all assigned to the Faculty of Medicine of the CES University, were trained at the beginning of the second semester of 2019. It was decided to appraise the Colombian CPG on the prevention, diagnosis, and treatment of suicidal ideation and/or behaviour (adoption), as it is one of the latest guidelines incorporated by the Ministry of Health and Social Protection (2017), and due to the importance of the issue addressed from the perspective of public health.

Participants

The CPG review group comprised four psychiatrists, one neuropsychologist with a doctorate in Epidemiology, and seven first—and second-year Psychiatry residents.

Guidelines

This is guideline No. 60 of the General System of Social Security in Health-Colombia. It was developed by the Instituto de Evaluación Tecnológica en Salud [Institute for Health Technology Assessment] (IETS). The adoption development group included five epidemiologists and public health doctors, two psychiatrists, a psychologist, an economist, a professional in international relations and political studies, and a graphic designer. There was a patient representative (family member). The guideline provides 85 recommendations classified according to the SIGN system into A, B, C and D, freely available online.⁸

The recommendations were evaluated and adjusted for implementation in the Colombian setting, attending to the following behaviours: adoption without any modification, non-adoption of the recommendation, adoption of the recommendation as a point of good clinical practice, adoption of the recommendation as a point for implementation of the guideline, or adoption with modifications. The recommendations in the final group were presented in a formal consensus meeting of experts through the modified Delphi methodology with a virtual and in-person round, the latter for those recommendations that did not reach 70% agreement in an assessment of the type: (a) agree/totally agree, or (b) totally disagree /disagree. To contextualise the recommendations to the reality of the country and the national health system, the development group appraised the validity of each recommendation, the availability of technologies, the approval of regulatory agencies, the training of health personnel and access, and the architecture of the health system. In addition to the development group, external experts in the different therapeutic modalities evaluated (psychiatrists, emergency physicians, psychologists, neurologists, nurses) were identified.

Standardisation

Two of the appraisers had prior experience with the AGREE-II instrument (DFRG and DR) and none with AGREE-REX. Before commencing the critical reading of the CPG, all participants were standardised against the application of the AGREE-II with the tool available on the official website of the My AGREE PLUS Consortium (www.agreetrust.org), developed by the AGREE A3 research team, which includes a tutorial + practical exercise. The reliability of these assessments was rated using the multi-facet Rasch model (0.93). Similarly, researchers reached a consensus in those cases where there was variability in the appraisal.⁹

Instrument application

Over the course of 5 months, monthly 4 -h meetings were held. The number of attendees varied at the first 4 meetings. The objective was to provide support to the appraisers by resolving doubts with the researcher with the greatest experience in using the instruments. As of the second meeting, the individual process of the methodological evaluation of the CPG with the AGREE-II instrument began, using the My AGREE PLUS tool (www.agreetrust.org). Once this stage was completed, all appraisers attended the fifth session, in which the AGREE-REX instrument was socialised and applied to the CPG. The entire evaluation process was carried out in 5 months.

Instruments applied

The AGREE Consortium developed in 2003 and improved in 2010 a generic instrument to appraise the rigour of the methodological process and transparency in the development and reporting of guidelines.¹⁰ Although strict methodological requirements are necessary for the appraisal and reporting of guidelines, it is not sufficient to ensure that the recommendations are clinically credible or implementable. The Consortium recently developed a new tool (AGREE-REX) to support the development, reporting, and appraisal of CPG recommendations and to complement AGREE II.¹¹

AGREE II

The AGREE Consortium developed the AGREE⁵ instrument and improved it in AGREE II (2010).^12,13 Both appraise the methodological quality of CPGs. The AGREE II consists of 23 items, organised into 6 quality domains: scope and purpose, stakeholder participation, rigour in development, clarity of presentation, applicability, and editorial independence. It also includes 2 general appraisal questions, in which the appraiser generally judges the items evaluated. To increase the reliability of the evaluation, the developers recommend having at least 2 appraisers.

The AGREE II instrument includes a user manual. Each of the items is rated from 1 (strongly disagree) to 7 (strongly agree). The total score for the domain is expressed between 0 and 100%. The evaluation of each domain includes guiding questions. Based on the record of each individual reviewer throughout the CPG assessment process, these responses are grouped into a number of categories arranged in the results section under the heading of narrative synthesis.

AGREE-REX

This is a reliable and valid instrument for appraising and optimising the quality of CPG recommendations that are defined as credible, implementable, and reliable. The three factors to consider to ensure that a recommendation is of high quality are: (a) the credibility of the recommendations based on the available evidence added to the suitability of its application to the target population; (b) the implementability in the objective context of the recommendations, and (c) the consideration of the values of all the authors in their formulation.

This instrument comprises nine items divided into three theoretical domains. The first is clinical applicability, which contains the evidence items, applicability to target users, and applicability to patients/populations. The second domain is values and preferences, and items include the values and preferences of the target users, patients/populations, decision/policy controllers, and guideline developers. The final domain is implementability, with the items of purpose and local request and adoption. Each of the items has a list of specific criteria that characterise the concept. Each item is rated on 2 scales: quality assessment (from 1, lowest quality, to 7, highest quality) and suitability for use (from 1, strongly disagree, to 7, strongly agree). Similarly to AGREE-II, the total domain score is expressed between 0 and 100%.

Inter-appraiser bias

One of the potential difficulties faced by the critical evaluation of a CPG is the disagreement between the appraisers' evaluations. Seeking to resolve this issue, the multi-faceted Rasch measure¹⁴ was used, which allows evaluating the biases that the appraisers may have incurred (rigour, leniency, degree of coincidence). It was found that all appraisers were systematic in their evaluations; no unacceptable inconsistencies were detected (outfit = 1.64), there was an appropriate correlation between the appraisers, and no unacceptable rigour or leniency was detected (< ±1 logit), owing to which the appraisers showed high degree of agreement and reliability in their evaluations. For both the AGREE-II and the AGREE-REX, the assessments were reviewed and agreed upon in periodic critical reading control meetings.

Analysis of the information

The information analysis was carried out in Microsoft Excel® for Mac, v. 16.31, following the guidelines of the AGREE collaboration manuals. For AGREE-II and AGREE-REX, the results in each domain were considered high quality with qualifications >70% and low quality <30%; intermediate values were considered moderate quality.

The scores in the items are presented in the results figures along with the domains, to contextualise the reasons why in some cases the cut-off of 70% was not achieved. To render this presentation comparable, the results of the items originally on a scale from 1 to 7 were re-expressed in their percentage equivalent, adapting the formula used to calculate the value per domain such that the equivalences give are 1 = 0%, 2 = 17%, 3 = 33%, 4 = 50%, 5 = 67%, 6 = 83% and 7 = 100%. score¯ n-1(7-1).

For the analysis of inter-appraiser bias, the free-to-use software Many-Facets Rasch Measurement (MFRM) for Rasch analysis (Facets version 3.83.6) was used.

Ethical considerations

This research is classified as risk-free according to Colombian regulations.

Results

In the global appraisal of the CPG using the AGREE-II instrument (Fig. 1), appraisals > 0.7 were obtained in the "scope and purpose" (0.86), "clarity of presentation" (0.89), "applicability" (0.73) and "editorial independence" (0.89) dimensions. The dimensions that did not attain this appraisal were "participation of those involved" (0.67) and "rigour in preparation" (0.69). In the analysis of the items that make up these 2 dimensions, it can be observed that the unfavourable appraisals are limited to the items "perspective of the population" (0.22) in the participation domain, and the items "strengths and limitations" (0.46), “benefits, side effects, and risks” (0.37) and “review by external experts” (0.33). Regarding the global appraisal with AGREE-II, seven readers concluded they would recommend it with modifications, and two concluded they would recommend it in its current state.

Fig. 1.

Appraisal of AGREE-II by items and domains. The red bars indicate the domains of AGREE-II and the blue bars indicate the items in each domain.

In the specific appraisal of the recommendations using the AGREE-REX, the results in all dimensions were <0.70 in both quality and suitability for use (Fig. 2). In the qualitative appraisal, it was established that a large part of these evaluations are due to the lack of local evidence, which affects their applicability. In the overall appraisal with AGREE-REX, to the question "Would you recommend the guide for use in the appropriate context?", readers responded yes by consensus. However, to the question “Would you recommend the guideline for use in my context?” readers responded yes with modifications.

Fig. 2.

Appraisal of AGREE-REX by items and domains. The bold bars are the domains of AGREE-REX.

Narrative synthesis of the appraisers' observations

The following aspects obtained a positive assessment from the appraisal group: presenting suicide as a serious, real and priority problem; promoting patient care based on respect and empathy; clarity in explaining the characteristics of a safe hospital environment; enhancing the patient support network, and promoting the use of the SADPERSONS instrument to screen for suicide risk.

The following aspects were considered with room for improvement: the lack of context of some recommendations; psychiatrist-centred care; the lack of specificity of some recommendations; wanting to cover so many different aspects, not considering perspectives, preferences or results from the patients' perspective (Table 1).

Table 1.

Reader comments based on the recommendations of the suicide CPG according to thematic section.

Section	Recommendation		Reader comments
Risk factors and suicide risk appraisal	R1−14	R14	Psychiatrist-centred care
Evaluation and treatment of suicidal ideation in primary care	R15−26	R19	Psychiatrist-centred care
		R21	Lack of specificity
Evaluation and treatment of patients with suicidal behaviour in the emergency department	R27−30	R28	Lack of specificity
		R29	Lack of specificity
		R30	Psychiatrist-centred care
Key components of a psychosocial psychiatric evaluation	R31−34	R33	Lack of specificity
Treatment of suicidal behaviour in specialised care (mental health)	R35−39	R38	Out of context
		R39	Lack of specificity
Psychotherapeutic interventions	R40	R40	Lack of specificity
Pharmacological treatment	R41−44.1	R41	Lack of specificity
		R42-R45	Lack of context and specificity
Sedation	R45−46	R45	Lack of specificity
Patient discharge planning	R48−51	R48	Lack of context and psychiatrist-focused attention
		R50	Out of context
General suicidal behaviour prevention/media access restriction programmes	R52−58	R52−54, R55	Out of context
Training programmes for the prevention of suicidal behaviour	R59−60	R59	Lack of specificity
Suicide risk screening	R61−65	R61−62	Lack of specificity
Suicidal behaviour in children and adolescents	R66−71	R66	Out of context
		R70	Lack of specificity
Assessment and treatment of people with chronic or continuous suicide risk	R74−77	R74	Lack of clarity
Prevention of suicidal behaviour in risk groups	R78−79.1	R78−79.1	Out of context, lack of specificity
Interventions in family, friends and professionals	R80−84	R80−84	Out of context

Discussion

Suicidal behaviour persists as a major public health problem around the world. Numerous countries and medical and psychological associations have developed CPGs to alert and prevent suicide deaths.^15–19

This study critically appraised the Colombian CPG on preventing, diagnosing, and treating suicidal ideation and/or behaviour using the AGREE and AGREE-REX instruments, which focus on methodological rigour in the guideline adoption process and the quality of the recommendations.^20,21 However, the potential benefits of guidelines depend largely on their quality, owing to which rigorous development methods and strategies are important factors for the implementation of CPG recommendations.^22–24

Not all CPGs attain the quality required to achieve the reported objectives.^25,26 Some studies have reported that high scores on the AGREE II instrument do not guarantee that the CPG recommendations are optimal.^27,28 Nuckols et al.²⁸ appraised the “perceived understandability” (perceived relevance to everyday clinical situations) and “perceived validity” (consistency with their understanding of existing evidence and opinions) of 5 musculoskeletal CPGs. AGREE II scores were of high quality, for example, in the domain of methodological rigour (>80), which indicates high technical quality. Nonetheless, four appraisers found that only four guidelines were moderately complete and valid and the fifth guideline was not valid overall. Of the topics covered by each guideline, they found that 50–69% were "complete" and 6–50% were valid. The appraisers reported that the CPGs omitted common clinical situations and that they doubted the validity of the recommendations. This shows some similarity to our findings of better scores on AGREE II than on AGREE-REX and raises the possibility that guideline developers and their users have different perceptions of what the CPG should offer.

When CPG developers or adapters choose a recommendation based solely on AGREE II quality scores, the CPG end user may consider it out of context, as occurred in our case. When a CPG does not consider the preferences and values of patients, families, and those who finance the health system, as in this case, barriers to its implementation can be increased, especially in a controversial topic such as suicide, which generates stigma in all levels of society and, of course, within the health system.²²

Despite being a recently published instrument, the AGREE-REX instrument has been used in the assessment of other guidelines^29–35: the CPG for uveal melanoma²⁹; the CPG for achalasia,³⁰ which included guidelines developed in accordance with the AGREE-REX guidelines; a systematic review of CPGs in pressure injuries (pressure ulcers),³¹ which assessed the quality and applicability of recommendations from 12 CPGs and 14 related documents; a systematic review and synthesis of CPGs to guide the intensive care of patients requiring mechanical ventilation in periods of high demand during the COVID-19 pandemic,³² which included 9 guidelines selected with AGREE II and AGREE-REX; a review of the methodological quality of 19 dietary guides³³ which, according to the authors, did not meet the minimum methodological threshold (≥50% in the 6 domains of AGREE-II), owing to which they did not use AGREE-REX; a critical appraisal of evidence-based and consensus guidelines for actinic keratosis,³⁴ which included 3 guidelines, and an evaluation of CPG on early mobilisation after a stroke,³⁵ which included 18 CPGs. Forty-seven percent of appraisers found that the guidelines were “too broad or vague,” which reduces their specificity and clinical applicability. In some of these studies^29,31,34 the authors reported higher scores for the domains of AGREE II than for those of AGREE-REX, which coincides with our findings.

New CPGs relating to suicide published after the Colombian one were identified. The first focuses on suicide risk among United States veterans,³⁶ and the second is a CPG on response to suicide-related crises in the emergency department, acute situations and subsequent care,³⁷ which included a developer panel of 89 members: 50 users with life experiences related to suicide and 39 professionals. In addition, a Consensus of the International Society of CNS Clinical Trials and Methodology³⁸ was found, focused on the evaluation of suicidal ideation and behaviour, and led by the pharmaceutical companies Pfizer and Janssen.

Limitations

This study has the following limitations: (a) as the group of appraisers belong to the same university institution, the opinions and evaluations of the CPG are the result of their study context and clinical practices, which renders them less generalisable; (b) there were no professionals from areas other than medicine to enrich the perspective and assessment of the CPG; and (c) along these lines, there was no participation of patients and family members to enrich the critical appraisal.

Based on this research, the authors recommend the joint use of AGREE II and AGREE-REX in future processes for updating the Colombian CPGs. This would provide information on the quality of the methodology and the recommendations and partially resolve the challenges of moving on directly to costly and complex implementation commitments with CPGs that may lack rigour and suitability to the setting in which they are to be applied.

Conclusions

The CPG adopted for the prevention, diagnosis, and treatment of suicidal ideation and/or behaviour, despite including high-quality methods, obtained an unfavourable appraisal of its recommendations. This is explained by the context of the applicability of the evidence produced in other countries and regions. Finally, we consider the need to strengthen the generation and synthesis of evidence in the country to provide greater support and applicability to clinical practice recommendations.

Conflicts of interest

The authors have no conflicts of interest to declare.