Suicide is a complex, global public health problem. The Colombian clinical practice guideline provides relevant input for its prevention, diagnosis and treatment. The objective was to evaluate the methodological quality, credibility and applicability of the Colombian clinical practice guideline for suicidal behaviour.
MethodsAn academic group of 12 evaluators was established to assess the guide and its recommendations in a standardised way, using the AGREE-II and AGREE-REX instruments. The evaluations were given in the range of 0.0–1.0 with 0.7 as a cut-off point for appropriate quality.
ResultsThe global assessment of the AGREE-II was greater than 0.7 in the dimensions: "scope and objective" (0.86), "clarity of presentation" (0.89), "applicability" (0.73) and "editorial independence" (0.89). The lowest scores were for “participation of those involved” (0.67) and “rigour in preparation” (0.69). With the AGREE-REX, the results in all dimensions were below 0.70, which indicates lower quality and suitability for use.
ConclusionsThe adoption process of the Colombian guideline for suicidal behaviour was a rigorous methodological process, while the practice recommendations were valued as of low applicability due to low support in local evidence. It is necessary to strengthen the generation and synthesis of evidence at the national level to give greater support and applicability to the practice recommendations.
El suicidio es un complejo problema de salud pública global. La guía colombiana de práctica clínica es un insumo relevante para su prevención, diagnóstico y tratamiento. El objetivo es evaluar la calidad metodológica, la credibilidad y la aplicabilidad de la guía de práctica clínica colombiana sobre la conducta suicida.
MétodosUn grupo académico de 12 evaluadores se estandarizó para la valoración de la guía y sus recomendaciones mediante los instrumentos AGREE-II y AGREE-REX. Las valoraciones se dieron en el intervalo de 0,0–1,0, con 0,7 como punto de corte para apropiada calidad.
ResultadosLa valoración global del AGREE-II fue > 0,7 en las dimensiones «alcance y objetivo» (0,86), «claridad de la presentación» (0,89), «aplicabilidad» (0,73) e «independencia editorial» (0,89). Los menores puntajes fueron para «participación de los implicados» (0,67) y «rigor en la elaboración» (0,69). Con el AGREE-REX los resultados en todas las dimensiones estuvieron por debajo de 0,70, lo cual indica menores calidad e idoneidad de uso.
ConclusionesEl proceso de adopción de la guía colombiana sobre conducta suicida fue un proceso riguroso en lo metodológico, en tanto que las recomendaciones de práctica se valoraron como de baja aplicabilidad por el escaso sustento en evidencia local. Se requiere fortalecer la generación y la síntesis de evidencia en el país para dar más soporte y aplicabilidad a las recomendaciones de práctica.
Suicide currently explains the death of 800,000 people a year, which represents 1.4% of all deaths and 3–5% of deaths at ages 15–35 years. More deaths occur from suicide than from malaria, breast cancer, wars, or homicide. For every completed suicide, there are more than 20 suicide attempts, which explains why suicide is considered a public health priority.1
Reducing suicide mortality has been prioritised by the World Health Organization as a global goal included among the sustainable development objectives.2 Since 2014, suicide rates in Colombia have shown an upward trend, after 12 years of lower rates. In the 2014–2017 period, the highest rates were reported in males aged over 64 years and in females aged between 15 and 29 years. According to the National Institute of Legal Medicine and Forensic Sciences, 2,696 suicides were recorded in 2018 (suicide rate, 5.93/100.000 inhabitants).3
Clinical practice guidelines (CPG) are recommendations systematically developed to help decision-making between health professionals and patients regarding health care in specific clinical circumstances.4 The quality of the guidelines is defined as the confidence that potential biases in their development have been adequately resolved and the recommendations have internal and external validity and apply to practice.5
A CPG can be adopted (received without any adjustments), adapted (modified and adapted to the new context, resulting in a new CPG), or created de novo (from new questions). Different aspects determine the choice of approach for a CPG: adoption and adaptation seek to use limited resources more efficiently by leveraging existing efforts and considering factors specific to the context, while the development of a new CPG involves developing new recommendations not included in other guidelines.6
In Colombia, there is a CPG on suicidal behaviour, which, when adopted, involved accepting the recommendations of other guidelines and securing the authorisation of the developer group. The regulations in force in the country establish that health service providers must ensure processes and procedures aimed at guaranteeing patient safety and risk management through the adoption of CPGs, preferably those issued by the Ministry of Health. Health and Social Protection. To date, there has not been an appraisal of the CPG independent of the developer group that contributes to contextualising its methodological quality and its applicability to the Colombian context and provides critical elements of judgement for its users. Thus, the objective of this study is to critically appraise the CPG on the prevention, diagnosis, and treatment of suicidal ideation and/or behaviour (adoption) in Colombia from the perspective of a group of clinical users and under the guidelines of the AGREE collaboration.7
MethodsBackgroundThe group of appraisers, all assigned to the Faculty of Medicine of the CES University, were trained at the beginning of the second semester of 2019. It was decided to appraise the Colombian CPG on the prevention, diagnosis, and treatment of suicidal ideation and/or behaviour (adoption), as it is one of the latest guidelines incorporated by the Ministry of Health and Social Protection (2017), and due to the importance of the issue addressed from the perspective of public health.
ParticipantsThe CPG review group comprised four psychiatrists, one neuropsychologist with a doctorate in Epidemiology, and seven first—and second-year Psychiatry residents.
GuidelinesThis is guideline No. 60 of the General System of Social Security in Health-Colombia. It was developed by the Instituto de Evaluación Tecnológica en Salud [Institute for Health Technology Assessment] (IETS). The adoption development group included five epidemiologists and public health doctors, two psychiatrists, a psychologist, an economist, a professional in international relations and political studies, and a graphic designer. There was a patient representative (family member). The guideline provides 85 recommendations classified according to the SIGN system into A, B, C and D, freely available online.8
The recommendations were evaluated and adjusted for implementation in the Colombian setting, attending to the following behaviours: adoption without any modification, non-adoption of the recommendation, adoption of the recommendation as a point of good clinical practice, adoption of the recommendation as a point for implementation of the guideline, or adoption with modifications. The recommendations in the final group were presented in a formal consensus meeting of experts through the modified Delphi methodology with a virtual and in-person round, the latter for those recommendations that did not reach 70% agreement in an assessment of the type: (a) agree/totally agree, or (b) totally disagree /disagree. To contextualise the recommendations to the reality of the country and the national health system, the development group appraised the validity of each recommendation, the availability of technologies, the approval of regulatory agencies, the training of health personnel and access, and the architecture of the health system. In addition to the development group, external experts in the different therapeutic modalities evaluated (psychiatrists, emergency physicians, psychologists, neurologists, nurses) were identified.
StandardisationTwo of the appraisers had prior experience with the AGREE-II instrument (DFRG and DR) and none with AGREE-REX. Before commencing the critical reading of the CPG, all participants were standardised against the application of the AGREE-II with the tool available on the official website of the My AGREE PLUS Consortium (www.agreetrust.org), developed by the AGREE A3 research team, which includes a tutorial + practical exercise. The reliability of these assessments was rated using the multi-facet Rasch model (0.93). Similarly, researchers reached a consensus in those cases where there was variability in the appraisal.9
Instrument applicationOver the course of 5 months, monthly 4 -h meetings were held. The number of attendees varied at the first 4 meetings. The objective was to provide support to the appraisers by resolving doubts with the researcher with the greatest experience in using the instruments. As of the second meeting, the individual process of the methodological evaluation of the CPG with the AGREE-II instrument began, using the My AGREE PLUS tool (www.agreetrust.org). Once this stage was completed, all appraisers attended the fifth session, in which the AGREE-REX instrument was socialised and applied to the CPG. The entire evaluation process was carried out in 5 months.
Instruments appliedThe AGREE Consortium developed in 2003 and improved in 2010 a generic instrument to appraise the rigour of the methodological process and transparency in the development and reporting of guidelines.10 Although strict methodological requirements are necessary for the appraisal and reporting of guidelines, it is not sufficient to ensure that the recommendations are clinically credible or implementable. The Consortium recently developed a new tool (AGREE-REX) to support the development, reporting, and appraisal of CPG recommendations and to complement AGREE II.11
AGREE IIThe AGREE Consortium developed the AGREE5 instrument and improved it in AGREE II (2010).12,13 Both appraise the methodological quality of CPGs. The AGREE II consists of 23 items, organised into 6 quality domains: scope and purpose, stakeholder participation, rigour in development, clarity of presentation, applicability, and editorial independence. It also includes 2 general appraisal questions, in which the appraiser generally judges the items evaluated. To increase the reliability of the evaluation, the developers recommend having at least 2 appraisers.
The AGREE II instrument includes a user manual. Each of the items is rated from 1 (strongly disagree) to 7 (strongly agree). The total score for the domain is expressed between 0 and 100%. The evaluation of each domain includes guiding questions. Based on the record of each individual reviewer throughout the CPG assessment process, these responses are grouped into a number of categories arranged in the results section under the heading of narrative synthesis.
AGREE-REXThis is a reliable and valid instrument for appraising and optimising the quality of CPG recommendations that are defined as credible, implementable, and reliable. The three factors to consider to ensure that a recommendation is of high quality are: (a) the credibility of the recommendations based on the available evidence added to the suitability of its application to the target population; (b) the implementability in the objective context of the recommendations, and (c) the consideration of the values of all the authors in their formulation.
This instrument comprises nine items divided into three theoretical domains. The first is clinical applicability, which contains the evidence items, applicability to target users, and applicability to patients/populations. The second domain is values and preferences, and items include the values and preferences of the target users, patients/populations, decision/policy controllers, and guideline developers. The final domain is implementability, with the items of purpose and local request and adoption. Each of the items has a list of specific criteria that characterise the concept. Each item is rated on 2 scales: quality assessment (from 1, lowest quality, to 7, highest quality) and suitability for use (from 1, strongly disagree, to 7, strongly agree). Similarly to AGREE-II, the total domain score is expressed between 0 and 100%.
Inter-appraiser biasOne of the potential difficulties faced by the critical evaluation of a CPG is the disagreement between the appraisers' evaluations. Seeking to resolve this issue, the multi-faceted Rasch measure14 was used, which allows evaluating the biases that the appraisers may have incurred (rigour, leniency, degree of coincidence). It was found that all appraisers were systematic in their evaluations; no unacceptable inconsistencies were detected (outfit = 1.64), there was an appropriate correlation between the appraisers, and no unacceptable rigour or leniency was detected (< ±1 logit), owing to which the appraisers showed high degree of agreement and reliability in their evaluations. For both the AGREE-II and the AGREE-REX, the assessments were reviewed and agreed upon in periodic critical reading control meetings.
Analysis of the informationThe information analysis was carried out in Microsoft Excel® for Mac, v. 16.31, following the guidelines of the AGREE collaboration manuals. For AGREE-II and AGREE-REX, the results in each domain were considered high quality with qualifications >70% and low quality <30%; intermediate values were considered moderate quality.
The scores in the items are presented in the results figures along with the domains, to contextualise the reasons why in some cases the cut-off of 70% was not achieved. To render this presentation comparable, the results of the items originally on a scale from 1 to 7 were re-expressed in their percentage equivalent, adapting the formula used to calculate the value per domain such that the equivalences give are 1 = 0%, 2 = 17%, 3 = 33%, 4 = 50%, 5 = 67%, 6 = 83% and 7 = 100%. score¯ n-1(7-1).
For the analysis of inter-appraiser bias, the free-to-use software Many-Facets Rasch Measurement (MFRM) for Rasch analysis (Facets version 3.83.6) was used.
Ethical considerationsThis research is classified as risk-free according to Colombian regulations.
ResultsIn the global appraisal of the CPG using the AGREE-II instrument (Fig. 1), appraisals > 0.7 were obtained in the "scope and purpose" (0.86), "clarity of presentation" (0.89), "applicability" (0.73) and "editorial independence" (0.89) dimensions. The dimensions that did not attain this appraisal were "participation of those involved" (0.67) and "rigour in preparation" (0.69). In the analysis of the items that make up these 2 dimensions, it can be observed that the unfavourable appraisals are limited to the items "perspective of the population" (0.22) in the participation domain, and the items "strengths and limitations" (0.46), “benefits, side effects, and risks” (0.37) and “review by external experts” (0.33). Regarding the global appraisal with AGREE-II, seven readers concluded they would recommend it with modifications, and two concluded they would recommend it in its current state.
In the specific appraisal of the recommendations using the AGREE-REX, the results in all dimensions were <0.70 in both quality and suitability for use (Fig. 2). In the qualitative appraisal, it was established that a large part of these evaluations are due to the lack of local evidence, which affects their applicability. In the overall appraisal with AGREE-REX, to the question "Would you recommend the guide for use in the appropriate context?", readers responded yes by consensus. However, to the question “Would you recommend the guideline for use in my context?” readers responded yes with modifications.
The following aspects obtained a positive assessment from the appraisal group: presenting suicide as a serious, real and priority problem; promoting patient care based on respect and empathy; clarity in explaining the characteristics of a safe hospital environment; enhancing the patient support network, and promoting the use of the SADPERSONS instrument to screen for suicide risk.
The following aspects were considered with room for improvement: the lack of context of some recommendations; psychiatrist-centred care; the lack of specificity of some recommendations; wanting to cover so many different aspects, not considering perspectives, preferences or results from the patients' perspective (Table 1).
Reader comments based on the recommendations of the suicide CPG according to thematic section.
Section | Recommendation | Reader comments | |
---|---|---|---|
Risk factors and suicide risk appraisal | R1−14 | R14 | Psychiatrist-centred care |
Evaluation and treatment of suicidal ideation in primary care | R15−26 | R19 | Psychiatrist-centred care |
R21 | Lack of specificity | ||
Evaluation and treatment of patients with suicidal behaviour in the emergency department | R27−30 | R28 | Lack of specificity |
R29 | Lack of specificity | ||
R30 | Psychiatrist-centred care | ||
Key components of a psychosocial psychiatric evaluation | R31−34 | R33 | Lack of specificity |
Treatment of suicidal behaviour in specialised care (mental health) | R35−39 | R38 | Out of context |
R39 | Lack of specificity | ||
Psychotherapeutic interventions | R40 | R40 | Lack of specificity |
Pharmacological treatment | R41−44.1 | R41 | Lack of specificity |
R42-R45 | Lack of context and specificity | ||
Sedation | R45−46 | R45 | Lack of specificity |
Patient discharge planning | R48−51 | R48 | Lack of context and psychiatrist-focused attention |
R50 | Out of context | ||
General suicidal behaviour prevention/media access restriction programmes | R52−58 | R52−54, R55 | Out of context |
Training programmes for the prevention of suicidal behaviour | R59−60 | R59 | Lack of specificity |
Suicide risk screening | R61−65 | R61−62 | Lack of specificity |
Suicidal behaviour in children and adolescents | R66−71 | R66 | Out of context |
R70 | Lack of specificity | ||
Assessment and treatment of people with chronic or continuous suicide risk | R74−77 | R74 | Lack of clarity |
Prevention of suicidal behaviour in risk groups | R78−79.1 | R78−79.1 | Out of context, lack of specificity |
Interventions in family, friends and professionals | R80−84 | R80−84 | Out of context |
Suicidal behaviour persists as a major public health problem around the world. Numerous countries and medical and psychological associations have developed CPGs to alert and prevent suicide deaths.15–19
This study critically appraised the Colombian CPG on preventing, diagnosing, and treating suicidal ideation and/or behaviour using the AGREE and AGREE-REX instruments, which focus on methodological rigour in the guideline adoption process and the quality of the recommendations.20,21 However, the potential benefits of guidelines depend largely on their quality, owing to which rigorous development methods and strategies are important factors for the implementation of CPG recommendations.22–24
Not all CPGs attain the quality required to achieve the reported objectives.25,26 Some studies have reported that high scores on the AGREE II instrument do not guarantee that the CPG recommendations are optimal.27,28 Nuckols et al.28 appraised the “perceived understandability” (perceived relevance to everyday clinical situations) and “perceived validity” (consistency with their understanding of existing evidence and opinions) of 5 musculoskeletal CPGs. AGREE II scores were of high quality, for example, in the domain of methodological rigour (>80), which indicates high technical quality. Nonetheless, four appraisers found that only four guidelines were moderately complete and valid and the fifth guideline was not valid overall. Of the topics covered by each guideline, they found that 50–69% were "complete" and 6–50% were valid. The appraisers reported that the CPGs omitted common clinical situations and that they doubted the validity of the recommendations. This shows some similarity to our findings of better scores on AGREE II than on AGREE-REX and raises the possibility that guideline developers and their users have different perceptions of what the CPG should offer.
When CPG developers or adapters choose a recommendation based solely on AGREE II quality scores, the CPG end user may consider it out of context, as occurred in our case. When a CPG does not consider the preferences and values of patients, families, and those who finance the health system, as in this case, barriers to its implementation can be increased, especially in a controversial topic such as suicide, which generates stigma in all levels of society and, of course, within the health system.22
Despite being a recently published instrument, the AGREE-REX instrument has been used in the assessment of other guidelines29–35: the CPG for uveal melanoma29; the CPG for achalasia,30 which included guidelines developed in accordance with the AGREE-REX guidelines; a systematic review of CPGs in pressure injuries (pressure ulcers),31 which assessed the quality and applicability of recommendations from 12 CPGs and 14 related documents; a systematic review and synthesis of CPGs to guide the intensive care of patients requiring mechanical ventilation in periods of high demand during the COVID-19 pandemic,32 which included 9 guidelines selected with AGREE II and AGREE-REX; a review of the methodological quality of 19 dietary guides33 which, according to the authors, did not meet the minimum methodological threshold (≥50% in the 6 domains of AGREE-II), owing to which they did not use AGREE-REX; a critical appraisal of evidence-based and consensus guidelines for actinic keratosis,34 which included 3 guidelines, and an evaluation of CPG on early mobilisation after a stroke,35 which included 18 CPGs. Forty-seven percent of appraisers found that the guidelines were “too broad or vague,” which reduces their specificity and clinical applicability. In some of these studies29,31,34 the authors reported higher scores for the domains of AGREE II than for those of AGREE-REX, which coincides with our findings.
New CPGs relating to suicide published after the Colombian one were identified. The first focuses on suicide risk among United States veterans,36 and the second is a CPG on response to suicide-related crises in the emergency department, acute situations and subsequent care,37 which included a developer panel of 89 members: 50 users with life experiences related to suicide and 39 professionals. In addition, a Consensus of the International Society of CNS Clinical Trials and Methodology38 was found, focused on the evaluation of suicidal ideation and behaviour, and led by the pharmaceutical companies Pfizer and Janssen.
LimitationsThis study has the following limitations: (a) as the group of appraisers belong to the same university institution, the opinions and evaluations of the CPG are the result of their study context and clinical practices, which renders them less generalisable; (b) there were no professionals from areas other than medicine to enrich the perspective and assessment of the CPG; and (c) along these lines, there was no participation of patients and family members to enrich the critical appraisal.
Based on this research, the authors recommend the joint use of AGREE II and AGREE-REX in future processes for updating the Colombian CPGs. This would provide information on the quality of the methodology and the recommendations and partially resolve the challenges of moving on directly to costly and complex implementation commitments with CPGs that may lack rigour and suitability to the setting in which they are to be applied.
ConclusionsThe CPG adopted for the prevention, diagnosis, and treatment of suicidal ideation and/or behaviour, despite including high-quality methods, obtained an unfavourable appraisal of its recommendations. This is explained by the context of the applicability of the evidence produced in other countries and regions. Finally, we consider the need to strengthen the generation and synthesis of evidence in the country to provide greater support and applicability to clinical practice recommendations.
Conflicts of interestThe authors have no conflicts of interest to declare.