Risk of bias assessment of clinical trials published in the Revista Colombiana de Anestesiología

Abstract

Introduction

Randomized clinical trials (RCT) are one of the most reliable methods of scientific investigation in health sciences. It is a corner stone of evidence based medicine and the backbone of high standard knowledge. Several types of errors can compromise the results and affect its validity.

Objectives

To assess the risk of bias of the clinical trials published in the Revista Colombiana de Anestesiología (RCA) medical journal by applying the “risk of bias detection” tool of the Cochrane Collaboration.

Methods

All the clinical trials in the RCA journal were found by carrying out a systematic research. These trials were randomly distributed among 6 evaluators trained in the use of the “risk of bias detection” tool of the Cochrane Collaboration. Results were presented descriptively, graphically and chronologically to each of the 6 parameters that conform the “risk of bias detection” tool.

Results

The RCA journal has published 40 volumes as of 1973. The searching process identified a total 75 RCT up until 2009. The frequency of RCT publication has risen with time. The cities with most publications were Bogotá DC and Medellín, and most trials were related to the management of acute and chronic pain. The greatest risk of bias (29% of all RCT) was found in the concealing of randomization sequences (parameter 2). 30% of the studies showed four or more parameter values of low risk of bias. A trend of decreasing proportion of high risk values was observed as time passed.

Conclusions

There is a sustained trend of improvement and risk reduction in RCTs in the RCA journal.

Resumen

Introducción

El ensayo clínico aleatorizado (ECA) es una de las mejores formas de adquisición de pruebas científicas en ciencias de la salud. Es catalogado como la piedra angular de la medicina basada en la evidencia y como eje de la formación de conocimiento de alta calidad. Diversos tipos de sesgos pueden comprometer sus resultados y afectar su validez interna.

Objetivos

Evaluar el «riesgo de sesgo» de los ensayos clínicos publicados en la Revista Colombiana de Anestesiología (RCA) mediante la aplicación de la herramienta para detección de «riesgo de sesgo» de la Colaboración Cochrane.

Métodos

Mediante una búsqueda sistemática se identificaron todos los ensayos clínicos publicados en la RCA. Estos se distribuyeron de forma aleatoria entre 6 evaluadores entrenados en la utilización de la herramienta para detección de «riesgo de sesgo» de la Colaboración Cochrane. Los resultados se presentaron de forma descriptiva, gráfica y temporal para cada uno de los 6 dominios que constituyen la herramienta.

Resultados

La RCA ha publicado 40 volúmenes desde 1973. El proceso de búsqueda identificó hasta el 2009 un total de 75 ECA. La frecuencia de publicación de ECA ha aumentado con el paso del tiempo, las ciudades con mayor publicación fueron Bogotá DC y Medellín, y en su mayoría están relacionados al manejo del dolor agudo y crónico. El mayor riesgo de sesgo (29% de los ECA) se identificó en el encubrimiento de la secuencia de aleatorización (dominio 2). El 30% de los estudios presentaron 4 dominios o más clasificados como bajo riesgo de sesgo. Se apreció una tendencia a la reducción de la proporción de dominios clasificados como alto riesgo de sesgo con el paso del tiempo.

Conclusiones

Existe una tendencia sostenida al mejoramiento y a la reducción del riesgo de sesgo de los ECA publicados en la RCA, con algunos puntos a fortalecer en el proceso de diseño, conducción, análisis y reporte.

Introduction

Clinical trials are quantitative, comparative and controlled experiments in which a group of researchers study two or more randomly assigned interventions in a group of individuals.¹ It is one of the finest methods of scientific research in health sciences and the design of it provides a greater input of causality. It is considered to be the corner stone of evidence based medicine and the backbone of high standard knowledge.² The findings are the basis for high quality scientific evidence, systematic reviews and clinical practice guides.

The place RCTs in the evidence hierarchy has given way to some erroneous conceptions of value, quality and absence of bias that have been related only to RCT in name only. In spite of its advantages, it is not without methodological issues that may compromise its internal and external values.³

Regarding internal value, the possibility of bias that pose a threat to the RCT may be a result of a flawed design, conduction, analysis, interpretation and report. Such possibilities have been classified in 4 groups: selection bias, execution bias, weathering bias and detection bias.⁴ An inadequate randomization, i.e. failure to conceal subject assigning, no double blind method or differential follow-up loss, has been proven to affect the effects of treatments.^5–8 Evidence suggests that improper subject assignment concealment and non-double-blind methods may result in overestimating the effect of the intervention on trial.⁹

The quality assessment method is a matter of controversy. The absence of a reference standard for the assessment and the uncertainty of it on the effect of the intervention have led to the creation of several scales and checklists.¹⁰ However, only 12% of the instruments has been assessed empirically.¹¹ Many of them include assessment of elements related to the report and design of the trial and not directly to the bias.⁴

In February 2008, the Cochrane Collaboration presented a tool for the assessment of bias risks designed to establish the internal validity of a clinical trial.⁴ The selection of components included (6 parameters) was based on evidence of its positive or negative associations with the effect.^7,8,12 The authors’ objective was to discern among the elements used for the design of a clinical trial and those necessary for its report. Every parameter is assessed in three categories of risk evaluation (high, low, unknown) that finally results in individual assessment of every study and for every parameter with two evaluation graphs. The trials are classified based on the original report and may also include additional documents such as the trial's protocol.¹³

The RCA journal is the official publication of the Sociedad Colombiana de Anestesiología y Reanimación (SCARE), and its purpose is to inform and update scientific knowledge of the speciality and related areas of medicine. Since its first issue in 1973, 40 volumes with 157 issues have been published. It is currently classified as A2 category in Publindex at Colciencias.

The aim of this study was to assess the risk of bias of all RCTs published in the RCA journal by using the Cochrane Collaboration “risk of bias detection” tool.

Methods

It is an observational study that included all randomized clinical trials published in the RCA journal since the publication of its first issue in 1973 up to the contents of volume 37, issue no. 1 in 2009.

A systematic search was designed for the identification of all published clinical trials which included the SCIELO database, a revision of all the RCA journal volumes and the research and location tool of the www.revcolanest.com website. Clinical trials and before-after experiments that were not performed on humans were excluded.

Once the studies were identified, they were divided into six equal groups using a computer generated random numbering table (R Development Core Team (2010)¹⁴). Each group of trials was further divided into six evaluators with previous experience and training in clinical trial assessment by the use of the Cochrane Collaboration methodology and/or systematic review publication. The original trial numbering and distribution was carried out by the main author of this study.

Each evaluator inputs every study into an electronic survey. This survey collected data corresponding to: 1. Title of the study, 2. Type of clinical trial, 3. Number of studied groups, 4. Number of authors, 5. The studied outcomes and time of study of the primary outcome, 6. Date and location of execution, 7. Number of participants and finally, the six parameters of the Cochrane Collaboration “risk of bias detection” tool.

Data analysis

A database was designed in the R¹⁴ statistics package for data analysis. The results are reported descriptively and graphically, and also include the description of the studies, the chronologic and geographic rates and the proportion of bias risk assessment for each parameter among its three categories (high risk, low risk, unknown). In addition, graphs were created for assessment of publication trends, their relation to bias risk and number of recruited subjects.

Results

The RCA journal has published 40 volumes to the present day, four issues per year for a total 157 issues (considering number 1, volume 40 in 2012). Its publication began in July 1973 (Volume 1, Issue 1) and has continued for the past 39 years.

The first clinical trial published in the RCA journal was in issue no. 10, in 1982. The search for articles began at that issue and up to issue no. 1 of volume no. 37 in 2009 and yielded a total 75 published clinical trials. The last eight issues of the RCA journal, which contain 13 new clinical trials were not included (issue no. 3 in 2010, no. 4 in 2011 and no. 1 in 2012), because data collection and analysis were carried out during this time. More details on the searching and selection processes are shown in Fig. 1.

Fig. 1.

Search and clinical trial selection processes.

Table 1 shows the general character of the studies and research areas organized by frequency and follow-up time of primary outcome. Fig. 2 shows the execution site and Fig. 3 shows the number of studies published in 3-year intervals in the History of the RCA journal.

Table 1.

General character, research areas and primary outcome follow-up time (n = 75).

Variable on trial	n (%)
Year of publication
1982–1984	3 (4)
1985–1987	1 (1)
1988–1990	7 (9)
1991–1993	5 (7)
1994–1996	10 (13)
1997–1999	7 (9)
2000–2002	11 (15)
2003–2005	13 (17)
2006–2008	15 (20)
2009–2012^a	3 (4)
Study groups in the trial
Parallel trial	68 (91)
More than two study groups	7 (9)
Trial type
Pharmacologic	68 (91)
Non pharmacologic	7 (9)
Number of authors
One	10 (13)
Two	17 (23)
Three	18 (24)
Four	16 (21)
Five	3 (4)
Six or more	11 (15)
Research areas
Chronic or acute pain management	15 (20)
Postoperative complications	9 (12)
Obstetric anesthesia	9 (12)
Local anesthesia	9 (12)
Inhalation general anesthesia	8 (11)
Intravenous general anesthesia	6 (8)
Airway management	6 (8)
Neuromuscular relaxation	5 (7)
Pediatric anesthesia	3 (4)
Intraoperative homeostasis and/or water balance	3 (4)
Education in Anesthesiology	2 (3)
Follow-up time of primary outcome
Percentile 25	30 min
Percentile 50 – median	2 h
Percentile 75	24 h
Percentile 90	55.2 h (2.3 days)
Range	30 s–2 months
Unreported	19 (25)

a

Included up to volume 37, issue no. 1, 2009. Trials not included in bias risk analysis: 8 issues 13 clinical trials.

Fig. 2.

Clinical trial execution sites of the Revista Colombiana de Anestesiología journal (national and international) (n = 75).

Fig. 3.

Number of clinical trials published in year intervals (3 years) in the history of the Revista Colombiana de Anestesiología (n = 75)*.

Classification of low risk of bias was achieved by 66% of the trials for parameter 4 (follow-up loss), 54% (selective report) and 45% in parameter 1 (randomization). The scores with highest risk of bias were those of parameter 2, related to randomization sequence concealment. 30% of all RCTs achieved four or more parameter scores for low risk of bias. Tables 2, 3 and Fig. 4 shows the frequencies and the proportions of the tool's parameters for risk of bias assessment achieved by the clinical trials published in the RCA journal.

Table 2.

Trial proportion and scores achieved in the tool's parameters for risk of bias assessment.

Risk of bias	Parameter 1	Parameter 2	Parameter 3	Parameter 4	Parameter 5	Parameter 6
	Was the randomization sequence carried out properly?	Was the randomization sequence properly concealed?	Was subject assignment properly concealed?^a	Were follow-up losses described properly?	Is the trial free of selective reporting?	Does the trial show other risks?
Low	34 (45)	21 (28)	30 (40)	50 (66)	41 (54)	24 (32)
High	17 (23)	22 (29)	19 (25)	8 (11)	2 (3)	19 (25)
Unclear	24 (32)	32 (43)	26 (35)	17 (23)	32 (43)	32 (43)

a

Assessment carried out for the primary outcome of each study (percentage of every parameter).

Table 3.

Number of parameter scores for low risk of bias in all clinical trials (n = 75) n (%).

Six parameters	7 (9)
Five parameters	7 (9)
Four parameters	9 (12)
Three parameters	12 (16)
Two parameters	17 (23)
One parameter	17 (23)
Zero parameters	6 (8)

Fig. 4.

Parameter fulfillment rate of all the assessed clinical trials (n = 75).

The relation between year of publication, number of published trials and assessment of the four main parameters of the tool are described in Fig. 5. It shows the increase in the blue area with the passing of time as evidence of sustained reduction in global risk of bias. In addition, Fig. 6 shows the rate of trials that achieve three or more low risk parameter scores in blue and the radius of the circles show the average size of recruited studies.

Fig. 5.

Assessment of the four main parameters of the tool according to year of publication in the Revista Colombiana de Anestesiología journal (n = 75).

Fig. 6.

Bubble graph comparing year of trial publication to average low bias risk parameter in Revista Colombiana de Anestesiología (n = 75)*.

Finally, Fig. 7 is a box diagram of the total distribution of subjects included in the trials according to year of publication (organized in 3-year intervals).

Fig. 7.

Number of patients recruited according to year of publication of clinical trials in the Revista Colombiana de Anestesiología journal (n = 75).

Discussion

This study shows a view of clinical trial publication in the area of Anesthesiology in Colombia. It was a research initiative aimed at documenting our scientific work and possibly to planning strategies for improvement in future trials.

We present the discussion of results in two parts: first, the general considerations of execution in the journal and then the risk of bias analysis for the published trials.

General considerations

Most of the trials were carried out in the cities of Bogotá and Medellín. This is probably due to the fact that these cities provide greater organization, collaboration, logistics, financing possibilities and conditions for clinical trial development, which are demanding because of their complexity. Several international trials have also been published in the RCA journal, although publication of foreign studies has decreased in the past years. The international presence of the RCA journal has grown due to several strategies provided by SCARE and its editorial committee. It is currently indexed in SciELO Colombia, LILACS, EBSCO, Imbiomed, Index Copernicus, Redalyc, LICOCS, Latindex and more recently, Elsevier-Doyma.¹⁵ It is in our general interest to become a journal of interest for publishing international clinical trials and indexed in Medline.

The number of clinical trials has increased over time and as of 2003 an estimated five articles are written every year. The graphic trend appears to be linear and sustained. Many authors, researchers and groups could well be influenced by methodology and analysis assessment teams. In a publication which assessed clinical trials of the Revista Española de Anestesiología, García-Alamino al.¹⁶ reported that 30% of trials were studying non-pharmacologic interventions. For the RCA journal, this rate was lower, however present. Because of the characteristics of the speciality, Anesthesiology is an area in which several non-pharmacologic interventions are used and they represent an opportunity for new challenges in the planning and design of experimental research.

Few trials carried out outcome follow-up for more than 24 h. In fact, most of them did so for two hours only. A great number of randomized clinical trials in Anesthesiology are limited to the immediate postoperative period, though many outcomes relevant for the patient (rather than the anesthesiologist or the researcher) may appear during hospital stay follow-up or late recovery phase. This consideration must be accounted for and adapt to the research question and the research scenario.¹⁷ Also, in case surrogate outcomes are employed (substitute, or intermediate measurements), which are usually easier to measure, they must be carefully selected to properly represent the clinical event of interest for the patient.¹⁸

Clinical trials are studies that demand a great amount of planning and resource availability, even if the design or the interventions are not complex. In the RCA journal, more than 50% of all trials were published by three authors or less. To our consideration, the researchers for the processes of planning, design, execution and publication are few for such an endeavor. Greater numbers of researchers are needed.

Publication bias risk analysis

Clinical trials and their systematic reviews provide the “most valid” evidence on effects of health interventions and their causal associations. However, inferences on causality extracted from these studies may be undermined by flaws in their design, execution, analysis and report. Such errors result in an inaccurate interpretation of the effects of the intervention (biases).^5,6,13 Usually, it is not possible to know how much the biases truly affect the results of a clinical trial. As a matter of fact, for readers, it is difficult to judge the adversities that a research team faces in a particular trial. For that reason, the current trend is to judge the “risk of bias”, understanding “quality” as the degree to which the design, execution, analysis and report are fit for providing a bias-free randomized clinical trial.

The tool used for bias risk assessment in this study is the same facility used for the development of systematic reviews in the Cochrane Collaboration. Few publications have used it with the intent of establishing the risk of bias of a particular journal (possibly because of its recent design).¹⁹ In spite of some of the tool's limitations mentioned in literature,^20–22 our aim was to show a global view of the development of randomized clinical trials in the RCA journal.

The analysis of tables and figures presented in this study shows that the amount of parameter scores for low risk of bias increases as time goes by. 30% of studies show four parameters or more with low risk of bias. This trend greatly represents a clear, sustained intent of improvement, formation and development of high quality research. In a future issue of this work team, we expect to include a sub-analysis of the time period between the year 2000 and the present day.

The results of this study are proof of the great interest of the RCA journal, its editorial committee and SCARE to strive forward in the improvement process and the presence of the publication.^15,23 However, this effort must also be shared by each of the authors in the journal, who take the first step with the quality of their reports.²⁴

The process of achieving a high quality randomized clinical trial with low risk of bias requires solid institutions and a health system that guarantees the users’ rights. A reform of the educational system in which professors at third degree educational institutions have resources enough to develop their research is also necessary. In Colombia, the current health system follows a poorly controlled search for profitability in medical attention, which is a limitation for clinical trial execution and all other research attempts. The system's narrow margins and the priority of profitability restrain the freedoms of thought and action necessary to achieve better and more cost-effective patient care.

Financing

Own resources.

Conflict of interest

Node declared.

Acknowledgements

To the Sociedad Colombiana de Anestesiología y Reanimación SCARE, the SCARE education department and the Clinical Epidemiology Unit at Universidad del Cauca.