Assessing the completeness of reporting of observational studies in Colombian Journal of Anesthesiology. Cross sectional study

Abstract

Introduction

The STROBE statement (Strengthening the Reporting of Observational Studies in Epidemiology), consisting of 22 points, was published in 2007 with the aim of improving the reporting of observational research.

Objective

To determine the completeness of reporting of observational studies published in the Colombian Journal of Anaesthesiology between 2000 and 2013 using STROBE.

Methods

Cross-sectional study. The units of analysis were observational studies published in Colombian Journal of Anesthesiology between 2000 (volume 28) and 2013 (volume 41). Of the randomly selected studies, 40% were included. The primary outcome was the completeness of the report of each study. Using two groups, the researchers applied the STROBE statement independently, supported by an arbitrator. Descriptive analyses, time series analyses and multiple linear regression analyses were used.

Results

Overall, 55 studies were identified and 22 (40%) were randomly chosen. The median overall completeness according to STROBE was 57%, 95% CI (48–66%). Items with higher completeness in the reports were the Title and Introduction (89% each), followed by Methods 53%, Results 50%, and Discussion 58%. There were no significant differences between before and after the publication of STROBE.

Conclusion

The completeness of reporting of observational studies in the Colombian Journal of Anaesthesiology is close to 60%, according to results worldwide. No change was found as a result of the publication of the STROBE Statement in 2007.

Resumen

Introducción

La declaración STROBE (Strenghtening the Reporting of Observational Studies in Epidemiology) se publicó en el 2007 con el objetivo de mejorar el reporte de la investigación de tipo observacional, consta de 22 puntos.

Objetivo

Determinar la completitud del reporte de estudios observacionales publicados en la Revista Colombiana de Anestesiología entre el año 2000 al 2013 mediante STROBE.

Métodos

Estudio observacional de corte transversal. La unidad de análisis fueron los estudios observacionales publicados en la Revista Colombiana de Anestesiología desde el año 2000 (volumen 28) hasta el año 2013 (volumen 41). Se incluyó un 40% del total de estudios seleccionados aleatoriamente. El desenlace primario fue la completitud del reporte de cada estudio. Mediante dos grupos, los investigadores aplicaron la declaración STROBE de forma independiente con el apoyo de un árbitro. Se utilizó análisis descriptivo, análisis de series temporales y regresión lineal múltiple.

Resultados

Se identificaron 55 estudios y se escogieron al azar 22 (40%). La mediana de completitud global de STROBE fue de 57% IC95% (48%-66%). Los ítems con mayor completitud de los reportes fueron en el título y la introducción (89% cada uno), métodos: 53%, Resultados 50%, discusión 58%. No se encontró diferencias significativas entre el periodo pre y post publicación de STROBE.

Conclusión

La completitud del reporte de los estudio observacionales de la Revista Colombiana de Anestesiología es cercana al 60%, acorde con resultados a nivel mundial. No presentó cambios con la publicación de la declaración STROBE en el 2007.

Introduction

Observational scientific research describes and analyses reality without intervening,¹ and does not establish definitive causal relationships, but it provides scientific bases regarding the health/disease process and guides future research.

Bearing in mind that in various settings there are implications that preclude intervention (such as studies on the safety of pharmacological or non-pharmacological interventions), observational studies account for the majority of publications in the area of health.² These are lower-cost studies, easier to design and perform, and useful for healthcare planning and administration.³

The STROBE statement (Strengthening the Reporting of Observational Studies in Epidemiology) was designed in 2004 in Bristol, United Kingdom, and published in 2007⁴ with the aim of improving observational research reporting. A framework comprising 22 items was developed with the goal of making observational results truly useful by means of an appropriate reporting of cohort, case–control and cross-sectional studies.⁴ The three research designs share 18 of the 22 items in the Statement, whereas the other 4 are specific for each. The items include the Title, Abstract, Introduction, Methodology, Results and Discussion sections.⁴ The STROBE statement is not a model for conducting an observational study and it does not assess the quality of a study directly. Rather, it is more a completeness guideline for authors or readers, and has been used to assess this characteristic in various medical disciplines.^5–12

The Colombian Journal of Anaesthesiology (RCA) was published for the first time in September 1973; the first few articles were narrative reviews related to the practice of anaesthesiology but the journal later became a medium for disseminating original research results. At present, it is the official communication medium of the Colombian Society of Anaesthesiology and the main medium for the dissemination of research in anaesthesiology and related areas in Colombia. In 1999, the Cochrane Collaboration found that RCA is the Colombian journal that publishes the largest number of reports on intervention studies, systematic reviews and meta-analyses.¹³

Recently, RCA adopted the guidelines of the EQUATOR network (Enhancing the QUAlity and Transparency Of health Research), which includes the STROBE statement for its editorial process as well as a firm suggestion about its use for its authors.¹⁴

The main objective of this study was to determine the completeness of reporting of observational studies published in RCA between 2000 and 2013, using the STROBE statement. Additionally, to analyse the completeness of observational study reporting before and after the publication of the STROBE statement, quantify the trend, and establish recommendations regarding complete reporting of observational research in anaesthesiology in Colombia.

Methodology

Cross-sectional observational study. The subjects for the analysis were all the observational studies published in RCA between 2000 (volume 28) and 2013 (volume 41). The RCA archive hosted in the website and the database of the tables of content and abstracts in Science Direct were initially explored.

Studies classified as “observational” were those defined as such by their authors: cohort studies, case–control studies and cross-sectional (or prevalence) studies, and they were initially classified based on the evaluation of each abstract. Studies that were excluded were those without a clear, well-defined methodological design, editorials, letters to the editor, reflection articles, narrative or systematic reviews of the literature, case reports and case series.

A total of 55 observational studies were identified. Because of the exploratory nature of the study, a random selection of 40% (22 studies) was made, and annual representation in the sample to analyse was ensured by means of proportional fixation by year. Based on reports on prior publications, the sample size was estimated using a completeness proportion of 60%, 15% accuracy and 95% CI, for n = 24.

An assessment tool developed on the basis of the STROBE statement and aimed at evaluating the reporting completeness for each study was applied. This tool and minor modifications thereof have ween widely used in the literature for similar purposes.^4–12

The selected studies were divided into 2 groups and two researchers already familiar with the research methodology, the health research designs and the STROBE initiative were assigned to each group (Group 1: JC-YA, Group 2: MB-LB). Each researcher, working separately, applied the list of items to each study in order to determine whether they were “met”, “unmet” or did “not apply”, in accordance with the specific type of design. Regular follow-up meetings were held, during which the researchers shared their results, reviewed and discussed the cases for which the assessment was inconclusive. In cases of difficulty, the rating was discussed individually with a research advisor (JAC).

The dependent variable (primary outcome) was completeness of the report of each study assessed. Additionally, other variables were explored, such as sample size, number of authors, number of professors, presence or not of individuals with Masters or PhD degrees in the research team, and the year of publication.

Statistical analysis

All statistical analyses were made using the SPSS 21.0 and R software package.¹⁵ Completeness rating for each study was the sum of all the items in the assessment. Considering that the total number of variables assessed was different for each research design (cohort, case–controls, and cross-sectional) the individual final result was weighted, and the median rating between the two reviewers and the range as a measure of scatter were estimated. The degree of agreement regarding completeness rating was analysed using the intra-class correlation coefficient (ICC). This agreement was assessed for study segments and also globally.

The general description of the results was made using descriptive techniques in accordance with the nature of each variable. Non-parametric tests were used for univariate contrasting (Mann–Whitney U Test) using report completeness as the dependent variable.

A time series analysis was made using an autoregressive integrated moving average (ARIMA) model that allows to assess trends before and after an intervention. This methodology considers multiple evaluation points before and after an intervention and the existing autocorrelation between those points.^16,17 Pre-intervention (2007: year of the STROBE initiative publication) and post-intervention periods were used. Their results are presented as the difference in the slope of the regression line before and after the intervention. Additionally, the effect of the intervention for 2009, 2012 and 2013 is also presented. Statistical significance was predefined as a p value of 0.05.

Finally, a multiple linear regression model was developed in which the degree of completeness was considered as the outcome, and the year of publication, sample size, the number of authors, the number of professors and the presence or absence of individuals with masters degrees or PhDs in the research team were considered as independent variables. The results are presented as the regression coefficient and their corresponding p values. The adjustment of this model was valued using the graphic residual analysis.

Results

Overall, 22 observational studies published in RCA between 2000 and 2013 were analysed.^18–39 One of them was reported as a two-phase study, so it was divided separately for a total of 23 (Fig. 1). Table 1 shows the general characteristics of the studies included. The median overall completeness of the items in the STROBE statement was 57%, 95%CI (48–66%). The sections for which greater completeness was documented were the Title and the Introduction (89% each). Completeness for each section of the research articles is shown in Fig. 2.

Fig. 1.

Selection of the studies included (n = 23).

Source: Authors.

Table 1.

General characteristics of the studies included (n = 23).

	Frequency (%) Mean μ ± 1 SD Median [IQR]
Year of publication
2000–2003	7 (30)
2004–2007	5 (22)
2008–2011	5 (22)
2012–2013	6 (26)
Total number of authors	μ = 3.7 ± 2
	Median = 3 [3–5]
Number of faculty	μ = 1.7 ± 1
	Median = 1 [1–3]
Study sample size	Median = 212 [65–260]
Presence of author with masters degree or PhD	18 (78)

Source: Authors.

Fig. 2.

Completeness for research article sections (n = 23).

Source: Authors.

Table 2 shows completeness results by reviewer and article sections, and the degree of agreement between reviewer pairs by group.

Table 2.

Completeness rated by each reviewer by article sections and degree of agreement between reviewer pairs by group (n = 23).

	Group 1 (n = 12)			Group 2 (n = 11)
	Reviewer 1	Reviewer 2	Agreement^*	Reviewer 1	Reviewer 2	Agreement^*
Title	95.8	95.8	1.00	77.3	81.8	0.86
Introduction	87.5	95.8	0.64	86.4	90.9	0.77
Methods	64.5	52.0	0.53	54.7	46.4	0.56
Results	70.6	33.6	0.28	54	45.2	0.56
Discussion	73.3	56.6	0.53	42.3	59.6	0.45
Total	70.8	52.8	0.49	55.9	52.2	0.59

*

Intraclass correlation coefficient.

Source: Authors.

The overall completeness trend is shown in graph form (Fig. 3) for before and after the publication of the STROBE strategy. The linear equation for completeness is shown for each of the sections as a function of the year of publication. Greater rating variability was found for the post-STROBE period. The slope of the straight line for the time period before the intervention was 0.85. The difference in the slope of the line between the before and after time periods was −0.47 (p = 0.709). Change coefficients in the slope of the line for 2009, 2021 and 2013 were, respectively, −3.2 (p = 0.698), −5.1 (p = 0.628) and −7.5 (p = 0.626).

Fig. 3.

Overall annual completeness trend of the observational studies published in RCA between 2000 and 2013 (n = 23). Note: The straight line equations in the upper part represent the proportion of increased completeness by year during the two periods. During the first period, the increase (B coefficient) was 0.8%, and during the second period, the increase was 0.4%.

Source: Authors.

The completeness of the reports that included the presence of an author with a Master's or PhD degree was higher (p = 0.03). The adjusted effect of the study variables on completeness is shown in Table 3.

Table 3.

Adjusted effect of the variables on report completeness for observational studies published in RCA between 2000 and 2013 (n = 23).

Variable	B coefficient	Standard error	p value
Year of publication	0.345	0.420	0.43
Number of authors	1.169	1.073	0.29
Number of faculty	−3.988	1.995	0.06
Sample size	0.003	0.009	0.73
Presence of author with Masters or PhD degree	10.310	4.452	0.03

Source: Authors.

Discussion

The main findings of this study are, first, that completeness of observational studies published in RCA is 57%, and the Title and Introduction are the sections with the greatest completeness, while Methodology and Results are the ones with the greatest deficiencies. Second, during the 13-year study period there is a very slow increase in completeness, and there is no evidence that the publication of the STROBE statement in 2007 had any effect on completeness for RCA.

The completeness of observational studies is similar to that reported by other authors. Several research studies have documented completeness ranges of 30–85%.^{5,6,9,10,12,40} In 2011, Poorolajal et al., analysed observational studies in the New England Journal of Medicine, Journal of the American Medical Association, Lancet, British Medical Journal, Archives of Internal Medicine, and the Canadian Medical Association Journal, and found an overall compliance of 69% (95% CI 59–79%).¹⁰ In 2013, Bastuji-Garin et al., published a similar study in which they analysed observational studies of four European dermatology journals. The mean rating using the STROBE tool was 57% (range 18–98%).⁹ Percentages vary in specific areas of knowledge, from satisfactory quality of reporting, like in the study by Papathanasiou et al. in oncology which included publications between 2008 and 2009, where completeness reported ranged between 70 and 90%,⁷ all the way to low proportions, such as those reported by Sorensen et al., where compliance in observational studies on hand surgery published between 2005 and 2011 was 38%, with a range between 10 and 54%.⁶ This might be influenced by the range of time during which the studies were developed in relation to the time of publication of the statement, and the awareness of scientists and authors regarding the importance of completeness when it comes to disseminating information. Despite having analysed 40% of the studies, the accuracy of our completeness estimation had a 95% CI of 48–66%, leaving an impression of the need for future improvement.

Authors like Fung and Langan et al., found that the items with the highest percentage of compliance were the Title, the Introduction and the Abstract. Of the publications analysed by Fung, 100% met the criteria established by STROBE for the Introduction, 86% and 93% for the Title and the Abstract, respectively. Compliance found by Langan et al. was 97% for the Introduction, 87% for the Title and 93% for the Abstract.^8,41 Consistent with the literature mentioned above, for RCA publications we found an 89% compliance with the STROBE guidelines for the Title and the Abstract.

The sections with the lowest completeness ratings were Methodology and Results, and this is consistent with results from similar studies. Galera et al., found that none of the articles reviewed were in compliance with the items recommended for Methods and Results.¹² In the work by Fung, reporting completeness for the section on Materials and Methods varied, with item ratings ranging between 7% and 97%.⁸ Low completeness for these sections may be related with their degree of complexity. For example, the methodology section requires detailed knowledge of epidemiology and biostatistics, which is not always widely available or accessible. It is worth noting that the section on Methods and Results had the lowest scores for domain agreement between reviewers (ICC of less than a 0.5). This fact reinforces the hypothesis that these are sections of a manuscript that require prior training and sound knowledge of research methodology both for writing as well as for analytical purposes.

During the 13 years selected for this assessment, there was evidence of a slight trend towards improved report completeness. This trend is quite slow (0.4 a 0.8% per year). In their study, Bastuji-Garin et al. show an increase during the study period (2004–2010) of close to 2% per year.⁹ The results of this analysis must be taken cautiously because of the great variability found during the post-STROBE period in the overall rating for certain studies. Additional studies regarding RCA have found that, over time, the risk of bias of the reported studies has diminished.⁴²

No evidence was found in this study in the sense that the publication of STROBE had any significant impact on the completeness of the observational research published in RCA. This fact has been replicated by other authors.⁸ Despite the publication of the strategy in 2007 and of its wide dissemination in the scientific media, many publications and journals were late in adopting it. RCA adopted the guidelines of the EQUATOR network, including STROBE, in 2013.¹⁴ With the use of other statements like CONSORT there has been a consistent increase in the completeness of published reports.⁴³ Fortunately, the scientific community in general is increasingly attaching greater importance to research completeness as a strategy to improve reproducibility and increase its validity.^44–46

The effect of some variables on the completeness of study reports was almost negligible. Only the presence of researchers with Masters and/or PhD degrees increased report completeness significantly (by approximately 10%). Although this is not a statistically significant increase and requires a critical analysis, it is plausible. Moreover, the size of the study sample may have lowered the power of the study for detecting actual differences (type II error).

As far as we know, this is the first study to analyse the completeness of observational research reports in anaesthesiology and other areas in Colombia. Noteworthy among the strengths of the study are the thorough search of observational studies in RCA, the analysis of completeness performed by 4 independent reviewers, and the participation of an arbitrator to solve discrepancies. This resulted in a lower possibility of selection and information bias. The main limitation was failure to study the entire population of observational studies, which undermined the confidence of our overall completeness estimate. However, our objective was to document its existence in order to promote improvement. Below are the main recommendations of this study.

Ethical disclosures

Protection of human and animal subjects

The authors declare that no experiments were performed on humans or animals for this study.

Confidentiality of data

The authors declare that no patient data appear in this article.

Right to privacy and informed consent

The authors declare that no patient data appear in this article.

Funding

None.

Conflicts of interest

None declared.