To describe the trends of research design in publications from high-impact medical journals.
MethodsA cross-sectional, descriptive study was conducted by searching the 2011 electronic publications of the journals: New England Journal of Medicine, Journal of the American Medical Association, The Lancet, British Medical Journal, and Annals of Internal Medicine. Studies were classified as primary and secondary. The journal impact factor was taken from the Journal Citation Report website. Descriptive statistics were used to analyze and interpret the data.
ResultsWe analyzed 1130 publications: 804 primary and 326 secondary studies, which represented 71.2% and 28.8% of the total publications, respectively. Among the primary studies, randomized clinical trials (30.4%) were the most prevalent, followed by cohort studies (21.9%) and case reports (9.0%).
ConclusionsThese findings can have implications in Evidence-Based Medicine programs. Literature review should focus on reviewing secondary articles first, then experimental studies and finally, observational studies.
In the area of continuous medical education, in order to achieve the gold standard in medical attention, growth and current information, frequent review of medical literature is necessary. In modern medicine, healthcare excellence is combined with scientific rigor in the practice of evidence-based medicine (EBM). This new paradigm in medical education integrates the use of the best clinical evidence and experience in the diagnostic and therapeutic decision-making process. Proposed in 1992,1 its implementation has been replacing the authoritarian management and the purely heuristic in medicine. Using competences such as search strategies, critical reading and the application of evidence to the context of the patient2,3 to create and base an intervention. Some of its objectives are: to promote critical thinking, to promote continuous learning, to reduce the impact of medical error and improve the patient's prognosis. Nevertheless, despite having acceptance among some medical circles; critics point out the difficulty of integrating EBM into clinical practice.4–6 They question its epistemological value7,8 and comment regarding the resistance of its implementation in some health centers and universities.9
Regarding medical education, the integration of a dynamic learning model must deliver results and content. An adequate curriculum should categorize content according to its level of complexity and difficulty. In the evidence-based medicine scenario, the competences must prepare doctors to evaluate and integrate the best evidence in order to generate an answer to a clinical problem. But, taking this into consideration, what does it take to answer a clinical question? In the process of critical reading, a study can apply to clinical context if it complies with internal and external validity. Therefore, the search of studies with a high level of evidence is the first step. This represents a fact in medical journals with a high impact factor. First, in the review-by-pairs process, reviewers included in the editorial committee publish in high impact journals.10 Concerning the researcher's participation, they recruit a higher number of patients, evaluate major results and analyze subgroups.11 With the selection of publications, the “method” section is evaluated on the integrity of the statistical analysis,12 while in “results”, the inclusion of a confidence interval13 and clinical significance range is common.14 Despite all of the above, research design may offer a replicable example of information with high clinical value. Thus, the objective of the present study is to examine the publication tendencies in the different research designs in high-impact medical journals.
Materials and methodsA transversal descriptive study was conducted. We evaluated issues of the following journals published during 2011: The New England Journal of Medicine, Journal of the American Medical Association, Annals of Internal Medicine, The Lancet and BMJ. The impact factor was obtained from the Journals Citation Report.15 These publications were then classified as primary (originals) and secondary (revisions) studies according to their focus. The main studies section was sub-divided according to the type of study and experimental (randomized clinical trials) and observational designs (prevalence, control case and cohort). The secondary studies section included: narrative review and systematic review, with and without meta-analysis, excluding genomic studies of this category.
Descriptive statistics were utilized for better data management and interpretation.
ResultsOut of the 1130 publications analyzed in 2011, the journals published an average of 3 main studies and 1 secondary study per issue (results shown in Table 1). 337 randomized clinical trials (30.4%) were published in the main studies category. On the other hand, observational studies 243 (21.9%), included 102 cohorts (9.02%), 59 case reports (5.2%) and 52 (4.6). Regarding the secondary studies, 226 publications were classified as narrative reviews (20%), 67 as systemic reviews (5.9%) and 33 as systemic reviews with meta-analysis (2.9%).
Distribution of publications according to study type in 2011.
| CR | PR | CC | Co | CCT | NR | SR | Me | Or | Sc | Ob | Ex | n | FI | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| JAMA | 3 (1.5) | 10 (5) | 32 (15.8) | 81 (40.1) | 42 (20.8) | 14 (6.9) | 2 (1) | 18 (8.9) | 168 (83.2) | 34 (16.8) | 126 (62.4) | 42 (20.8) | 202 | 30.02 |
| Annals | 4 (3.7) | 15 (14) | 2 (1.9) | 34 (31.8) | 20 (18.7) | 6 (5.6) | 18 (16.8) | 8 (7.5) | 75 (70.1) | 32 (29.9) | 55 (51.4) | 20 (18.7) | 107 | 16.73 |
| Lancet | 53 (15.7) | 5 (1.5) | 4 (1.2) | 46 (13.6) | 100 (29.7) | 120 (35.6) | 3 (0.9) | 6 (1.8) | 208 (61.7) | 129 (38.3) | 108 (32) | 100 (29.7) | 337 | 38 |
| NEJM | 42 (19.6) | 5 (2.3) | 6 (2.8) | 17 (7.9) | 110 (51.4) | 33 (15.4) | 1 (0.5) | 0 (0) | 180 (84.1) | 34 (15.9) | 70 (32.7) | 110 (51.4) | 214 | 53.29 |
| BMJ | 0 (0) | 17 (6.3) | 15 (5.6) | 69 (25.6) | 72 (26.7) | 53 (19.6) | 43 (15.9) | 1 (0.4) | 173 (64.1) | 97 (35.9) | 101 (37.4) | 72 (26.7) | 270 | 14.09 |
| 102 (9) | 52 (4.6) | 59 (5.2) | 247 (21.9) | 344 (30.4) | 226 (20) | 67 (5.9) | 33 (2.9) | 804 (71.2) | 326 (28.8) | 460 (40.7) | 344 (30.4) | 1130 |
Data are shown as n (%); CR, case report; PR, prevalence report; CC, case–control; Co, cohort; CCT, controlled clinical trial; NR, narrative revision; RS, systematic revision; Me, meta-analysis; Or, original publication; Sc, secondary publication; IF, impact factor; Ob, observational study; Ex, experimental study.
The New England Journal of Medicine published 241 articles, divided as follows: 0 (0%) meta-analysis, 1 (0.5%) systematic review, 33 (15.55%) narrative reviews, 110 (51.4%) clinical essays, 17 (7.9%) cohorts, 6 (2.8%) control cases, 5 (2.3%) prevalence and 42 (19.6%) case reports.
The Lancet published 337 articles, divided as follows: 6 (1.8%) meta-analysis, 3 (0.9%) systematic reviews, 120 (35.6%) narrative reviews, 110 (29.7%) randomized clinical trials, 46 (13.6%) cohorts, 4 (1.2%) control cases, 5 (1.2%) prevalence and 53 (15.7%) case reports.
The Journal of the American Medical Association, for its part, published 202 studies, divided as follows: 18 (8.9%) meta-analysis, 2 (1%) systematic reviews, 14 (6.9%) narrative reviews, 42 (20.8%) randomized clinical trials, 81 (40.1%) cohorts, 32 (15.8%) control cases, 10 (5%) prevalence and 3 (1.5%) case reports.
BMJ included 270 publications, divided as follows: 1 (0.4%) meta-analysis, 43 (15.9%) systematic reviews, 53 (19.6%) narrative reviews, 72 (26.7%) randomized clinical trials, 69 (25.6%) cohorts, 15 (5.6%) control cases, 17 (6.3%) prevalence and 0 (0%) case reports.
Last but not least, Annals of Internal Medicine published 107 studies, divided as follows: 8 (7.5%) meta-analysis, 18 (16.8%) systematic reviews, 6 (5.6%) narrative reviews, 20 (18.7%) randomized clinical trials, 34 (31.8%) cohorts, 2 (1.9%) control cases, 15 (14%) prevalence and 4 (3.7%) case reports.
Regarding original studies, NEJM published 180 (84.1%), The Lancet 208 (61.7%), JAMA 168 (83.2%), BMJ 173 (64.1%) and Annals of Internal Medicine 75 (70.1%). The tendencies for secondary studies were NEJM 34 (15.9%), The Lancet 129 (38.3%), JAMA 34 (16.8%), BMJ 97 (35.9%) and Annals of Internal Medicine 32 (39.9%).
DiscussionIn our frequency distribution analysis, publication distribution was linked to evidence levels. Randomized clinical trials were the most represented studies, followed by cohorts. Our results suggest that medical journals with a high-impact factor publish studies with a high evidence level (Fig. 1).
A relevant finding significantly documented in The New England Journal of Medicine and The Lancet was the high frequency with which case reports were published, representing up to 15% of the total of their publications. Another major finding is the tendencies the journals have according to their geographic location. European journals (The Lancet and BMJ) published secondary studies to a greater extent (Fig. 2). These findings raise a question: What implications can this have? In order to answer this question correctly, the limitations and advantages of each design were evaluated.
Experimental studies represent a high degree of evidence since they allow a direct determination of the causal connection of two phenomena. Another significant point is its ability to reduce the amount of systematic errors.16 With randomization, an equilibrium between the characteristics of the compared groups is established. Blinding, concealment and intention-to-treat analysis allow the reduction of skewing in any type of randomized clinical trial. Unfortunately, one of the biggest disadvantages of RCT is in the context of clinical practice. Paradoxically, internal factors such as the strict inclusion criteria, evaluation of efficiency and short-term follow-up17 limit the study's ability to establish a general conclusion. The fact that significant results are obtained does not necessarily mean that they can be applied in a “real life” scenario. In other words, the results at that moment only apply to the population being studied. Also external factors of the study like high costs, ethical considerations18 and low financing in the case of rare diseases are also an important issue. Consequently, the difficulty in clinical trials relies more importantly on external validation.
Observational studies represent the lowest level in the evidence hierarchy. They are useful in clinical scenarios like in evaluating disparity and load and determining risk factors which contribute to the development of a disease.19 In medical history, cohorts play an important role in documenting associations, like in the Framingham study, or the conclusive data of the effectiveness of insulin analogs for the treatment of patients with type 1 or 2 diabetes.20 Additionally, cohort studies help determine safety profiles of medications. For example, the SCOUT study assessed the effectiveness of sibutramine in 10,744 patients with obesity and high cardiovascular risk for a period of 6 years. Compared to placebos, the sibutramine group showed a noticeable increase in mortality rates and cardiovascular events despite the significant weight loss.21 This influenced its recall by the FDA in 2009.
In contrast with experimental studies, studies without intervention represent a clearer, uncontrolled and unadjusted model of the disease. Nevertheless, internal validation is an important issue. The highest difficulty in observational studies lies in interpretation, since there are some factors like bias in selection, confusion factors or recall bias and thus the results may lead to different conclusions.
According to the information obtained, it is possible to conclude that there are different problems in research within primary studies. European and American magazines have different criteria concerning their publication's guidelines; in other words, there is no standard in the level of evidence regarding article revision. Perhaps a major implication in relation to the impact factor lies more in the context of transnational medicine. That is to say, The New England Journal of Medicine published mostly randomized clinical trials, studies frequently cited and with a higher impact in medical practice. Hence, a dichotomy is formed between the management of concepts: validity versus generalization. Observational studies are not as valid; however they include concepts which are more applicable in populations. On the contrary, experimental studies have a great epistemological value, but only represent the set of the population being studied.
Secondary studies represent the highest level of evidence. They offer a summary of the investigation question. In the context of evidence-based medicine, both validity and generalization are important. The findings in these studies have a greater approximation to the Bradford–Hill criteria; nevertheless, the limitation in secondary studies lies in the presence of publication bias.
One of the strengths of this study was the analysis of a representative sample of the publications in different journals in the yearly period. A similar trend was reported in a study conducted in 2003, but with different objectives22 and a much smaller sample. A limitation of the study was the classification of evidence according to the study design, without evaluating the existence of discrepancies in the methodology, like the presence of bias or the weight of confusion factors.
In conclusion, the tendency of publications or journals with a high impact factor is oriented to a greater extent to the publication of primary and observational studies; however, European journals such as The Lancet and BMJ publish a good amount of secondary studies. Future studies are necessary to determine not only the validity, but also the impact of an article. The amount of quotations correlated with the publication of studies with a high level of evidence.
Conflict of interestThe authors have no conflicts of interest to declare.
FundingNo financial support was provided.