Systematic reviews and meta-analyses in surgery

Introduction

Systematic reviews (SR) are a type of research that answers clearly formulated clinical questions of interest through a systematic and explicit process to search for and select all potentially relevant primary studies and to assess, analyse and interpret them, using rigorous methods to limit bias and random error¹. This methodological note presents the main methodological aspects for developing and interpreting a SR, based on the formulations of Cochrane and other relevant guidelines, with emphasis on the specificities of SRs in surgery^2,3.

Protocol drafting

Good quality SRs are based on previous protocols, prospectively registered to avoid redundancies and avoid biases in the reports, which guarantee transparency and rigour during the development and a higher methodological quality of the resulting SR. resultante^4,5. The main international registry of protocols for systematic reviews is PROSPERO (http://www.crd.york.ac.uk/prospero/).

The protocol of an SR should present the planned methods for each of the 6 steps common to all SRs, which we will review below.

Stages in the development of a systematic review

Definition of the clinical question of interest

The first step is to correctly formulate the clinical question of interest that motivates the SR. SRs for interventions aim to answer therapeutic questions that can be structured, in a general way, following the key elements of the PICO model⁶. SRs that answer other types of questions, such as prognostic questions, follow other structures, with elements specific to each type of question⁶.

The selection criteria for the primary studies in the review are derived from the elements of the question of interest:

• 1.

  Type of study participants. It is important that the criteria defining the disease or condition of interest are well detailed so that they can be verified in the studies identified by the search. Patient presentation (previous surgeries and treatments) should be considered.

• 2.

  Type of intervention assessed in the study. The intervention should also be well defined, although it is important to anticipate that there may be acceptable variations between studies in terms of the specific treatment modality applied. In SRs of surgery, a particular effort should be made to describe not only the surgical technique following standardised guidelines, e.g., TIDieR and Blencowe guidelines^7,8, but also the characteristics of its implementation, such as team profile, training and experience.

• 3.

  Type of comparison used in the study. Sometimes, depending on the context, the appropriate control for comparison will be a simulated intervention, not an active intervention, another modality of surgery, or a non-surgical intervention.

• 4.

  Types of outcomes of interest assessed in the study. It is important that the authors of the review establish a priori a list of outcomes, ordered by their clinical relevance, referring to efficacy and safety, and also patient-reported outcome.

• 5.

  The type of study(ies), referring to the eligible study design for SR.

Identification and selection of the relevant studies

SRs are characterised by comprehensive literature searches that have, at a minimum, an electronic database search component. Generally speaking, it may be sufficient to search MEDLINE and the Cochrane Central Register of Controlled Trials (CENTRAL) to obtain an efficient search of an SR of a surgical intervention⁹. It should be noted that EMBASE does not contribute substantially to reviews of surgical interventions, but may be valuable in reviews that also consider a pharmacological intervention. Furthermore, these searches should be complemented by searches of other databases and alternative sources, such as prospective study registries (e.g., Clinicaltrials.gov).

Evaluation of the risk of bias of included studies

Current tools for assessing risk of bias are organised on the basis of classical epidemiological biases related to each type of research question. In the case of interventional SRs, the preferred study design to be included is randomised clinical trials. For this design, the most common risk of bias scale is Cochrane’s Risk of Bias, which assesses the domains of randomisation of participants, blinding of interventions (to patients, research staff and evaluators), loss following and publication bias².

In surgery studies, the actual feasibility of masking the intervention should be considered, as well as the risk of bias related to the dependence of the intervention on the personnel delivering the intervention (training, characteristics, criteria for allocation of professionals to intervention groups) and on the commercial funding of the studies.

Incomplete description of interventions has been reported in primary surgery studies, which may limit the assessment of bias sources^10,11.

Intervention SRs involving observational studies, as well as those answering non-intervention questions, should apply other specific risk of bias assessment tools¹².

Summary of the evidence and presentation of outcomes

The type of synthesis of results that will be possible in the SR and its degree of precision will depend, among other things, on the amount of information available in the primary studies and the homogeneity between them. The quantitative synthesis of the results is carried out using the meta-analysis technique, the methodology of which is widely described².

It should be emphasised that a necessary step prior to any meta-analysis is the evaluation of the clinical and statistical heterogeneity existing in the set of available studies, which will inform us whether it is reasonable to make a quantitative synthesis of their findings, which statistical model of meta-analysis should be applied and whether additional research is required into the causes of the heterogeneity detected, for example, by means of subgroup analysis and sensitivity or meta-regression. In the field of surgical studies, it is common to observe great clinical variability (either in the techniques evaluated, in the contexts considered in the studies or in the definition and measurement of the results), which discourages quantitative synthesis with meta-analysis. In this situation, the authors have the SWiM guide to formulate and describe the process of narrative synthesis of evidence¹³.

Interpretation of outcomes and evaluation of the quality of the evidence

The quality (or also confidence or certainty) of the evidence in an SR is the degree of confidence we can have that the effect estimate observed in the meta-analysis is close to the true value of the effect. The certainty of the evidence is best assessed using the GRADE system, in which the certainty in the estimates obtained for each of the SR outcomes of interest is classified as high, moderate, low or very low. Factors that will influence the level of certainty assigned are the design of available studies, limitations in study design or execution, inconsistency between estimates (heterogeneity), indirect evidence, imprecision in estimates, and publication bias^14,15.

Systematic review report

Several guidelines are available to guide transparent and complete reporting of SRs, such as the PRISMA statement for reporting SRs and meta-analysess¹⁶ and its extensions, generated by the EQUATOR initiative (https://www.equator-network.org/). In addition, the evidence-based TIDieR-SR guideline for the reporting of SR interventions, evaluated in a cohort of surgery reviews, has been proposed¹⁷.