Surgery as a Scientific Discipline and the IDEAL Collaboration

Surgery has achieved tremendous things over the last century. Dramatic advances have included the development of cardiac surgery, transplant surgery and minimally invasive surgery, whilst improved peri-operative care has dramatically improved the outlook for patients.

Few of these advances, however, were based on randomised controlled trials (RCTs), and the number, size and quality of RCTs of surgical techniques remain much lower than for most medical disciplines. As this surgery has been severely criticised as unscientific, and surgeons have responded by claiming that there are special difficulties in performing such trials for surgical techniques. In 2009, the Ballioll Collaboration^1–3 tried to analyse the problems of performing scientifically valid studies of surgery, and to propose solutions to them. This led to the development of the IDEAL Framework and Recommendations.

Most surgeons agree that RCTs are the most reliable method for comparing treatments, but most would also agree that there are difficulties in applying them to surgical operations. Many of our objections, however, when closely examined, turn out to be ill-founded. For example, surgeons often say that urgent life threatening illness or rare conditions prevent them from randomising patients −but RCTs for cardiac arrest resuscitation⁴ and for childhood leukaemia⁵ have both been done successfully. Of course surgeons cannot be blinded to the operation performed −but the surgeon should not evaluate his own results, and it is often possible to blind the person who does. Again, it is common for surgeons to claim that the benefit of their operations is so obvious that a trial would not only be unnecessary but unethical (the “RCT of parachutes” argument⁶). Unfortunately we surgeons are a self-selected group of very confident people, to whom things often appear “obvious” even when they later turn out to be wrong.⁷

However there are some genuine reasons why RCTs may sometimes be difficult, impossible, or simply not the right tool for evaluating surgery. First, surgical innovations are performed by an operator, whose skill is critical to success, and whose professional identity is closely tied to the results of her/his operations. Each patient, surgeon and operation is slightly different, and it is therefore more difficult to produce a standard definition of an operation than of a medicine. It is also difficult for a surgeon who has perfected one technique to go back to one he abandoned. Secondly, surgery always implies the risk of adverse outcomes, which are usually difficult or impossible to reverse. It therefore stimulates strong anxieties and preferences in both patients and surgeons, based mainly on the individual's attitude to risk. These two fundamental properties of surgery present difficulties for evaluation in different ways during the life cycle of a new operation. The IDEAL Framework describes the stages through which new surgical procedures develop (Table 1). The IDEAL Recommendations are suggestions for the types of studies, which should be conducted at each stage of the Framework (Table 2). The Framework does not reject RCTs, but it does recognise that there is often a lot of preliminary work to do before an RCT can occur.

Table 1.

Defining Characteristics of IDEAL Framework Phases.

Phase 1: Idea	Phase 2a: Development	Phase 2b: Exploration	Phase 3: Assessment	Phase 4: Long term monitoring
Initial report	“Tinkering” (rapid iterative modification of technique and indications)	Technique now more stable	Gaining wide acceptance	Monitoring late and rare problems, changes in use
Innovation may be planned, accidental or forced	Small experience from one centre	Replication by others	Considered as possible replacement for current treatment
Focus on explanation and description	Focus on technical details and feasibility	Focus on adverse effects and potential benefits	Comparison against current best practice
		Learning curves important
		Definition and quality parameters developed

Table 2.

Key Recommendations for Research Design at Each IDEAL Phase.

Idea: Professional innovation database	Development: Prospective development studies	Exploration: Phase IIS study	Assessment: Surgical RCT	Long term monitoring: Prospective registries
Compulsory reporting of all new innovations	Detailed description of selection criteria	To evaluate technique prospectively and cooperatively	RCT – question agreed in Phase IIS	Should monitor indications as well as outcomes
Confidential entry allowed to encourage reporting of failed innovations (similar to CHRP system)	Detailed technical description	To develop a consensus over definition of the procedure, quality standards and indications	Use power calculations from Phase IIS	SPC used for quality control (Shewart charts, CUSUM, VLAD)
Hospital or institution to be informed separately as a professional duty	Prospective account of all cases consecutively, including those not treated with new technique/device	To gather data for power calculations	Use learning curve data to decide entry points for clinicians
	Clear standardised definitions of outcomes reported	To evaluate and monitor learning curves	Use Phase IIS consensus to define operation, quality control AND outcome measures
	Description of ALL modifications, and when they were made during the series	To achieve consensus on the trial questions	Use modified RCTs or recognised alternative if RCT not feasible:
	Registration of PROTOCOL before study starts	To develop a multi-centre randomised trial (RCT)	Feasibility RCT
			Expertise-based RCT
	Use of Statistical Process Control (SPC) methods to evaluate progress		Cohort multiple RCT
			Step-wedge design
			Controlled-interrupted time series

The first stage of the Framework (Idea) is the “First-in-man” study. Where the new procedure is planned, a protocol should be published in advance, but sometimes new techniques are developed in emergencies or even by accident. The key IDEAL Recommendation is simply that all potential “first-in-man” studies are registered in a database searchable by all health professionals. This will allow others to learn from success and prevent the repetition of failed or harmful procedures. To avoid the risk of litigation when innovations fail, confidential reporting should be allowed, following the good examples available in transport and energy production industries.⁸

In the second stage of the Framework, Development, the key feature is that the procedure is still being rapidly adapted and changed as the investigators learn from experience.⁹ Instead of the traditional case series, IDEAL recommends Prospective Development Studies at this stage. These are prospective cohort studies which report the outcome for every patient referred sequentially, whether or not they received the treatment. The reasons for rejecting patients and for changing the technique are reported as well as the point in the sequence when this occurred. This allows colleagues to understand how and why the technique has evolved, and avoids repeating errors.

By the third stage, Exploration, techniques are relatively stable, although there may be several alternative versions: they are being practised by several centres, and the number of cases performed is rising rapidly. This is the stage at which controversy has often prevented surgeons from performing an RCT. If surgeons are not agreed about the details of the procedure or the indications, and many are still on their learning curve, getting consensus on a patient group and a question for an RCT may be impossible. The IDEAL Recommendation is therefore a large prospective collaborative cohort study.¹⁰ Surgeons enter patient data using the indications and techniques they prefer, and this is all analysed and discussed after 1–2 years. This allows controversies to be resolved using data, and may also allow surgeons to demonstrate completion of their learning curve. This study type is called “Phase IIS” because of it's similarity to Phase II oncology studies. The aim is to help surgeons reach consensus on a question for an RCT, by reducing uncertainty about which treatment is better in which situation. The Phase IIS study is therefore often referred to as a “bridge” from small development studies to an RCT.

The last two stages of the IDEAL Framework are definitive comparison with current best treatment, preferably in an RCT, and long term surveillance, preferably using a registry.¹¹ Getting surgeons and patients to agree to be randomised remain challenging, but there are several ways to make this easier. Patients can be followed up in parallel cohort groups if they refuse randomisation. Consent and randomisation can be done by trained nurses, instead of surgeons, who find it very hard to hide their personal biases. Or patients can be randomised not to particular operations but to particular surgeons, each of which then does their favoured operation (expertise based randomisation). Blinding of the observers who measure outcome is often possible.

Registries need to be carefully thought out by the professional community. They should collect a very minimal set of important data, but should do so very reliably. It is best (though often difficult) if all patients with a condition are included regardless of treatment: more commonly registries are based on a particular treatment, such as hip replacement. In such registries it is important to have some information about the numbers and types of patients who were not selected for treatment.

These suggestions for improving surgical research are now being adopted by various groups, who are trying out the new proposals for study design and reporting. This process will lead to further development, changes and improvements to the Recommendations. Regulatory authorities have also shown interest in using the IDEAL framework to devise better programmes for evaluating medical devices. The IDEAL Collaboration (http://www.ideal-collaboration.net/), an international group of surgeons and methodologists interested in promoting these ideas, are advocating changes to current practice, including the extinction of the case series, which should be replaced by Prospective Development Studies. We look forward to a future in which surgical research is conducted using tools appropriate to the job, including but not exclusively RCTs, and as a result surgery is more evidence based, safer, and more effective.