Essential elements to elaborate a study with the (e)Delphi method

Introduction

What method enables me to establish a competency framework for specialist critical care nurses?¹ How can I exercise a protocol, based on the best evidence available, for the care of a person who suffers from a pilonidal sinus wound healing by secondary intent?² What is the best method for compiling a basic minimum data set for the prevention, diagnosis and treatment of venous conditions of the lower limb in the adult population?³

Undoubtedly on some occasions we have asked ourselves similar questions, related to our field of expertise. Sometimes it is difficult to find sets of recommendations for a situation, setting, specific problem, since evidence is insufficient to encompass the whole process or simply, we wish to reach an agreement about which competences should have a professional figure. One of the options available are the expert group consensus methods, understood to be a systematic medium of measurement and development of consensus. Its aim is to establish to what extent experts agree on a specific subject.⁴

The two most commonly used methods of consensus are the nominal group technique (NGT) and the Delphi method,^4,5 followed by the RAND/UCLA method, which is a cross between the previous two.⁶

The NGT takes place through group interaction. It is a highly structured, face-to-face method, where participants express their opinion and the other members listen to their opinions,⁵ allowing for the possibility to debate issues where there is little consensus and to generate a more robust idea.⁶

The Delphi method also provides the opinion of a group of experts, called a panel, on a subject matter in a structured format,^5,7 although interaction between the different members is through a questionnaire.⁵ Due to Internet use application, several authors have suggested calling this e-Delphi, since pen and paper have been replaced by the benefits offered through the online platform that helps to organise and provide communication between the researchers and the experts.⁸ Both terms are used in the literature, however. The advantage of this method is it allows for a larger number of people to participate, eliminating geographical distance whilst maintaining anonymity of participants, with avoidance of any influence from the response of any panel member.^6,9 Furthermore, it is relatively cheap, helps to organize data and its importation for database analysis.⁹ There are also disadvantages. For example, there is no certainty whether the people who respond to the questionnaires are experts. To resolve this, a single link may be sent which only one expert can use.^8,9 Another disadvantage is the use of electronic mail firewalls, an aspect that may prevent the invitation to participate in the study from arriving.

Essential elements of the Delphi method

Delphi is a qualitative technique, although there are authors who defend that it is a mix and others who say it is quantiative¹⁰ in its final phase.

The first step is to organize a group of researchers who are responsible for creating the whole process and for carrying out follow-up. One person should be appointed to act as the visible head of the group. They will direct communication with the panel of experts, resolve any possible incidences and drive the whole process. The essential elements of the Delphi method, as may be observed in Fig. 1 may be summarized as six^6,11: identification of the research problem, construction of the questionnaire, selection of the panel of experts, provision of information to the panel members to help with their decision, questionnaire distribution, data analysis, provision of feedback to the panel and report on findings.

Figure 1.

Summary of the Delphi method’s essential elements.

Identification of the research problem

As with any investigation, the first step is to be clear on what the research objective is.^4,6,11 We could start with any of the questions with which we introduced this article. For example, with the COVID-19 pandemic where critical care nursing is in demand, we could ask: what are the specific competences that must be acquired to be considered specialist critical care nurses¹ in a certain country?

Construction of the draft questionnaire

To construct the draft questionnaire one frequently used option is the creation of a bibliographic review^2,3 which includes aspects of grey literature.^6,11 The information obtained allows us to create items: the number and ideal nature of them must be agreed upon among the researcher group, in keeping with the initial objective. One aspect which will help us to organise the items, if there are many, is to group them into categories. It is important to record the whole step-by-step process of questionnaire creation as this is the basis of the study.

Some studies also obtain the draft questionnaire from information obtained from focus groups,¹ made up of the panel of experts they have invited¹¹ or of the same experts who are in charge of the study.¹² For example, in the study by Raurell-Torredà et al.¹² the researchers themselves, experts in nursing simulation and taxonomy, created the initial group of nursing interventions classification (NIC) to be used in the design of high quality simulation clinical case studies for the training of nursing students in non technical skills.

Once the items have been created, we must determine their grading using a Likert type scale⁵; those most used are item 3, 5, 7 and 9, and the expert has to indicate their level of agreement with the statement from “totally disagree” to “totally agree” passing through intermediate terms, depending on the number of response options on the scale.

One example of an evaluation scale of 5 items was found in the study by Zhang et al.¹ on the creation of a competence framework for the critical care specialty in China. In the complex decision domain, the expert should indicate the degree of agreement in three competences, giving them a score on the scale from 1 = totally disagree to 5 = totally agree (Table 1).

Table 1.

Example of the Likert type assessment scale from the complex decision domain from the study by Zhang et al.¹

Complex decisions	Totallydisagree	Disagree	Neither agree nor disagree	Agree	Totally agree
Evidence-based decision
Clinical judgement
Integrated reasoning

An open-ended response option should be present in the questionnaire to offer the respondent the opportunity of adding new items or for the experts to make a comment on any aspect considered appropriate or relevant.

The decision to determine what percentage we consider consensus to be reached should be established in this phase. Despite the fact there is no existing definition of consensus, depending on the type of study conducted, justification of why this decision is taken must be made.¹¹ Some authors consider consensus in one item with 70% of experts responding that they “agree” or “very much agree” with that item.⁴

It is necessary to consider the number of items contained in the questionnaire and their complexity, with a pilot study⁹ with a small number of experts, to assess aspects such as comprehension or duration time.⁴ A questionnaire which has not been tested may affect expert participation, especially in successive rounds. There are mechanisms for facilitating the experience of response to the questionnaires: creating definition with simple language, grouping items into categories and arranging them in alphabetical order whenever possible.¹³

Panel of expert selection

In consensual methods the experts are those people who have knowledge and experience on the study subject.^5,11 A good descriptions of the defining characteristics of whom we consider an expert is required, similar to the inclusion criteria of any study.

Following on with the example of the study by Zhang et al.,¹ to participate in the panel the expert must have: a) a graduate or higher degree; b) a deputy director qualifications or higher; c) over 10 years of professional experienced and working in the field of critical care with a solid theoretical base; d) the ability to give comprehensive opinions and make suggestions and e) be highly motivated and easily able to participate in the study.

There is no particular minimum number of experts for a panel, but results will be more stable the higher the number of experts.¹¹ A minimum number of 6 and maximum of 12 is desirable,⁴ and if they are from the same discipline Toronto⁹ considers that from 12 to 20 experts are sufficient. It is difficult for all experts to continuously participate throughout the whole process, with this ranging between 35% and 87%, and it is therefore advisable to invite a minimum of 30 experts.⁹ The percentage of expert participation must be monitored in each round, from first to last.

To recruit the experts several options are available: sending an invitation to different scientific organisations related to the study theme, requesting they notify their members^1,3,12,13; searching in health databases, identifying authors with publications which are relevant to the field of study, from recent years^9,13; sometimes it may be necessary to create a snowball effect when access to the study population is more complicated.

Distribution of the questionnaire with information to the panel members to aid evaluation

At present questionnaires may be sent online. Different platforms exist to help us in this phase, some of which are free (like for example Google forms)¹⁴ and others to be paid for (like SurveyMonkey).¹⁵ Responses may also be subsequently downloaded onto a database or software may be used to analyse the responses.

In this part of the process strict safety measures must be used to maintain participant anonymity, since group mailing of the questionnaire link can reveal identities through email mailing, if the right measures of concealment are not used.

The questionnaire must contain instructions for completion and information on the review of the literature undertaken with an explanation, using definitions, of several aspects to help with evaluation.^4,11 A time interval must be programmed for response. An interval of 30–45 days may be sufficient, although in some case flexibility is required so as not to lose experts.¹³ it is also recommended to send personalized reminders, if possible, a fortnight and one week prior to the finalisation of the response period, indicating the importance of the experts’ participation.¹³

An initial definition of the number of rounds planned for the study should be made to avoid reaching a “false consensus” due to exhaustion of the experts, who agree to finalise the process. Although there is no unanimity, most Delphi studies used between two and three rounds,¹³ and therefore a minimum of two and maximum of four rounds should be considered.^4,16

Data analysis and feedback to the panel

Data obtained from the expert panel responses must be previously defined through “consensus” on the item. There is no single definition, but the one most used is the consensus percentage.¹³ Humphrey-Murto et al.⁴ consider that when 70% of the experts “agree” or “very much agree” over an item then it is reasonable. 70% of response “in disagreement” or “totally in disagreement” are sufficient to eliminate the item, whilst the rest should be reassessed in the next round. There is also the possibility of using central tendency measures, correlation coefficients, the kappa index, Cronbach’s alpha, etc.⁷

At the end of each round a report on findings obtained is required. A good option is to provide information on each item, with the response from the author and the mean obtained by the rest, so that its positioning with the rest can be viewed and opinion may vary, if applicable, in the following round.¹¹ Feedback must be provided at the end of each round, with as little time as possible passing from its termination, so as not to delay the following round and remainder of the process.

Report on findings

The results from several rounds of a Delphi study may be difficult to comunicate.¹¹ One very summarised version when there are few items would be to only show those that reached consensus, although it is also advisable to express added items, eliminated ones and those which were altered.^11,16 A diagram like the one in Fig. 2 may be used for the report, where it is observed that through the bibliographic review the authors started with 7 categories and 70 items. Using two evaluation rounds, the final version reached a consensus in 57 items, with the whole process being observed and the degree of consensus in all items. For example, we observe that in category 1, where there were 12 items in the first round, consensus was reached regarding 11, the experts added another 3 items and reassessed one in the next round. In round 2 these 4 items were assessed, for which a consensus was reached in 3, eliminating one due to the absence of the necessary consensus. At the end of this category, where there had been 12 items, the final number was 14.

Figure 2.

Graphic examples on the evolution of the items during the whole process for presenting in the report n findings.

n¹: number of items assessed in round 1.

n²: number of items assessed in round 2.

n^f: number of final items for which consensus was reached.

✓: number of items accepted (agree > 70%).

х number of items removed (disagree > 70%).

±: number of items to be reassessed.

+: number of items added by the experts.

Manuscript publication phase

When the final phase is to be published of the results obtained by consensus, Diamond et al.⁷ suggest a series of key methodological criteria which should be responded to in the writing up of the manuscript:

• 
  ▪

  Those relating to the objective: does the Delphi study address consensus as its objective? Is the objective to present the findings which reflect group consensus or to quantify the degree of consensus?

• 
  ▪

  Those relating to the participants: how were the experts selected or excluded?

• 
  ▪

  The definition of consensus: how will the consensus be defined? If applicable, what will the threshold be to consider that consensus has been reached? What criteria have been used to determine the termination of the Delphi in absence of consensus?

• 
  ▪

  In the Delphi process: were items eliminated? What criteria were used to eliminate them? What criteria were used to determine study termination or were used during a specific number of rounds?

Final considerations

Unlike other methods, in Delphi there is no standardisation of definitions, and its use and presentation of reports may lead to a sensation of it being a method which supplies little evidence. This guideline wishes to offer basic aspects for the creation of a Delphi study, offering maximum guarantees. With regard to the type of evidence provided, this depends on approach. A good review of the literature, combined with the experience of the experts with systematization of the whole process may obtain a final product with the maximum rigour possible and provide a type of evidence which is impossible to obtain using regular research methods.

Conflict of interest

The author has no conflict of interest to declare.