The Delphi method in rheumatology research: are we doing it right?

Abstract

Introduction

The Delphi method is a technique for reaching consensus by a group of experts that express their opinion on a particular issue. It has been widely used in different areas of knowledge, including health sciences. Rheumatology is one of the medical specialties that has most widely used this consensus technique.

Objectives

To review the ideal process for implementing the Delphi method. To describe the current impact of the Delphi method on health research, and take a critical look at its application in rheumatology research.

Materials and Methods

The frequency of use of the Delphi method in publications indexed in MEDLINE was evaluated, as an approach to measure its use in health sciences. A search of the articles reporting the use of the Delphi method as a research tool in rheumatology was conducted.

Results

A total of 4,574 articles were found when searching Medline using the MeSH “Delphi Technique”. The analysis included 148 articles that applied the Delphi method in rheumatology research. According to the findings, the application of the method did not follow the guidelines originally defined, either because of failure to meet its distinctive characteristics, or because of omission of one its phases, or lack of rigor in the implementation of the different phases; so much so, that in the end, only one fifth of the articles analyzed met the distinctive characteristics of the methodology, which could jeopardize the validity of the research results reported.

Conclusions

There is a growing trend to use the Delphi method in health research, and rheumatology is no exception. The lack of standardization and failure to adhere to the Delphy methodology may jeopardize the validity of the results obtained from its use in research. Researchers should take into account the basic methodological premises of the Delphi method and include them in their work.

Resumen

Introducción

El método Delphi es una técnica que permite llegar al consenso en un grupo de expertos que opinan sobre un asunto específico. Se ha usado ampliamente en diferentes áreas del conocimiento, entre ellas las ciencias de la salud. La reumatología es una de las especialidades médicas que más ha empleado esta técnica de consenso.

Objetivos

Realizar una revisión del proceso ideal que debe cumplir la aplicación del método Delphi. Describir el impacto actual del método Delphi en la investigación en salud y hacer una mirada crítica a su aplicación en la investigación en reumatología.

Materiales y métodos

Se evaluó la frecuencia del uso de método Delphi en las publicaciones indexadas en MEDLINE como una aproximación a la medición de su empleo en ciencias de la salud. Se realizó una búsqueda de los artículos que reportan el uso del método Delphi como método de investigación en el área de reumatología.

Resultados

Al realizar una búsqueda en Medline usando el término MeSH “Delphi Technique” se encontraron 4.574 artículos. Se incluyeron 148 artículos que aplicaron el método Delphi en investigación de reumatología. Se encontró que la aplicación del método no ha seguido los lineamientos definidos originalmente, ya sea por no cumplir sus características definitorias u omitir alguna de sus fases, como por la falta de rigurosidad en el desarrollo de las mismas, a tal punto que, solo una quinta parte de los artículos analizados cumplía las características definitorias del método, lo que puede poner en riesgo la validez de los resultados reportados por estas investigaciones.

Conclusiones

El uso del método Delphi en investigación en salud es cada vez mayor y reumatología no es la excepción. La no estandarización y apego a la metodología Delphi puede poner en riesgo la validez de los resultados que se obtienen de su uso en investigación. Los investigadores deben tener en cuenta los aspectos metodológicos básicos que definen el método Delphi para incluirlos en sus trabajos.

Introduction

The study of health and disease processes gives rise to a number of questions that are not always easily solved, using research methods based on measuring and quantifying an event and its impact on a patient, on a particular population or the community. For many health-related questions there is little information and occasionally, the clinician faces situations in which the answer to the issue may not be available in the classical sources of evidence. Therefore, the knowledge and perceptions of experts on a subject matter is often required, or there is a need to reach an agreement on a particular issue among those involved in healthcare, in order to guide decision-making.¹ Expert consensus is a valuable tool when searching for answers in the above situations, and it has also been crucial in the development of various scientific and research processes. For instance, in the development of measurement instruments, consulting experts helps to decide whether the items in a measurement tool actually represent the construct to be measured. When making decisions on the allocation of research resources, the publication of articles in scientific journals, and other similar processes involving selecting and choosing, consulting experts helps to define the most appropriate selection criteria, by considering different viewpoints and opinions of the participants in the process.²

Reaching a consensus among several experts or individual stakeholders is not an easy task. Hence, consensus techniques have been developed in order to reach agreements and set criteria to facilitate fulfilling the objectives of the consultation process. One of these techniques is the Delphi method, probably considered one of the most popular methods currently used in the area of health. Year after year, there is a growing use of this methodology, and rheumatology is not the exception in the growing use and implementation of this consensus methodology in research papers on various aspects dealing with the diagnosis and treatment of rheumatic diseases. However, as a result of the potential variations that researchers may introduce when implementing the methodology, the validity of the process, and therefore of the results, has been questioned; particularly because many of these variations are implemented without assessing their validity and suitability.

This article is intended in the first place, to provide the reader with a review of the ideal process to be followed in the implementation of the Delphi methodology; and second, to discuss its current impact on health research with a critical view towards the implementation of the Delphi methodology in rheumatology research, as an example of its use, since this is one of the areas of medical knowledge where the method has been most widely applied as a research technique

Materials and methods

A literature review following 3 complementary search strategies was conducted: the first one was intended to identify any articles reporting methodological aspects of the Delphi method and variations in its implementation in health sciences; The second strategy was aimed at identifying the frequency of use of the Delphi method in Medline indexed publications, as a quantitative approach in health sciences. And the third strategy was designed to identify any articles reporting the use of the Delphi method in rheumatology research. With regards to first strategy, a search was conducted in databases including Medline, Embase, Clinical Key and Scielo-Bireme; in every case, the term MeSH Delphi technique was used, together with filters such as review, systematic review and technical report. The results were restricted to articles in English and Spanish, with unrestricted date of publication. The consultation was conducted on August 18, 2018. After excluding any duplications, all the titles and abstracts were reviewed. All articles reporting the use of the Delphi method as a consensus technique in health-related matters were included. The second strategy was based on an analysis of the number of articles under the term MeSH Delphi technique, that had been published between January 1st, 1975 and December 31st, 2017. The consultation was conducted on January 19, 2019. To identify the studies that followed the Delphi method in rheumatology, the search was conducted in Medline, Embase, Clinical Key and Scielo, using the MeSH terms Delphi Technique, Rheumatology, Rheumatic Diseases y Consensus. The search was limited to articles in English and Spanish, with unrestricted date of publication. 517 articles were identified. All of the abstracts in these studies were read, excluding duplications, articles with a consensus methodology other than Delphi, book chapters, review articles, letters to the editor, and studies from other medical specialties. 148 articles were included, all of which were fully read and reviewed, including any supplements. The consultation took place on October 30, 2018.

History, definition and characteristics of the Delphi method

The term Delphi in English, from the word Delphos (ancient Greek city home to Apollo’s temple, which the Greek people visited looking for Pythia, a priestess and clairvoyant), refers to a technique used to reach a consensus by a group of experts who express their opinions on a particular issue.³ The method was devised in Santa Mónica, USA, in the early 50’s, by Olaf Helmer and Theodore J. Gordon, who were part of The RAND Corporation (ResearchANdDevelopment) center, a US think tank that provides support to the US Armed Forces. The first implementation of the methodology asked 7 national defense experts about the potential industrial targets in case of a probable Soviet bombing, and how many «A» bombs would be required to accomplish that goal.⁴ Rather than holding joint meetings with the experts, each one of them was asked to provide information which was then collected and contrasted against the information from the other experts. Then, the individual opinions were submitted to each expert anonymously, so that a consensus could progressively be reached. The original document explained the advantages of the methodology which was «more conducive to independent thinking» rather than direct confrontation, to prevent the participants from rejecting new ideas, or from being adamant in defending their own personal position once it had been expressed; or on the contrary, to give in to someone else’s ideas simply because they were presented in a persuasive manner (or by a superior).

The Delphi method is an iterative process, designed to combine the opinion of a panel of experts into a consensus.^5–7 It is a structured methodology designed to systematically collect expert judgment about a particular problem,⁸ process the information,⁹ and finally reach a general group agreement.^10–12 4 distinctive characteristics of the Delphi method were identified¹³:

• •

  Iterative process: experts shall express their opinion more than once, through several rounds leading to stabilizing the opinions, so that the expert may be able to reflect (reconsider or reaffirming his/her opinion), in the light of his/her own views or those of other experts.

• •

  Anonymity: none of the members of the panel knows to whom a particular opinion or answer belongs to, thus preventing negative or positive influences from the stronger members of the panel. There is no direct communication among the experts since all communications are managed by the coordinating team

• •

  Feedback: before starting each round, the participants receive their overall positions vis a vis the problem or situation being considered, highlighting the significant contributions by one particular expert, any conflicting opinions, or any additional information requested by any of the participants. So, before each round, the participants may compare his/her opinion against the opinions of the rest of the group, and reconsider or reaffirm their position with respect to the issue involved.

• •

  Developing a consensus: the final goal of the Delphi method is to reach a general group agreement through the statistical processing of the differences and coincidences, among the individual assessments and change of mind from one round to the next.

With regards to the methodology, 3 key components were identified in the Delphi method^14–19:

• •

  The issue: usually difficult (or impossible) to solve with a traditional research methodology and therefore the knowledge and opinion of experts is sought-after.

• •

  Coordinating team: structures and coordinates the research process. It is responsible for designing the protocol, setting the criteria for the selection of experts, preparing questionnaires, ensuring the flow of information among the experts throughout the iterative process of consultation and its corresponding feedback, analyzing the answers from each round, preparing any subsequent questionnaires, and finally draft the consensus document. The number of people in the coordinating team may range from 2 to 5.

• •

  Experts panel: The key to a successful Delphi process lies in the proper selection of its participants. The underlying issue is to define expert, since this definition varies in accordance with the issue at stake. Some of the criteria to be used include: medical specialty, years of experience in the area, being a member of a particular group or institution, prestige in the field or number of publications on the topic of interest. The number of experts depends on the objectives of each particular study, but in general the number ranges between 7 and 30. More than 30 does not contribute to improving the accuracy per additional expert included, and the increased costs and additional research effort do not compensate for the marginal improvement in outcomes. For the selection criteria, the coordinating team should also consider the level of interest, cooperation and ability to contribute to the process, since the Delphi methodology demands the active participation of all the experts for a continuous length of time, in order to finally reach a group consensus.

The method can be divided into 3 phases²⁰:

• •

  Preparation phase: his involves preparing the instrument to be submitted to the the group for consideration. It is usually a questionnaire which for the first round should preferably be mostly based on open questions, though “yes / no, or agree / disagree” questions may also be included, as well as rating scales or Likert-type.

• •

  Consultation phase: Internet is the recommended method to make consultations because it is expeditious, practical, provides privacy and is low cost. Based on the answers to the questions of the first questionnaire, new questions and assessment items shall be designed for the second round. The second round will circulate a questionnaire with closed questions, asking the experts to do one of the following: establish a hierarchy by listing in order of importance the issue vs the situation being studied; assessing or assigning scores based on a predetermined scale (for example, from 0 to 5 or a Likert scale); quantitatively estimating (for instance, assigning a percentage value). The third round will integrate the answers individually given by the experts, and each expert shall receive the questionnaire again (or a version thereof if any changes had been made), accompanied by his/her answers and the results of the statistical analysis of the group answers for the previous round. The statistical analysis depends on the nature of the item being assessed, usually using measures of central tendency and dispersion, in addition to other measurements such as Kendall’s W coefficient of concordance, Wilcoxon’s test, or Cronbach’s alpha coefficient. Then, the expert is asked to reconsider his/her opinions, taking the opinion of the group into account, so that the expert may keep or change the answer given in the previous round, in light of the new information received. Usually a total of 2–4 rounds are needed to increase the convergence of the opinions, and finally arrive at a consensus.

• •

  Consensus phase: right from the beginning of the study, the researchers shall establish how the agreement will be defined and the expected percentage to decide that a consensus has been reached. The percentage agreement varies and to a large extent depends on the issue studied; usually the goal is to reach an agreement above 70% in the answer given to each item under consultation. Fig. 1 summarizes the basic components and the various phases of the Delphi methodology.

  
  

  Fig. 1.

  Basic components and phases of the Delphi Methodology.

Once a consensus has been reached, the final document is prepared. This document shall list the characteristics of the experts panel, the criteria used for their selection, the way in which the answers by experts evolved throughout the various rounds, the statement of procedures chosen to define the consensus and the level of consensus achieved.²¹

Strengths and weaknesses of the Delphi method

The Delphi method has been criticized for several reasons. The first reason pertains to one of its basic principles: anonymity and lack of direct interaction among the experts; the criticism is that no discussion and debate is allowed among the experts, which could presumably enrich the process.²² This may be partly offset by the ability of the group to ensure the adequate flow of information (i. e. identification, aggregation and synthesis of the answers), in order to maximize feedback at each round. The second reason refers to the length of the process, since a single round Delphi exercise may last 3 weeks and a 3-round process could take between 3 and 4 months.²³ The method is very sensitive to the way the questions are asked, hence the coordinating team must be knowledgeable about the topic to be able to prepare the questionnaires as comprehensively and accurately as possible, with the right proportion of open and closed questions, maximizing the quantity and the quality of the information obtained from the panel of experts.²⁴ The advantages include: the possibility to have experts from around the world with the current Internet-based technologies, which reduces considerably the management costs,²⁵ and the wealth of the information collected that empowers the experts. After participating in the exercise, experts will have a more comprehensive vision of the health problem approached using the Delphi methodology.²⁶

Variations of the Delphi method

Few authors use a pure approach to the Delphi technique as was originally described²⁷; furthermore, there are no universally accepted formal guidelines to implement a Delphi exercise; so over time, and based on the specific needs of the various research processes, variations to the Delphi methodology have been introduced. These changes represent a potential risk for the validity of the outcomes since they are introduced by the researchers based on the research question to be answered and the objectives of the research project, without adhering to concrete parameters. This results in a relatively broad range or variation. There are three known variations to the Delphi method²⁸:

• •

  Conventional Delphi: classical iterative process to combine the opinions of a panel of experts into one consensus.

• •

  Modified or real time Delphi: this is the shortest variation of the method in which the process takes place at a meeting, using tools to immediately summarize the answers of the participants.

• •

  Policy Delphi: Ideas forum where decision-makers seek to educate a group that will then submit their opinions and potential options about an issue, in order to have an informed group.

Current impact of the Delphi method on health-related research projects

The Medline search using the term MeSH Delphi Technique, between January 1st, 1975 and December 31st, 2017, identified 4,574 articles. The use of this type of research methodology has been growing steadily over the last few years, as shown by the analysis of the results from identifying the number of publications indexed under the term MeSH Delphi Technique in Medline. Fig. 2 depicts the tendency followed by these publications over the period of time analyzed.

Fig. 2.

Distribution of articles relating to the Delphi methodology in the health area, per year. January 1975 through December 2017. Taken from Medline. Accessed date: January 9, 2019.

Delphi method in rheumatology

A systematic search of the research papers in rheumatology where the Delphi methodology was used, identified 148 articles. ^9–173 The journals with the largest number of this type of publications were “Reumatología Clínica” (30 articles), followed by “Annals of The Rheumatic Diseases” (24 articles), Journal of Rheumatology (15 articles), Arthritis Care and Research (14 articles), and Joint Bone Spine (9 articles). 89.9% (n = 133) of the articles were on adult rheumatology, 5.4% (n = 8) in pediatric rheumatology, and 4.7% (n = 7) approached the two specialties. The most frequent topic researched using the Delphi methodology was the treatment of rheumatic diseases in 43.2% (n = 64) of the articles, followed by a diagnosis of rheumatic diseases in 18.9% (n = 28) and ultrasonography in rheumatology in 8.1% (n = 12) (Table 1).

Table 1.

Topics discussed with the Delphi methodology for research.

Topic	N	%
Treatment of rheumatic disease	64	43.2
Diagnosis of rheumatic disease	28	18.9
Ultrasonography un rheumatology	12	8.1
Measuring the rheumatic disease activity	8	5.4
Quality of care of rheumatic patients	7	4.7
Reports on clinical experiments in rheumatology	7	4.7
Remission of patients in rheumatology	4	2.7
Treatment of comorbidities in rheumatic patients	4	2.7
Education in rheumatology	4	2.7
Rheumatic patient-focused outcomes	3	2.0
Transition clinics in rheumatology	3	2.0
Establishment of clinics specialized in rheumatology	1	0.7
Functionality of rheumatic diseases	1	0.7
Definition of surgical indication in rheumatology	1	0.7
Electronic medical record in rheumatology	1	0.7

With regards to the rheumatic diseases studied, rheumatoid arthritis was the first one with

23% (n = 34) of the studies, followed by arthrosis with 12.2% (n = 18) and psoriatic arthritis in 8.1% (n = 12) (Table 2).

Table 2.

Rheumatic diseases studied using the Delphi methodology.

Rheumatic disease	N	%
Rheumatoid arthritis	34	23.0
None in particular	27	18.2
Arthrosis	18	12.2
Psoriatic arthritis	12	8.1
Gout	10	6.8
Ankylosing spondylitis	8	5.4
Systemic sclerosis	7	4.7
Idiopathic juvenile arthritis	7	4.7
Rheumatic polymyalgia	4	2.7
Fibromyalgia	4	2.7
Non-specific IJD	3	2.0
Spondyloarthritis	3	2.0
Osteoporosis	3	2.0
Inflammatory myopathies	2	1.4
Chikungunya infection	2	1.4
Sjogren’s syndrome	2	1.4
IBD-associated spondyloarthritis	1	0.7
Systemic lupus erythematous	1	0.7

IJD, Inflammatory joint disease; IBD, Inflammatory bowel disease.

As previously mentioned, the Delphi method has 4 distinctive characteristics (iterative process, feedback, anonymity, and building of a consensus), all of which should be fully met by the studies using this consensus methodology. However, in the case of the articles reviewed using the methodology in rheumatology, only 18.2% (n = 27) were found to be compliant with the 4 characteristics. 27.7% (n = 41) met 3 characteristics, 23.6% (n = 35) met 2 characteristics, and 30.4% (n = 45) met only one of the distinctive characteristics of the methodology. Feedback is one of the most important aspects of the methodology since it enables each participant to contrast his/her views with those of the rest of the panel, in order to restate or reaffirm his/her position vis a vis the issue or the question of interest. Among the articles reviewed, only 41.2% (n = 61) implemented feedback. The process was iterative in just 68.2% (n = 101) of the studies conducted, using 2 rounds of consultation in 31% (n = 49), 3 rounds in 31.1% (n = 46), and 4 or more rounds in 3.4% (n = 5). In 12.2% (n = 18) of the articles the authors did not establish the number of rounds conducted during the Delphi exercise. Anonymity is the third very relevant characteristic, since it avoids the negative or positive influences of the stronger panel members. With regards to studies in the area of rheumatology, only 24.3% (n = 36) of the articles explicitly stated that the Delphi exercise was conducted anonymously. Consensus building was the only characteristic of the 4 Delphi method characteristics that was fully met in the studies analyzed. In terms of the method to define the agreement, the most frequently used was to establish a minimum number of experts assigning a predetermined minimum score to the item studied (38.5%, n = 57), followed by choosing the items rated with a particular minimum average score (30.4%, n = 45).

In 21.6% (n = 32) of the studies, the coordinating team was reported to have in average 5 (±3) coordinators per study. A mean of 28 (interquartile range: 40.5) experts in each panel was reported per study. Most of the rheumatology studies that used this technique were conducted with experts from one country (58.1%, n = 86). The most frequently used method of communication was the e-mail (37.2%, n = 55), followed by meetings in person (13.5%, n = 20), use of the digital platform Survey Monkey (7.4%, n = 11), use of a digital platform specifically designed for the exercise (6.1%, n = 9), and use of postal mail (1.4%, n = 2). The means of communication with the experts panel was not reported in 34% (n = 45) of the studies.

The modification to the Delphi technique was introduced in 14.9% (n = 22) of the studies; however, in 18 of them no explanation was given about what the changes involved; in 4 studies, the modification was to hold a meeting of the panel of experts in person following the rounds of consultation; one of the studies specified implementing a «RAND/UCLA» method, without giving any additional details about what the method involved.⁴⁷ In order to make reference to the modified versions of the method, some particular terms are coined, including informal Delphi⁵⁶ or Delphi like study,⁸⁶ without any major considerations or clarifications about the meaning of these terms, but this is a clear indication that the Delphi methodology was not followed as such, but was adapted to the needs of the researchers. This is clearly evidenced in a consensus on osteoporosis treatment: «A Delphi methodology, adapted to this consensus, was used».¹²⁴ Finally, with regards to the articles that claim having used the modified method, none of them mentions an assessment of the validity of the methodology or its outcomes.

Discussion

The Delphi method has become increasingly relevant over the past few years, with a trend towards a growing number of articles discussing the use of this technology in the area of health. The studies conducted with the Delphi method usually answer a certain number of questions that could not be solved using traditional research methods. It is therefore necessary to seek an expert consensus to complement the study of various health-related issues.

As previously discussed, implementing the Delphi method demands a structured technique that may turn out to be complex, depending on the issue and the number of items for which a consensus needs to be reached. The success of the study and the validity of its results, depends on the right selection of experts, as well as on the proper management of the questionnaires and an optimal flow of information during the consultation rounds. The example of the use of the Delphi method in rheumatology research herein discussed, shows that in general, the implementation of the method did not follow the original guidelines, either because of failure to comply with the distinctive characteristics, or because one of the phases was missing; or because of the lack of rigor in the development of the various phases. So much so, that only one fifth of the articles analyzed met the distinctive characteristics of the methodology, which may put at risk the validity of the results reported by these research projects. Moreover, modified versions of the method were used in almost 15% of the publications reviewed and this is further aggravated by the fact that such modifications are not explicitly described by the researchers and hence it is impossible to assess their suitability, relevance and validity. None of these aspects are reported in the studies reviewed when introducing variations to the methodology, or when changing any of the 3 methodological variations acknowledged.²⁸ These findings are consistent with the reports by other authors. Boulkedid et al., in 2011, conducted a systematic review which included more than 80 studies that used the Delphi methodology, in order to reach a consensus in the selection of health-related quality indicators. They found that less than 40% of the studies included followed the adequate methodology in terms of informing about the answers and giving feedback to decide whether a consensus was reached.¹⁵ Already in 1987, Goodman¹⁷⁴ criticized the not so rigorous implementation of the method and its variations in nursing; he feels that apparently both, researchers and editors of scientific journals, have not tackled the issue. More recently, other authors are urging the scientific community to review the limitations and scope of the consensus and agreement techniques,¹⁷⁵ and embrace again the research of the Delphi method and its health applications.¹⁷⁶ It is surprising to see that although it is a widely used and old methodology, to date there are no universally accepted methodological guidelines for its implementation, to ensure the validity of its results, notwithstanding the ongoing concerns of several authors that have sought to establish methodological guidelines for the Delphi technique.^19,21,177 There is then a clear need to establish and validate certain minimum performance criteria so that an expert consensus exercise using the Delphi method validly accomplishes its objective. This brings up an opportunity to do methodological research on approaches and consensus strategies that – as was recently suggested by Humphrey-Murto and de Wit¹⁷⁸ - shall among other aspects, includes the development of a clear and standardized definition of the term «consensus», the use and validity of the Delphi technique and any variations thereof, including its combined application with other techniques (for example, nominal group, focus group). As clinicians and users of research outcomes, we shall encourage a critical attitude towards the assessment of studies that report consensus-based results, and accordingly establish which of them proof to be valid and applicable in patient care.

Conclusion

There is a growing use of the Delphi method in health research. However, changes to the original methodology are often introduced, but the impact of these variations is not assessed and therefore, the validity of the outcomes is at risk. Researchers, and even the scientific journals themselves, are required to be more rigorous in the implementation of the methodology. Further research on consensus strategies is needed, with a view to ensure the validity of the research results.

Financing

Financial support was provided by the Colombian Association of Rheumatology.

Disclosures

None.