ASPIRE to excellence: A novel approach toward benchmarking excellence in medical schools

Abstract

Aspiring to excellence in medical education is a worthy theme for this special issue because in medical schools large and small, new and long-established, the work to improve continues day by day, but there are limited opportunities to document and celebrate excellence. Rankings based on research productivity do not reward excellence in medical education.

ASPIRE to Excellence, begun in 2012, brings experts in different subjects from around the world together to write standards of excellence in specific topics. Student assessment, student engagement, social accountability, faculty development, and simulation have benchmarks of excellence that schools can compare themselves to and submit evidence of their accomplishments. International panels review these documents and determine if the school merits an ASPIRE recognition award. As of August 2016, 71 schools have submitted documentation and 28 have been awarded ASPIRE recognition. This paper discusses the challenges and opportunities of using accreditation like standards written specifically for excellence to identify schools that can serve as benchmarks and role models for others.

Resumen

La iniciativa ASPIRE para reconocer la excelencia en educación médica es un tema apropiado para este monográfico porque las facultades de medicina, tanto grandes como pequeñas, tanto nuevas como con larga trayectoria, realizan una labor continúa de mejora día a día. Sin embargo, existen muy pocas oportunidades para documentarlo y celebrar su excelencia. Y además, los rankings de clasificación universitaria están basados en la productividad científica y no recompensan la excelencia en la educación médica.

La iniciativa ASPIRE, puesta en marcha en 2012, reúne a expertos en diferentes ámbitos de la educación médica de todo el mundo para establecer juntos los estándares de excelencia en áreas específicas. Cómo la evaluación de los estudiantes, la participación y contribución de los estudiantes al currículo y a la facultad, responsabilidad social y el desarrollo de la facultad, aportando así criterios de referencia para que las facultades puedan valorarse a sí mismas y presentar evidencias de sus logros. Paneles de expertos internacionales revisan esta documentación y determinan si la facultad merece ser premiada con el reconocimiento ASPIRE. Hasta agosto del 2016, 71 facultades han presentado documentación y 28 han sido galardonadas con el reconocimiento ASPIRE. Este artículo discute los desafíos y las oportunidades de utilizar este conjunto de estándares, escritos específicamente para la excelencia, como los que se usan para la acreditación, para identificar las facultades que pueden servir como referencia y modelo a seguir por las demás.

Background

Aspiring to excellence in medical education is a worthy theme for this special issue because in medical schools large and small, new and long-established, the work to improve continues day by day, but there are limited opportunities to document and celebrate excellence. Having a valid methodology to define standards of excellence that allow schools to benchmark themselves against those standards has the possibility to facilitate the identification of role model programs and allow schools to focus their quality improvement efforts.

Until now, the opportunities for medical schools to be compared to others have been limited.¹ The move to recognize excellence in universities has been reflected in increasing attention being paid to university league tables such as the Times Higher World University Rankings and the Shanghai Academic Ranking of World Universities.² Such university league tables or rankings, however, fail to do justice to the main task of a university: teaching and learning.³ Strong research programs, ‘star’ faculty, and the reputations of the universities are mistakenly taken as tokens of quality of the education program delivered. Indeed Barbara Ischinger, Director for Education at OECD, suggested that current university rankings do more harm than good because they largely ignore a key measure of quality, namely what goes on in the seminar rooms and lecture theaters. A fundamental problem is that research rather than teaching is rewarded. How likely are measures of research intensity and objective timed test scores to identify and excellent integrated curricular design? How would those simple measures take into account value of different assessment systems that go beyond knowledge and contribute to the assessment of skills, behaviors and attitudes? How would high-functioning academic advising system that has ample variety of mentors contribute to a rating of excellence? Would an excellent interprofessional educational program be captured in these simple measures? Even less likely to score high on these rankings are those schools whose mission driven focus is to improve health of their surrounding community through social accountable actions. In June 2010 Annals of Internal Medicine summarized his analysis as follows: “School rankings based on the social mission score differ from those that use research funding and subjective assessments of school reputation.”⁴ So if none of these variables are taken into account by these popular press ratings, how can we turn to them trust that we are identifying excellent medical education programs?

One might consider turning to the accreditation process for help in identifying excellence in medical education programs. After all, standards that are developed by peers are used to drive a school's own self-study which contributes to an on-site visit carried out by peers and this is capped off with an extensive report sent to the school. Surely this information could be used to identify schools that are performing at the highest level? Unfortunately, this is not the case. Accreditation is a powerful tool for feedback but by its very nature is limited to establishing and evaluating minimum levels of achievement. It is not, however, suited to identify exemplary components of a medical school. This hard truth was confirmed by efforts of the Secretariat of the Liaison Committee on Medical Education (LCME) as they worked to try and devise a way to use the data generated by these survey visits to identify exemplary practices. The LCME is the accreditation agency that oversees the quality of MD granting programs in the US and Canada. Created 74 years ago, it is one of the world's oldest if not the oldest accreditation organizations that focuses exclusively on undergraduate medical education programs. Beginning in 2010, the Secretariat, one of whom is the first author of this paper, for the LCME tried two different methods of using data generated by the accreditation process in an effort to identify schools that were exemplary. The intent was to identify programs that could serve as positive examples for other schools to learn from. It was also hoped that the accreditation process could offer a “carrot” to complement the traditional “stick” that goes with a bad accreditation outcome. Survey team members were trained to identify strengths when they visited schools in their role on an accreditation survey team. These strengths were aligned with the standards in hopes of reducing subjectivity and generalizability. Unfortunately, these efforts were abandoned due to concerns about the validity of the findings and the risk of mistakenly identifying an area of strength that might then guide others in the wrong direction. The analysis of this failure led to the truism, that when the standards are written to identify gaps and weaknesses, it is almost impossible to use them to identify strengths. In July 2016, these concerns contributed to the decision to remove the category of strengths entirely from the team report and process. Accreditation standards, by definition are written to look at a program from the bottom-up and to identify gaps and deficiencies. They are not designed to differentiate the very best. The lesson learned was that regardless of how rigorous the accreditation process is, unless the standards themselves are designed to identify excellence there is high risk for false positives and false negatives.

ASPIRE to Excellence

Thus, in this relative vacuum emerged the ASPIRE to Excellence initiative. ASPIRE was first announced in August 2012 at the annual meeting of the Association of Medical Education in Europe (AMEE) in Lyon, France. A gathering of international experts in medical education were convened and initially explored how to go about identifying an excellent medical school. However, it soon became clear that this was too large of a first step given the complexities inherent in training a physician. So instead of the school as a unit, the group identified important components of a school that could be evaluated for excellence. The first three areas that were selected were social accountability, student engagement, and student assessment. Panels of experts from around the world set to work to write standards of excellence in these three areas. Rather than accreditation standards that focus on minimum achievement, these new standards were written to specifically identify the evidence that would be needed to demonstrate that a given school was excellent in one of these specific areas. For example, as you will read in a later paper in this issue, involving students in the management of the curriculum is a key indicator that a school is engaging their students as adult learners. As a testimony to the interest and perceived need for this benchmarking process, experts from around the world readily volunteered and worked hard to reach consensus on language that would separate the good from the excellent. The process of reaching consensus from so many points of view was in and of itself an important contribution to the scholarly understanding of peer evolved definitions of excellence in each of these areas. Since this beginning, the topic of faculty development was added in 2015–2016 and standards for excellence for simulation programs will be available January 2017.

Schools that believe that they are excellent these areas are encouraged to apply for ASPIRE recognition in one or more of the categories. Just like an accreditation system, schools use the guidelines available on the ASPIRE website http://www.aspire-to-excellence.org/ to answer questions related to these standards. These applications are reviewed by panels of international experts in that particular field. In order to keep costs down for medical schools, there is no on-site visit to validate the documents. This is recognized as a limitation of this effort and in an effort to counter this gap, an independent survey of students is required. Even without a costly school visit, an application fee of £2500 ($3675.00) is charged to offset the cost of the review process.

The reviewers are experts from all over the world and independently rate the school based on the ASPIRE standards. These panels made up of three individuals then interact with e-mails and skype calls to debate the merits of the school and then reach a consensus on a recommendation to the ASPIRE board on whether to award a school recognition. The ASPIRE board oversees the panel's recommendations, adjudicates unresolved issues and makes the final decision. As a measure of the global nature of the effort, the panels and the board have representatives from 23 different countries.

Results

The table below indicates the number of submissions and awards for the first three years with the first awards for faculty development slated for August 2016 (Table 1).

	Assessment		Student engagement		Social accountability		Faculty development		Total submissions	Total awards
Year	Submissions	Awards	Submissions	Awards	Submissions	Awards	Submissions	Awards
2013 (inc pilots)	7	2	12	6	10	3			29	11
2014	3	1	6	1	5	2			14	4
2015	2	0	7	6	7	0			16	6
2016	1	0	2	1	4	4	5	2	12	7
Category totals	13	3	27	14	26	9	5	2	71	28

The international enthusiasm for this effort has been encouraging. For example, there are 12 different countries represented in the 13 awards for student engagement.

It is important to understand that ASPIRE is not just about recognition. It is the first attempt by any international body to utilize a peer based approach to recognize excellence in medical education programs. The very existence of standards of excellence developed by peers for these subjects is, in and of itself, a remarkable achievement. That panels of busy medical educators are willing to take the time to read through voluminous documents to get at a truth is a testimony to the interest and value of this work. But most importantly, it is about encouraging all schools to improve and to help each other by sharing best practices with each other. Out of this effort, schools that have won these awards have stepped up in unexpected ways to help others to improve. On the AMEE 2016 meeting in August in Barcelona there are five workshops or symposiums hosted primarily by schools that have won an ASPIRE award. These workshops are less about how to apply for recognition and more about the key ingredients for excellence in these domains. To encourage and galvanize these efforts, ASPIRE recognized schools are invited to join the ASPIRE academy and meet with the ASPIRE board to further the dialogs and to find ways to help them help other schools.

Challenges remain. In each of the ASPIRE topics there are issues that have emerged that require continued work. The most challenging is to ensure consistency across the various decision making panels in order to ensure comparability of evaluation. Similarly, the challenge of achieving consensus across so many different perspectives and cultures can be challenging when strongly held opinions by individuals about what constitute excellence must be molded into a group consensus. Applications are in English and so schools from non-English speaking areas are at an inherent disadvantage even though short summaries of appendices are accepted in the application. The cost of the program still exceeds the revenues from the applications and requires continued subsidization and ideally, the application fee should be lower to allow lower resourced schools to participate. Recent donations from supporters have helped but there is still a long way to go. Nonetheless, the very fact that a set of criteria now exist for these four areas, with others such as simulation being considered for development, is in itself a remarkable achievement given the nature of the topics and the diversity of the experts. Questions remain which will frame an ASPIRE retreat:

• •

  How can excellence be equated in contexts where available resources may differ significantly.

• •

  Should excellence be awarded at different levels – gold, silver merit for example?

• •

  Can criteria developed in medicine be equally applied in other professions such as dentistry and veterinary medicine?

• •

  Is excellence recognized in only a small number of world leading outstanding schools or can it be found more widely as a measure of quality significantly above what would be found in an accredited or even ‘good’ school?

• •

  Do excellent schools require to demonstrate excellence in all the criteria specified under a theme or can they be regarded excellent through their performance in some of the areas?

• •

  Should a detailed numeric coding be applied to a range of pre-determined criteria and an award made on this basis, or should there be greater emphasis on a more holistic global rating assessment?

• •

  Should the information with regard to submissions and recommendations be publicly available?

The ASPIRE for Excellence is a relatively new program that has achieved considerable success for such a short period of existence. Where it goes for the future is currently being discussed at the board level. One consideration is to find a way to go back to the original goal of identifying excellence at the level of the school rather than components. Could one identify a set of topics that if a school achieved an ASPIRE award in each, would that constitute evidence of an excellent school? One school, Southern Illinois University has already been awarded ASPIRE recognition in three areas, are they on the way to that status already?

Conflict of interest

The authors of this article declare no conflict of interests.