Portfolios in medical education: A scientometric mapping of the research field

Abstract

Introduction

Competency-based medical education (CBME) faces assessment challenges, necessitating tools to collect evidence across various domains and levels. Portfolios, a learner-driven collection of evidence enriched through reflective practices, have emerged as a solution. Despite their importance, the research field of portfolios in medical education lacks a structured approach. Hence, our aim was to analyze publications indexed in the Web of Science database from 1994 to 2022 related to portfolios in medical education.

Material and methods

A scientometric mapping study was conducted using publications from the Web of Science database. The search strategy targeted articles or reviews on portfolios in medical education from 1994 to 2022. Data analysis involved performance analysis, conceptual structure analyses, thematic evolution analysis, and identification of emerging trends using bibliometric indicators and visualization tools.

Results

There was an annual growth rate of 9.28% in publications related to portfolios in medical education. The United States, United Kingdom, and Netherlands emerged as the leading contributors. The themes identified included CBME, assessment, e-portfolios, professionalism, and lifelong learning. Thematic evolution analysis highlighted shifts in research, with portfolios initially supporting CBME and later being utilized for entrustable professional activities (EPAs).

Conclusion

Portfolios have become integral to medical education, supporting CBME and facilitating the implementation of EPAs. The study underscores the need for expanded global research to enhance the generalizability and applicability of portfolios across diverse regions. Future research should focus on integrating portfolios more systematically into educational programs and exploring the integration of artificial intelligence to enhance decision-making processes based on portfolios.

Resumen

Introducción

La educación médica basada en competencias (CBME, por sus siglas en inglés) se enfrenta a retos de evaluación, necesitando herramientas para recopilar evidencias a través de varios dominios y niveles. Los portafolios, una colección de evidencias impulsada por el alumno y enriquecida mediante prácticas reflexivas, han surgido como una solución. A pesar de su importancia, el campo de investigación de los portafolios en la educación médica carece de un enfoque estructurado. Por lo tanto, nuestro objetivo fue analizar las publicaciones indexadas en la base de datos Web of Science desde 1994 hasta 2022 relacionadas con portafolios en educación médica.

Materiales y Métodos

Se realizó un estudio de mapeo cienciométrico utilizando publicaciones de la base de datos Web of Science. La estrategia de búsqueda se centró en artículos o revisiones sobre portafolios en educación médica desde 1994 hasta 2022. El análisis de datos incluyó análisis de rendimiento, análisis de estructura conceptual, análisis de evolución temática e identificación de tendencias emergentes utilizando indicadores bibliométricos y herramientas de visualización.

Resultados

Hubo una tasa de crecimiento anual del 9,28% en publicaciones relacionadas con portafolios en educación médica. Los Estados Unidos, el Reino Unido y los Países Bajos fueron los principales contribuyentes. Los temas identificados incluyeron CBME, evaluación, portafolios electrónicos, profesionalismo y aprendizaje permanente. El análisis de la evolución temática puso de relieve los cambios en la investigación, con los portafolios inicialmente apoyando CBME y más tarde siendo utilizados para las actividades profesionales encomendables (EPAs).

Conclusiones

Los portafolios se han convertido en parte integral de la educación médica, apoyando CBME y facilitando la implementación de EPAs. El estudio subraya la necesidad de ampliar la investigación global para mejorar la generalizabilidad y aplicabilidad de los portafolios en diversas regiones. La investigación futura debería centrarse en integrar los portafolios de forma más sistemática en los programas educativos y explorar la integración de la inteligencia artificial para mejorar los procesos de toma de decisiones basados en los portafolios.

Introduction

Competency-based medical education (CBME) represents the forefront of educational methodologies, surpassing traditional models in several aspects.¹ Despite its advantages, CBME encounters significant assessment challenges, necessitating comprehensive tools that consolidate various evidence forms across several domains and levels.^2,3 In this context, portfolios are a solution that effectively addresses these challenges.⁴

Defined as learner-driven collections of evidence enriched through reflective practices,⁵ portfolios are predominantly student-centered from traditional assessment methods. Studies across the spectrum of medical training suggest that portfolios significantly promote reflection, feedback, and learning, offering substantial benefits to educators in health professions.^6–8

Despite its importance in medical education, portfolio research lacks a structured approach and exploration of its thematic progression, which impedes the identification of future research directions. To bridge this gap, we propose employing scientometric mapping, a quantitative publication analysis technique that employs quantitative methods to visualize a research field based on bibliographic data.⁹ This approach has been widely used in medical education research.^10,11 Thus, in this study, we aimed to elucidate the research landscape of portfolios in medical education to guide future scholarly endeavors.

Material and methods

Study design

We conducted a scientometric mapping study to assess publications indexed in the Web of Science (WoS) from 1994 to 2022 in the field of portfolios in medical education.

Procedures

In January 2024, we conducted a systematic search on the WoS database employing the topic subject of portfolios and medical education with their synonyms. The search strategy was as follows: “TI = (portfolio OR portfolios OR e-portfolio OR e-portfolios) AND ( TS = ("Medical education" OR "Medical students" OR "Clinical clerkship" OR "Medical residency" OR "Medical residents") OR AB = ("Medical education" OR "Medical students" OR "Clinical clerkship" OR "Medical residency" OR "Medical residents") OR AK = ("Medical education" OR "Medical students" OR "Clinical clerkship" OR "Medical residency" OR "Medical residents") OR TI = ("Medical education" OR "Medical students" OR "Clinical clerkship" OR "Medical residency" OR "Medical residents"))”. We only included articles or reviews in English or Spanish from 1994 to 2022.

Data analysis

Retrieved studies in CSV format were then imported to biblioshiny from bibliometrix¹² and Vosviewer,¹³ both open-access software. Biblioshiny is primarily used for performance analyses, whereas Vosviewer is better suited for scientometric mapping. Bibliometric indicators used were the number of citations, publications, authors, countries, journals, authors' keywords, and the reference list of each article.

The following analyses were done: (1) Performance analysis in biblioshiny to identify the key characteristics of the dataset, growth in publications from 1994 to 2022, most productive countries and most impactful authors, journals, and articles. (2) Conceptual structure analyses using thematic map in biblioshiny, and co-occurrence of keywords in VosViewer, both based on author keywords. (3) Thematic evolution based on author keywords in biblioshiny. (4) Emerging trends trough bibliographic coupling in VosViewer.

Results

Fig. 1 shows the main information regarding portfolios in medical education. The years with the most publications were 2009 and 2012, with 15 each, while the year with the lowest were 1994, 1997, and 2002. Although there has been no exponential growth in the last decade, the annual growth rate was 9.28%.

Fig. 1.

The main information on the research field of portfolios in medical education and the number of publications per year from 1994 to 2022 in the Web of Science.

Table 1 offers insights into performance metrics per country, authors, journals, and articles. The United States of America, the United Kingdom, and the Netherlands were in the top 3, with 45, 36, and 18 publications each. The production of these 3 countries represents 50% of the research field. Regarding journals, the one with the most citations and publications was Medical Teacher, with 1564 and 33, respectively. Furthermore, from the top 10 most impactful journals, 8 were fully focused on health professions education. When analyzing the top 10 most cited articles, most were guides, recommendations, or evidence synthesis.

Table 1.

Key countries, authors, journals, and articles in the research field of portfolios in medical education from 1994 to 2022 in the Web of Science.

Rank	Country		Authors			Journals			Article
	Name	NP	Name	NP	TC	Name	NP	TC	Title	Author, year	TLC	TC
1	United States of America	45	Erick Driessen	14	925	Medical Teacher	33	1564	Portfolios in medical education: why do they meet with mixed success? A systematic review	Driessen, 2007	56	204
2	United Kingdom	36	Jan Van Tartwijk	7	799	Medical Education	18	1157	AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment	David, 2001	43	147
3	Netherlands	18	Cesar Van Der Vleuten	7	702	Academic Medicine	13	400	The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11	Buckley, 2009	42	284
4	Australia	11	Margery Davis	5	311	BMC Medical Education	10	151	Portfolio assessment in medical students' final examinations	Davis, 2001	41	94
5	Spain	11	Sharon Buckley	2	305	Teaching and Learning in Medicine	4	149	AMEE Medical Education Guide No.11 (revised): Portfolio-based learning and assessment in medical education	Chalis, 1999	40	115
6	Canada	10	Jamie Coleman	2	305	Advances in Health Science Education	3	112	Portfolios for assessment and learning: AMEE Guide no. 45	Van Tartwjik, 2009	36	127
7	Saudi Arabia	5	Tamasine Ashcroft	1	284	Nurse Education Today	1	65	The use of portfolio learning in medical education	Snadden, 1998	35	130
8	Belgium	4	Ian Davison	1	284	Patient Education and Counseling	1	61	Portfolios in continuing medical education – effective and efficient?	Mathers, 1999	35	105
9	Brazil	4	Khalid Khan	1	284	Australasian Journal of Educational Technology	1	48	The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12	Tochel, 2009	31	144
10	Chile	3	Sadia Malick	1	284	Journal of Surgical Education	4	41	Portfolio use in general practice vocational training: a survey of GP registrars	Pearson, 2004	25	70

NP - Number of publications; TC - Total citations; TLC - Total local citations.

Note: Each column is different from the others; in the case of countries, these are ordered by the number of publications (NP) and authors based on total citations (TC).

Fig. 2 shows the conceptual structure of the research field through 2 analyses of a thematic map and keyword co-occurrence. In the thematic map (2A), motor themes—essentials for the research field—were CBME, the use of e-portfolio, and assessment. The major niche theme—solely explored by a group of researchers—was lifelong learning, and emerging or declining themes were entrustable professional activities (EPAs) and the validity of portfolios. In the keyword co-occurrence analysis (2B), 5 clusters emerged. Cluster 1 (red) covered keywords related to CBME. Cluster 2 (blue) keywords’ related validity of portfolios. Cluster 3 (Green) covered the theme of assessment and feedback. Cluster 4 (light green) covered the theme of professionalism and reflection, and Cluster 5 (purple) focused on e-portfolios. Hence, the conceptual structure of the research field is characterized by the following themes: assessment, CBME, reflection and professionalism, validity of portfolios, and e-portfolios. A topic that is gaining interest is using portfolios in EPAs.

Fig. 2.

Conceptual structure through the thematic map (2A) and keyword co-occurrence (2B) in the research field of portfolios in medical education from 1994 to 2022 in the Web of Science.

A thematic evolution analysis, portrayed in Fig. 3, was performed to analyze the field's evolution. Four timeframes emerged, with respective themes explored in each. Between 1994 and 2008, the most essential theme was CBME, while in this frame, assessment was solely a niche theme. Between 2009 and 2012, the theme of feedback added to assessment, and e-portfolios emerged. Between 2013 and 2018, 2 new themes arose: the validity of portfolios and the use of portfolios for developing lifelong learners through reflection. Lastly, between 2019 and 2022, previous topics consolidated as motor themes, while an emerging theme appeared—EPAs. Therefore, portfolios were conceived for competency-based assessment as described in the literature. Later on, they were tailored as a feedback and reflection tool, and due to their benefits, they began to be used for EPA assessment.

Fig. 3.

Thematic evolution in the research field of portfolios in medical education from 1994 to 2022 in the Web of Science.

To identify emerging trends, we conducted a bibliographic coupling analysis depicted in Fig. 4. This analysis creates a cluster based on the similarity of the reference lists of publications and enables the analysis of current themes. Five clusters covered themes: assessment, e-portfolios, professionalism and lifelong learning, and undergraduate and graduate medical education. Hence, this analysis confirmed previous findings regarding core topics in the field of portfolios in medical education.

Fig. 4.

Emerging trends through bibliographic coupling analysis in the research field of portfolios in medical education from 1994 to 2022 in the Web of Science.

Discussion

Summary of findings

Here, we conducted a scientometric mapping of the research field of portfolios in medical education of publications indexed in the WoS database between 1994 and 2022. Our major findings were as follows: (1) The field has shown an annual growth of 9.28%, going from 1 publication in 1994 to 195 in 2022. (2) The leading contributors of the field were the United States of America, the United Kingdom, and the Netherlands, which highlights the concentrated geographic distribution of the research. (3) Most cited publications were guides or recommendations to implement portfolios, and evidence synthesis, this may suggest a well-developed research field. (4) Through different analyses, the major themes contributing to the research field were CBME, assessment, e-portfolios, professionalism, and lifelong learning. (5) The evolution of themes mirrored medical education, as portfolios were conceived for CBME, and now, they are being used for EPAs, as well as using to teach and assess competencies such as lifelong learning and professionalism, in line with contemporary educational needs.

Strengths and limitations

Although our findings were corroborated through several methods and we adhere to reporting guidelines, this study may possess some limitations. First and foremost, we rely solely on the WoS database, which may hinder the generalizability of our findings. Furthermore, our analysis may not cover emerging topics in the field from publications without citations. Thus, there is a need to consider this, as we cannot describe “actual” discussions due to the nature of the study.

Interpretation of findings

There was a growth in publications, being more marked since 2004. This was in accordance with CBME introduction in medical education between 1996 and 1999.¹⁴ Thus, a possible explanation may be that ss CBME shifted the paradigm in assessment toward a formative and student-centered approach. Portfolios were aligned with its core principles, as they were a purposeful collection of evidence—not just a snapshot. Moreover, as portfolios augment feedback opportunities and are student-driven,^4,6 this mirroring between the growth of CBME and them may not be a surprise.

Most publications came from 3 countries—the United States of America, the United Kingdom, and the Netherlands. This is similar to most medical education research, with overrepresentation from developed countries.¹⁵ This hinders the generalizability of portfolio usage, as assessment culture, resources, and overall quality of medical education vary between regions. For example, in less-developed countries, limitations such as reliability and validity, lack of time for assessors, lack of resources to employ an e-portfolio platform, or negative student sentiments toward portfolios may be more prominent.⁶ Hence, there is a need for further research in underrepresented regions such as South America or Africa to understand better how portfolios can be implemented across regions.

We identify a shift toward EPAs from CBME from 2019 to 2022. This may be explained by the portfolios being well-suitable for assessment in CBME and EPA, as they are longitudinal assessment tools covering a wide range of assessments from lower to higher levels in the Miller Pyramid.¹⁶ For example, in EPAs, there is the need to obtain evidence of knowledge (multiple choice questions exam on a procedure), skills (simulation procedures), and attitudes (questionnaire or direct observation) in several scenarios and later on in the specific EPA by several observers, for this case, portfolios serve to collect all this evidence for competence committees to making decisions on entrustment.¹⁷

Regarding professionalism and lifelong learning themes, a possible explanation is that portfolios require reflection from the student's point of view. They are considered a major tool for reflection in medical education.¹⁸ These reflections enable students to gain insights into previous experiences and induce them to take them into account when planning future actions.¹⁹ Considering these 2 points, portfolios reflect the learner's journey across several competency domains in graduate and undergraduate medical education.

Implications and future research

To our knowledge, this study presents the first scientometric analysis of portfolios in medical education, revealing a significant research gap in regions such as South America and Africa. Addressing this gap requires more support from funding bodies and international collaborations to improve portfolios' generalizability and applicability. Pivotal in CBME and EPAs, portfolios should be more systematically integrated into educational programs. This integration necessitates clear guidelines for both educators and learners. Incorporating portfolios into faculty development programs, covering design, implementation, evaluation, and effective feedback mechanisms, is essential for educators. This integration might now include artificial intelligence to enhance decision-making processes based on portfolios. However, determining the optimal way to integrate this technology into educational practices warrants further investigation.

Lasty, this study underscores the increasing interest in using portfolios for medical education, primarily focused on developed countries. This trend underscores the necessity for expanded global research to deepen our understanding and application of portfolios across diverse regions. Portfolios have become essential in medical education, initially supporting CBME and aiding in implementing EPAs. This evolution reflects portfolios' capacity to assess a broad array of competencies, extending beyond clinical skills to include professionalism and lifelong learning. Their comprehensive assessment potential explains the shift towards EPAs, highlighting portfolios' role in advancing medical education globally.

Funding

This study was self-funded by the authors.

Ethics statement

Do not apply as this study employs public available articles data and does not involve human-subjects.