Assessment takes a fundamental role in medical education. In Portugal, the main assessment method used is multiple-choice questions (MCQ). The virtual interactive patients (VIP) as an assessment method can be indicated for the development of clinical reasoning for medical students of the clinical years. This study compared the use of MCQ and VIP as assessment methodologies in medical students.
MethodsA randomized crossover study carried out, which 35 Portuguese medical students from clinical years evaluated by MCQ and VIP in four medical specialties. In each specialty, students randomized into two groups. One group started being evaluated by MCQ and finished with VIP, the other vice-versa. After the evaluation, students asked to complete a satisfaction questionnaire.
ResultsThe final score was higher with the VIP than MCQ method (MCQ: 67.40±8.96 vs. VIP: 73.34±10.8; p<0.01) with significant differences higher in Clinical History and Physical Examination. The satisfaction quiz showed although both methodologies are adequate and satisfactory, but VIP method allows to improve knowledge.
ConclusionVIP is more accurate for clinical reasoning, accepted by medical students and should be integrated into the pre-graduate medical curriculum with other methodologies already used. The COVID-19 pandemic allowed a digital transformation of medical education and VIP will have a crucial role as a knowledge and evaluation method.
La evaluación tiene un papel fundamental en la educación médica. En Portugal, el principal método de evaluación utilizado son las preguntas de opción múltiple (MCQ). El paciente virtual interactivo (VIP) como método de evaluación puede estar indicado para el desarrollo del razonamiento clínico para estudiantes de medicina en años clínicos. Este estudio comparó el uso de MCQ y VIP como metodologías de evaluación en estudiantes de medicina.
MétodosSe realizó un estudio cruzado, aleatorizado, en el que 35 estudiantes de medicina portugueses de años clínicos fueron evaluados por MCQ y VIP en cuatro especialidades médicas. En cada especialidad, los estudiantes se asignaron al azar en dos grupos. Un grupo comenzó a ser evaluado por MCQ y terminó con VIP, el otro viceversa. Después de la evaluación, los estudiantes pidieron completar un cuestionario de satisfacción.
ResultadosLa puntuación final fue mayor con el método VIP que con MCQ (MCQ: 67,40±8,96 vs. VIP: 73,34±10,8; p<0,01) con diferencias significativas mayores en historia clínica y exploración física. El cuestionario de satisfacción mostró que ambas metodologías son adecuadas y satisfactorias, pero el método VIP mejora el conocimiento.
ConclusiónVIP es más preciso para el razonamiento clínico, aceptado por los estudiantes de Medicina y debe integrarse en el plan de estudios de Medicina de pregrado con otras metodologías ya utilizadas. La pandemia por la COVID-19 permitió una transformación digital de la educación médica y VIP tendrá un papel crucial como método de conocimiento y evaluación.
Assessment is one of the main moments in the learning process in medical education.1,2 Multiple-choice questions (MCQ) are the main assessment method used by Portuguese Medical Schools. The MCQ has the advantage of allowing the evaluation of high quantity of knowledge in a variable number of students and in a short time. However, good questions are difficult to elaborate without errors.3
Simulation in medical education has grown exponentially. Although it is typically used as a tool for teaching and evaluating the technical skills, interest has arisen in the possibility of using simulation as a method for knowledge assessment, namely, using virtual interactive patients (VIP), however, there is a lack of evidence.1,2,4–9 Studies suggested VIP could be a new method to evaluate medical students during their training course. Further studies are needed.1,6,7,9–14
Several factors have contributed to the decrease in the real contact of medical students with patients: limiting the exposure, reducing the time of patients in hospitals and the pressure on hospital funding. VIP has been recognized as an ideal and valid tool for medical education to teaching and assessment.7
VIP as a method of knowledge assessment has been suggested to be an efficient method with pedagogical value for the development of reasoning and clinical skills of medical students, given the possibility of completely managing a patient in a given clinical situation previously contextualized, where it is possible to collect a clinical history, perform a physical examination, prescribe complementary means of diagnosis and institute therapy. Thus, it is not limited to factual knowledge.7,10,15,16 This method has several advantages: efficiency, accessibility, interactivity and exploring rare and/or critical clinical cases. However, this method does not allow the assessment of cognitive skills (empathy, negotiation and communication of bad news).12 According to the opinion of medical students, this method has allowed an assessment of knowledge and clinical skills for clinical practice in the future.17
A direct comparison between the use of MCQ and VIP methods can be extremely difficult to apply since the use of VIP addresses skills and knowledge that are not present in MCQ. However, considering that the assessment also provides learning, the use of VIP can have positive implications both at the teaching and learning levels.2
In order to assess the impact of using VIP as a method of assessing knowledge in medical students from clinical years, this study was carried out, with the main objective of comparing the use of VIP and MCQ testing as assessment methods.
All medical students attending the clinical years (4th, 5th and 6th years) from all Portuguese Medical School were invited to voluntarily participate in 4 evaluation sessions of the present study. In each assessment session, each participant was assessed using 2 different methods: MCQ and VIP.
This study was approved by the Ethics Committee of the University of Beira Interior.
MethodsSimilar studies have been developed.1,6,9,11 In the present study, 4 evaluation sessions were organized. Each session contained a different clinical case, from different medical specialties (Cardiology, Pulmonology, Gastroenterology and Neurology).
The clinical case presented was adapted so that it could be evaluated using the MCQ and VIP methods. The questions of the MCQ and the tasks to be performed in the VIP were grouped into five parameters and final score as shown in Table 1. All parameters and the final score were graded from 0 to 100%.
Parameters evaluated during the evaluation sessions, as they were by methodology and their weighting in the final score.
| Parameters | MCQ | VIP | Final score |
|---|---|---|---|
| Clinical history (Hx) | 3–4 questions | Tasks performed to resolve the case, according to guideline | 15% |
| Physical examination (PE) | 3–4 questions | 15% | |
| Complementary diagnostic means (Tests) | 4–5 questions | 30% | |
| Therapy (Tx) | 4–5 questions | 30% | |
| Diagnosis (Dx) | 1–2 questions | 1 question | 10% |
To avoid selection biases, in each assessment session, participants were randomized into 2 groups, according to a cross-study design model. One group started being evaluated by MCQ and finished with VIP, the other vice-versa. At the end, participants were asked to fill out a satisfaction questionnaire.
Multiple-choice questions (MCQ)The MCQ tests were solved online and built according to the recommendations.3,5,18,19 The MCQ followed the model of single best answer among 5 possible options. They were scheduled to be performed in about 20min, with the following order of linked questions: Clinical History (Hx), Physical Examination (PE), Complementary Diagnostic Means (Tests), Therapy (Tx) and Diagnosis (Dx). Participants were not allowed to go back to the previous question.
Virtual interactive patient (VIP)We use BodyInteract™ as a VIP method. This is a medical simulator created in Portugal capable of creating an immersive experience, in which the user assumes the role of physician. VIP cases were scheduled to be resolved in 20min. Simulation ends when all recommended tasks are performed according to guidelines, the patient dies, or when this time ends. After solving the case, the participants had access to feedback on their performance.
Satisfaction questionnaireA satisfaction survey was used, with the consent of its author.1 The questionnaire aimed to analyze the students’ experience and opinion about their performance in each assessment method.
Statistical analysisThe statistical analysis was performed using the SPSS Statistics 27 and the GraphPad Prism 9.
Descriptive statistical techniques were used: mean and standard deviation for the final scores obtained in each method (VIP and MCQ) and scores by parameters (Hx, PE, Tests, Tx and Dx) of each method of all participants.
We use the paired t-test for the null hypothesis between the difference in the final score between the methods and the difference between the same parameters of each method.
In the analysis of the satisfaction questionnaire, given that each participant could have answered the questionnaire several times, depending on the number of evaluation sessions they participated, the mean and standard deviation of points attributed to each item of the questionnaire per participant and then a paired t-test was performed for the null hypothesis between the different points given in each item of each evaluation method.
ResultsVIP and MCQ methodsThe mean final score of the participants was higher with the VIP compared to the MCQ (VIP: 73.34±10.82 vs. MCQ: 67.40±8.96; p<0.01). Statistically significant differences were found in the Clinical History (VIP: 80.88±24.42 vs. MCQ: 69.70±19.30; p<0.5) and in the Physical Examination (VIP: 87.92±1.09 vs. MCQ: 64.03±19.21; p<0.001), where this was superior in the VIP. No statistically significant differences were found for the remaining parameters (Fig. 1).
Satisfaction questionnaire
Participants consider that both assessment methods are adequate and satisfactory. VIP allows for an improvement in knowledge, as well as a better approximation to the clinical reality (Table 2).
Satisfaction questionnaire: the results are expressed as mean and standard deviation of an ordinal numeric variable between 1 (strongly disagree) to 7 (strongly agree).
| Question | MCQ | VIP | p-value | |
|---|---|---|---|---|
| Did you have difficulties to answer the questions (MCQ)/to solve the clinical case (VIP)? | 5.59 (±0.64) | 5.21 (±0.98) | <0.05 | |
| Were you able to concentrate to answer the questions (MCQ)/solve the clinical case (VIP)? | 5.93 (±0.83) | 5.64 (±1.06) | 0.11 | |
| Was solving the test (MCQ)/clinical case (VIP) stressful? | 3.19 (±1.58) | 5.18 (±1.52) | <0.001 | |
| Do you feel satisfied with your performance in the test (MCQ)/clinical case (VIP)? | 4.83 (±1.07) | 4.64 (±1.37) | 0.47 | |
| Do you think that this evaluation method represents an adequate evaluation? | 5.39 (±0.91) | 5.39 (±1.23) | 0.98 | |
| Do you consider that your knowledge improved after the test (MCQ)/clinical case (VIP)? | 5.17 (±1.30) | 5.96 (±0.90) | 0.001 | |
| Are you satisfied with this evaluation method? | 5.54 (±0.84) | 5.78 (±0.93) | 0.15 | |
| Do you consider that the test (MCQ)/Clinical Case (VIP) is close to the clinical reality? | 5.46 (±1.09) | 6.24 (±0.61) | <0.001 |
*Represents p-value <0.5 and *** represents p-value <0.01.
It is expected that the use of VIP will play an important role in medical education in the coming years, both in the learning and evaluation processes, both for medical students and other health professionals.15 The use of equal parameters between both methods allowed the comparison capability in this study.
Our investigation suggests that medical students have better final scores when assessed by the VIP than by the MCQ. However, our study is limited by the fact that it did not take into account the tasks performed incorrectly when approaching the clinical case by the VIP, which could have harmful effects in real life.
Although our study has limitations in terms of the sample of participants that is not statistically significant and students participate voluntarily, these facts can compromise the quality of our study data. VIP has been well accepted by medical students, as has already been shown.16
In this study it was also possible to confirm 2 basic pillars of simulation in medicine with the VIP, approximation of the clinical reality and improve knowledge after the assessment, by feedback.20
Our investigation is favorable to the use of the VIP for the evaluation process of medical students in clinical years. A systematic review points out that the VIP proves to be more effective in improving clinical skills and knowledge.8 However, we know that the methods used in the assessment process in medical education have an impact on learning and have unintended effects, limitations and cultural trends.21 Therefore, a longitudinal assessment process combined with various methods must be implemented, in order to allow teachers to identify and correct the learning needs of medical students for the training of educated physicians, efficiently and with a good relationship cost-effectiveness.6,21 VIP is already used and shared among medical schools in the United States of America and Canada.22
In the future, it is suggested that an equitable standard system for scoring the performance of students when assessed using the VIP remains to be validated,15 as well as a fair penalty system for performing tasks that are incorrect or less cost-effective should also be considered.
Currently, the COVID-19 pandemic is affecting medical education, that must be seen as opportunities for change for medical education, based on teaching knowledge and ignore clinical reasoning and professional skills.23 The suspension of medical education has harmful consequences for the future of health care. To ensure the safety/health of medical students during the pandemic, it is necessary to ensure the continuity of their educational process and evaluation.
Quickly, all medical education has been adapted into online formats.24 Thus, the need to implement new innovative processes in medical education may require deep changes in many disciplines of the medical curriculum, thus contributing to a scenario of innovation and transformation in the medical curriculum. With new virtual educational systems will certainly play an important role as a support tool for the continuity of the educational and evaluation process in worldwide.24,25
FundingThis article was not funded.
Others information'sArticle was made to obtain the Master's Degree in Medicine by the corresponding author from the Faculty of Health Science from University of Beira Interior (FCS-UBI), Covilhã, Portugal.
Abstract was accepted at the 7th Portuguese Nacional Congress os Pedagogical Practices in Higher Education (CNaPPES.21) for an oral communication of the work developed.
Conflict of interestThe authors of this article declare no conflict of interest.
Elisa Cairrão, Assistant Professor at the Faculty of Health Science from University of the Beira Interior (FCS-UBI) and Investigator at the Health Sciences Research Centre of the University of Beira Interior (CICS-UBI), Covilhã, Portugal.
Regina Rodrigues, Anesthesiologist at the Hospital Central do Funchal and Director of the Madeira Clinical Simulation Center, Madeira Island, Portugal.
