A new method for the assessment of patient safety competencies during a medical school clerkship using an objective structured clinical examination

INTRODUCTION:

Patient safety is seldom assessed using objective evaluations during undergraduate medical education.

OBJECTIVE:

To evaluate the performance of fifth-year medical students using an objective structured clinical examination focused on patient safety after implementation of an interactive program based on adverse events recognition and disclosure.

METHODS:

In 2007, a patient safety program was implemented in the internal medicine clerkship of our hospital. The program focused on human error theory, epidemiology of incidents, adverse events, and disclosure. Upon completion of the program, students completed an objective structured clinical examination with five stations and standardized patients. One station focused on patient safety issues, including medical error recognition/disclosure, the patient-physician relationship and humanism issues. A standardized checklist was completed by each standardized patient to assess the performance of each student. The student's global performance at each station and performance in the domains of medical error, the patient-physician relationship and humanism were determined. The correlations between the student performances in these three domains were calculated.

RESULTS:

A total of 95 students participated in the objective structured clinical examination. The mean global score at the patient safety station was 87.59±1.24 points. Students' performance in the medical error domain was significantly lower than their performance on patient-physician relationship and humanistic issues. Less than 60% of students (n = 54) offered the simulated patient an apology after a medical error occurred. A significant correlation was found between scores obtained in the medical error domains and scores related to both the patient-physician relationship and humanistic domains.

CONCLUSIONS:

An objective structured clinical examination is a useful tool to evaluate patient safety competencies during the medical student clerkship.

INTRODUCTION

Patient safety has drawn the attention of healthcare organizations and the public, especially after the 1999 Institute of Medicine report To Err is Human: Building a Safer Health System, which estimated that as many as 98,000 deaths annually are due to medical errors in the United States.1 Since that report, patient safety has come to the forefront of public health problems. The World Health Organization has estimated that millions of patients worldwide suffer disabling injuries or death every year due to unsafe medical care; therefore, the World Alliance for Patient Safety was launched in late 2004.2

Recently, patient safety concerns have caught the attention of medical educators in postgraduate training and medical practice improvement programs.3 Education, an essential tool for promoting safety in healthcare, mitigates the prevailing name-and-blame culture in healthcare organizations and reinforces teamwork. Furthermore, education promotes the use of errors as learning opportunities.4 Improvements in safety and quality in healthcare require the following changes: the delivery of patient-centered care; effective communication with patients and their relatives; the demonstration of an awareness of medical error occurrence, prevention, management and disclosure; safety while working in interdisciplinary teams; the upholding of medical ethics; the use of evidence-based practices; and finally, the demonstration of a familiarity with information technology and quality improvement strategies.7 Throughout their education, future generations of physicians must be adequately taught, sufficiently trained, and satisfactorily assessed in this new dimension of patient care4. Currently, most medical students receive insufficient training in these areas; as a result, they are not armed with the skills that are required to fulfill their responsibilities.4,6,8,9–12

A growing body of evidence suggests that education on patient safety must begin during undergraduate medical education, preferably during clinical rotations or clerkships, when students are in direct contact with patients.6,13,14 A recent study of the experiences of trainees with patient safety showed that most students were involved with medical errors during clinical rotations. Despite this exposure, the majority considered themselves insufficiently trained in safe practices, medical error recognition and disclosure at graduation.12 Additionally, the students' exposure to medical errors and adverse events negatively affected their attitudes and competencies.12

Patient safety educational programs are generally offered during postgraduate training and medical practice improvement activities. The general educational strategy includes formal lectures and open discussions, which are enhanced by multifaceted training approaches.6–8,10,14 Unfortunately, this approach is seldom incorporated into undergraduate medical education.13–17 Moreover, few studies have focused exclusively on the patient safety programs that are provided during medical school clerkships.12,18–20

In addition to the adequate inclusion of patient safety concerns in medical school curricula, valid and reliable assessments of students' learning outcomes must also be incorporated.3,4,6,7,16,17 The objective structured clinical examination,21 which uses standardized patients, is a powerful tool for the evaluation of patient safety competencies. There are several positive aspects of this tool, including professional development in a low-risk environment and the incorporation of skills and attitudes for handling medical error recognition and disclosure, patient-centered care, communication, and interpersonal skills and humanism issues.4,6,7,10,22 In addition, the objective structured clinical examination provides valuable feedback to both examinees and educators4,6,7,23 and offers reinforcement that is critical for promoting lasting changes in physician behavior.4,6,7,23 Despite the above considerations, the objective structured clinical examination is seldom incorporated as an educational strategy to assess the development of skills and attitudes related to patient safety. The few relevant studies published on this topic have described excellent results from the use of patient safety-based objective structured clinical examinations during postgraduate training and medical practice.7,8,22,24–26 To our knowledge, no previous study has analyzed the use of the objective structured clinical examination to assess patient safety competencies in undergraduate medical training, particularly during the clerkship experience. The objective of the present study was to analyze fifth-year medical student performance in patient safety using an objective structured clinical examination scenario with standardized patients following the introduction of a one-time new patient safety curriculum. Specifically, the assessment focused on medical error recognition and disclosure, the patient-physician relationship and humanism.

METHODS

Subjects and Setting

This study was conducted at the São Paulo University School of Medicine, which admits 180 students annually to a six-year program in undergraduate medical education ranging from primary to tertiary care. The clerkship experience occurs during the final two years of undergraduate education and involves supervised, hands-on training in two university hospitals of increasing complexity: a 258-bed secondary hospital and a 1,200-bed tertiary hospital. Students are divided into small groups and rotate among five main areas: pediatrics, internal medicine, obstetrics and gynecology, surgery, and preventive medicine. The internal medicine clerkship also corresponds to a 12-week program for fifth-year students, during which small groups of students participate in supervised clinical activities in two internal medicine wards and ambulatory clinics. This program includes simulations of medical scenarios that focus on effective interpersonal communication, invasive procedures, and resuscitation. Student performance is evaluated using the following three complementary tools: a supervisor's overall rating, written exams, and an objective structured clinical examination using a standardized patient. Since the objective structured clinical examination was first introduced at our institution in 2002, this tool has been incorporated in almost all clerkship rotations.

The present study enrolled 95 fifth-year medical students who were rotating through the internal medicine clerkship from July to December of 2007. This study was exempt from the need for review by the University of São Paulo School of Medicine Institutional Review Board, but informed consent was obtained from every student.

Patient Safety Educational Program

The patient safety educational program was first introduced into the clerkship training at our institution in 2007 and developed over time into two lectures per rotation. The program focuses on the following factors: medical error definition, epidemiology, and disclosure; near misses; adverse events; patient-centered care; the patient-physician relationship; and ethics. This program has adopted the Institute of Medicine's definitions of adverse events, near misses and medical errors. After attending lectures on these topics, students participate in two additional activities. They were invited to discuss real cases that they had witnessed or in which they had participated. After this discussion, they performed a root cause analysis. The anonymity of the patients and the health professionals involved in these situations was always maintained.

Objective Structured Slinical Examination Elaboration and Design

There were two rotations during the study period. At the end of each rotation, students were evaluated using an objective structured clinical examination with five 10-minute stations that involved standardized patients. One station focused on patient safety and included three domains: medical error, patient-physician relationship attitudes, and humanistic behaviors. The patient safety scenarios and tasks are briefly described in Table 1.

Table 1.

Patient safety stations: scenarios and tasks.

Station	Scenarios	Tasks
Station 1	Patient 1: A 70-year-old male with renal insufficiency and lumbar pain received a non-hormonal, anti-inflammatory prescription during his hospitalization. His renal function progressively deteriorated, and on the fourth day of hospitalization, dialysis was indicated.	Explain to the patient's daughter what happened with her father.
Station 2	Patient 2: A 50-year-old female with community-acquired pneumonia was admitted to the ward overnight. In the same room was another person (Patient 3) with a similar name who also had diabetes. On the following day, Patient 3 received insulin instead of Patient 2 and presented with confusion due to hypoglycemia. She received hypertonic glucose and recovered completely.	Explain this situation to the patient.

The other four stations included the following assessments: 1) a procedure station for the assessment of venipuncture and paracentesis skills; 2) an orientation station to evaluate the student's ability to teach the use of medication (e.g., insulin injection); 3) a clinical station that focused on physical examination skills; and 4) a health promotion station at which students were asked to perform a health promotion consultation on a healthy patient.

Standardized Patients

The standardized patients had previous experience with assessing the communication skills of students. They belong to the administrative staff without contact with the students. All the standardized patients received training in both the scenario that corresponded to their role as a patient and the detailed checklist that was used to assess the students. Subsequently, the scenarios were pilot-tested by the researchers until the standardized patients accurately portrayed the scenarios, reliably scored the previously described domains and adequately completed the checklist.

Patient Safety Checklist

The patient safety checklist was constructed based on a literature review by a group of medical educators and patient safety researchers. It was pilot-tested for use in objective structured clinical examinations with a group of 45 fifth-year medical students (June 2007). The revised checklist contained 21 items grouped into three domains (medical error, patient-physician relationship, and humanistic behavior) (Tables 2, 3, and 4). To summarize, the medical error domain consisted of eight questions on medical error recognition and related patient outcomes, medical error risk factors, human error theory, and medical error disclosure. The patient-physician relationship domain was composed of nine questions concerning expected behaviors and communication skills. Medical student competency in both of the above domains was rated as follows: nonexistent = 0 points; present but insufficient = 50 points; and present and adequate = 100 points. Medical student competency in the humanistic behavior domain was evaluated with five questions using a Likert scale ranging from 1 (definitely disagree) to 5 (definitely agree); the corresponding scores were then converted to percentages. The standardized patients completed one checklist for each student during a 3-minute break between the exit of the previous student and the entrance of the next student.

Table 2.

Patient safety checklist: medical error domain with the related competency and student scores.

Medical error domain	Main skills and attitudes	Mean±SE(Min-Max Values)
Did the student tell you that a preventable adverse event (medical error) occurred during your (your father's) hospitalization?	Medical error recognition	87.23±2.78(0.00−100.00)
Did the student explain to you what type of error occurred and how it will impact your (your father's) health using comprehensible language?	edical error outcomes	94.77±1.75(0.00−100.00)
Did the student sincerely apologize or say that he/she was truly sorry for what occurred?	Apology	57.37±4.79(0.00−10.00)
Did the student identify preventive actions to avoid this error?	Medical error-specific preventive measures	79.79±5.19(0.00−100.00)
Did the student recognize the situations that contributed to the medical error's occurrence?	Medical error risk factors	87.90±2.88(0.00−100.00)
Did the student explain that there was no single person responsible for the error to avoid finger-pointing or blaming attitudes?	Blame-free posture	84.04±4.83(0.00−100.00)
Did the student tell you that the entire staff would be involved in working to minimize adverse consequences, staying beside you (your father)?	Staff involvement	85.26±2.78(0.00−100.00)
Did the student tell you that this situation would be analyzed to avoid similar errors in the future?	Assuring the patient that the case will be reviewed	59.47±4.38(0.00−100.00)
Medical error domain score		77.96±2.21(18.75−100.00)

SE = Standard error; min = minimum; max = maximum.

Table 3.

Patient safety checklist: the patient-physician relationship domain with the related competency and student scores.

Patient-physician relationship domain	Main skills and attitudes	Mean±SE(min-max values)
Did the student introduce him/herself to you before the interview?	Verbal communication	93.68±2.15(0.00−100.00)
During the interview, did the student look you in the eyes and pay attention?	Patient-centered careNon-verbal expressions	91.05±2.11(0.00−100.00)
Did the student respect your perspective?	Patient-centered care	96.32±1.35(50.00−100.00)
During the interview, did the student assume a physical posture of attention?	Non-verbal expressions	97.37±1.15(50.00−100.00)
Did the student use pleasant, patient facial expressions during the interview?	Non-verbal expressions	95.26±1.51(50.00−100.00)
Did the student avoid impatient and/or censoring facial expressions during the interview?	Non-verbal expressions	98.95±0.74(50.00−100.00)
Did the student respect your silence?	Patient-centered care	84.77±3.18(0.00−100.00)
Did the student show an ability to handle conflicts during the interview?	Communication skills	88.42±2.18(50.00−100.00)
Patient-physician relationship domain score		93.16±0.92(55.56−100.00)

SE = standard error; min = minimum; max = maximum.

Table 4.

Patient safety checklist: humanistic behavior domain with the related competency and student scores.

Humanistic behavior domain	Main skills and attitudes	Mean±SE(min-max values)
Did the student show interest in and concern for your thoughts and feelings?	Interest	91.05±1.42(20.00−100.00)
Did the student respect your rights and values?	Respect	92.00±1.37(30.00−100.00)
Did the student help you in a non-humiliating and empathetic way?	Empathetic help	91.90±1.36(20.00−100.00)
Did you feel supported in your distress?	Support	88.53±1.65(20.00−100.00)
Did you feel confident with your (your father's) future health care?	Confidence	87.47±2.00(0.00−100.00)
Humanistic behavior score		90.19±1.41(26.00−100.00)

SE = standard error; min = minimum; max = maximum.

Feedback

All students received the checklist results and detailed written feedback regarding their performance at each objective structured clinical examination station via e-mail. Students were also strongly encouraged to discuss their performance with the objective structured clinical examination coordinators.

Students' Impressions

At the end of the objective structured clinical examination, students were invited to anonymously complete a questionnaire to evaluate the patient safety program and the objective structured clinical examination assessment.

Data Collection and Statistical Analysis

All of the collected information was transferred to a database that had been created specifically for this program. Independent double data entry and comparisons of the two databases were adopted to ensure internal validation. Quantitative variables were described using the mean value, standard error (SE) and minimum (min) and maximum (max) values. The objective structured clinical examination overall score, individual station scores, and patient safety domain scores were calculated and compared. The scores were also evaluated according to student gender. Multiple comparisons were performed using Dunn's method. Correlations were determined using Spearman rank order. A p-value ≤0.05 was considered statistically significant. The data were analyzed using SigmaStat software (SPSS, Chicago, IL).

RESULTS

A total of 95 fifth-year medical students (63 males and 32 females) participated in the objective structured clinical examination. The overall objective structured clinical examination mean score was 85.89±0.66 (mean ± SE). The mean grades for each station were as follows: orientation 90.57±0.67; patient safety 87.59±1.24; procedures 85.53±1.45; health promotion 85.12±1.41; and clinical exam skills 80.97±1.33 (Tables 2, 3, and 4). The comparisons among all but one of these scores showed no significant difference; the clinical exam skills station had a mean score that was statistically significant (p<0.05). Another noteworthy finding was that student performance in the patient safety station had a high mean score that was surpassed only by the score in the orientation station. No significant differences were found between genders in either the global or specific station scores.

Despite the relatively few significant differences in the data, a detailed analysis of the patient safety station data revealed some intriguing results (Tables 2, 3, and 4). When students' patient safety performances were divided into the domains of medical error, patient-physician relationship, and humanistic behavior, the mean scores in the latter two domains were high, at approximately 90 points. Conversely, the medical error domain resulted in the lowest mean score (77.96±2.21) compared to the other two domains (p<0.05).

In the medical error domain, the best performance was obtained for the item “explaining the medical error and its consequence to the patient or relative using comprehensible and colloquial language” (94.77±1.75). Regarding approaches to medical errors, students recognized the systemic chain of errors and the importance of avoiding behaviors related to blame and punishment (84.04±4.83). Also, students performed well in the area of medical error identification by recognizing potential risk factors and preventive measures. Nevertheless, performance related to “apologizing” and “ensuring that the case will be reviewed” resulted in the lowest mean scores in this domain (57.37±4.79 and 59.47±4.38, respectively).

A strong and significant correlation was found between the medical error and humanistic behavior domain scores (Table 5). The medical error domain scores also correlated significantly to the patient-physician relationship scores, albeit at a lower level. Finally, a significant correlation was detected between the scores from the patient-physician relationship and the humanistic behavior domains.

Table 5.

Correlations among the medical error, patient-physician relationship and humanistic behavior domains.

	Humanistic behavior domain score	Patient-physician relationship domain score
Medical error domain score	R = 0.62p<0.01	R = 0.41p<0.01
Patient-physician relationship domain score	R = 0.60p<0.01

R = Spearman rank-order correlation coefficient.

The patient safety curriculum and assessment were highly rated by students (94%), who recognized the importance of learning, discussing, assessing, and receiving feedback on medical error issues during the clerkship.

DISCUSSION

Patient safety is a growing field of medical education. The relationship between the experience of residents and medical error has drawn increasing attention in the medical literature in recent years.7–12,22,24–31 However, few studies have addressed this topic in undergraduate medical education.12–20 Teaching hospitals and medical schools must better incorporate formal curricula on patient safety to provide students with the opportunity to discuss errors, practice related competencies and receive adequate assessment in this area.32

It is important to note that previous research has shown that the first experience of most physicians with medical error occurred during their undergraduate training.32 Nevertheless, most medical students receive ambiguous messages on this topic. In a recent study, White and colleagues described the attitudes and experiences of 629 medical students and 509 postgraduate trainees about medical errors and disclosure to patients.12 Despite only 35% of these students reporting previous education or training in medical error disclosure, as many as 79% reported involvement with cases involving medical errors, which were predominantly minor errors and near misses.12 Martinez and Lo, who analyzed the experience of 142 medical students with medical errors, found that 76% of students reported having witnessed a medical error, and 18% reported having committed a medical error during their clinical training.33 Furthermore, many students disapproved of the senior doctors' attitudes toward medical errors, which were manifested in the doctors' attempts to hide the errors from patients and relatives.33 Both findings reinforce the necessity to teach, train, and assess medical students in patient safety competencies and to provide them with adequate role models.12,33

To our knowledge, this is the first study to describe the performance of medical students in an objective structured clinical examination using standardized patients to teach patient safety during clerkships that included patient safety in the curriculum. The scarce data that have been published on patient safety in undergraduate medical training have predominantly assessed students' experience with medical error using surveys and specific questionnaires that were anonymously completed by students before and after the incorporation of a patient safety program.12,14,15,17–20,33 These survey-based studies are limited by a dependence on students' self-reporting of achievements rather than the application of observational methods that objectively evaluate student performance.17 Moreover, few of these studies have focused exclusively on the clerkship opportunity, despite recognition of the clerkship as the ideal time for teaching and evaluating these skills.12,18–20 Seiden and colleagues concluded that to teach safety in healthcare, students should be provided with the appropriate knowledge of medical error epidemiology and also be effectively trained in the systems approach to medical error, patient-centered care, interpersonal communication and professionalism skills.16 Because assessment drives behavior, Seiden et al highlighted the importance of the incorporation of multifaceted patient safety concerns in routine evaluations during clinical clerkships. Although the ideal approach would be to assess how students handle medical error by observing them in actual clinical settings, ethical concerns prevent this approach.8 Practical evaluations with standardized patients and the objective structured clinical examination are an excellent alternative for the evaluation of patient safety competencies because they are conducted in low-risk scenarios, incorporate issues related to communication, patient-centered care, empathy and medical error disclosure, and provide valuable feedback for examinees and medical educators.4,6,7,10,22,23 The use of standardized patients is extremely effective for evaluating the performance of healthcare professionals. This approach has been adopted for graduate physician assessment,7,8,22,24–26 but to our knowledge, no published data have yet described this experience during clerkships.

In the present study, our students performed well at the patient safety station compared to the other scenarios. Nevertheless, a detailed examination of the results raised intriguing concerns. It is important to note that students performed significantly poorer on all of the medical error issues compared to their performance in the humanistic behavior and patient-physician relationship domains. As previously noted, the only exception was “the ability to disclose medical error using understandable language.” Moreover, “apologizing” and “ensuring that the case would be reviewed” were the issues with the lowest scores (less than 60%). Concerning issues that were exclusively related to “apologizing,” most previous studies on medical error disclosure have exposed the presence of a critical gap between expected attitudes and the manner in which physicians actually behave.9,34–36 Our results are in accordance with previous data from medical error disclosure simulations that describe doctors as seldom apologizing after a medical error occurs (47%) and almost never ensuring that the case will be reviewed to prevent future similar occurrences (8%).8 A survey focusing on residents' changes in practice after their involvement with a medical error showed that only 54% of the residents reported that they had discussed the mistake with their attending physicians. Only 25% of these residents reported that they had disclosed the error to patients, and only 21% reported that they had apologized.31 In addition, these authors demonstrated that accepting responsibility for the mistake and extensively discussing it with supervisors and patients were both predictors of constructive changes in practice on behalf of these residents.31 However, studies describing the experiences of patients (including physicians) who have suffered from a medical error have shown that only one third of them had been offered an apology or had perceived genuine interest in the situation by the hospital staff.37 This finding conflicts with the ethical standards and expectations of professional guidelines and accrediting organizations, all of which dictate that physicians have a responsibility to disclose medical errors to patients34 to enhance transparency in healthcare.12 The failure to provide truthful and compassionate explanations to patients and their relatives after medical errors occur decreases patient trust and satisfaction in healthcare.35,36 Patients and families want to be told about any error that causes patient harm.34,38,39 Furthermore, previous research on medical error and litigation has shown that the absence of a sincere apology and the lack of effective explanations were key factors related to legal claims.34

We observed that our students were highly rated on communication skills and expected humanistic behaviors. These two important competencies have been discussed, trained, and assessed using the objective structured clinical examination throughout the past ten years of our clerkship program. Effective communication among health professionals, patients, and their relatives is essential for ensuring safety in healthcare.34,36,39 Previous studies have shown that communication failure was an important contributing factor in almost 90% of adverse events and incidents.1 Moreover, once an error occurs, effective communication is a fundamental component of the disclosure process.34–36,39 Similar to previous researchers, we believe that interpersonal skills, empathy, and humanism can and should be included in the educational process and incorporated into multifaceted assessments to promote longer lasting change among trainees and physicians.6,7,34,35 The strong correlation between the scores in the humanistic behavior and medical error disclosure domains reinforces the crucial role of compassion and empathy in the externalization of medical errors to patients.8,34–36,39

This study has several important limitations. First, we analyzed the performance of students in one objective structured clinical examination station related to patient safety using only two different scenarios in a single school of medicine. Therefore, our results are limited and cannot be generalized to all medical students or patient safety competency acquisition in general. Second, our patient safety checklist was constructed by a group of medical educators and members from a patient safety group that incorporated content described in previous research. A pilot study was conducted with preceptors and fifth-year medical students during a pilot objective structured clinical examination, and a final review of the tool was performed. Nevertheless, this patient safety checklist is essentially a new instrument that may be limited by its lack of formal validation and reliability testing. Furthermore, there is not always a correspondence between the behaviors observed in structured, practical evaluations using standardized patients and the behaviors that occur in real life. Issues of inter-rater reliability must also be addressed in future studies to reduce inherent bias and subjectivity. In addition, this was a preliminary study in a newly emerging field; we hope that our findings will contribute to future avenues for research.

CONCLUSIONS

In conclusion, although several questions remain unanswered, we provided for the first time the results of a new curriculum and assessment in patient safety attitudes and competencies. The objective structured clinical examination is a useful tool for the evaluation of this new curriculum during medical school clerkships because it represents an opportunity to provide feedback and reinforce positive role-modeling. This approach also presents a unique opportunity to assess the complex interfaces among the dimensions of humanism, patient-centered care, the patient-physician relationship, and patient safety. We hope that the implementation of this interactive patient safety program and the related objective structured clinical examination assessment will transform the current unfortunate scenario characterized by a large proportion of young trainees who enter practice without adequate training in medical error recognition and disclosure.