Introduction
Developed over 20 years ago, objective structured clinical examination (OSCE) is an assessment method that is used by a growing number of health institutions, medical schools and scientific societies. Its use has spread throughout nearly all English-speaking countries.1
In Spain, specifically in Catalonia, the Institute of Health Studies of the Catalonian Regional Government (Generalitat) has carried out assessment exercises since 1994 to evaluate graduate and postgraduate trainees in different medical specialties.2,3 This has made it possible to detect gaps in health care skills among both students and practitioners, and to identify changes needed in medical training activities.4
The number of clinical cases that should be included in an OSCE reflects a compromise between validity, reliability and feasibility. The use of 10 or more cases ensures good reliability, but in general most OSCE include approximately 20 cases distributed in 30 stations.5
The specialty designated family and community medicine comprises a wide range of knowledge fields and skills, and family physicians have pioneered efforts to implement innovative training methods such as a specific program for this specialty, continuing education, certification for training activities, and postgraduate training and evaluation.6
Among the criteria proposed for certification and recertification for trainers of family and community medicine residents is the trainer's professional competence. The OSCE has been proposed as a reliable way to evaluate this.7
Few assessment exercises have been carried out in Spain with the OSCE instrument to evaluate trainers of family and community medicine residents, and no such assessments have been attempted in the autonomous community of Andalusia, the region comprising most of southern Spain. We report the first such experience, carried out at the Family and Community Medicine Teaching Unit in the city of Huelva, southwestern Spain. The aim of the exercise was to test the usefulness of the OSCE in evaluating clinical skills and detecting areas in need of improvement, to enable trainers to enhance their own training.
Material and Methods
The test committee was constituted in 2001. The committee consisted of 6 family physicians experienced in clinical practice at primary care centers, and in the training of residents as tutors and course leaders. All committee members had knowledge of and experience in the evaluation of competence. The functions of the test committee were to define components of competence, prepare the list of clinical situations that merited evaluation, construct the table of specific components of competence, and design some of the clinical cases to be used in the evaluation.
The components of competence and weightings, chosen by consensus, were based on recommendations of the Catalonian Society of Family and Community Medicine,8 and were weighted as follows: clinical interview, 12%; physical examination, 12%; interpersonal communication, 16%; technical skill, 10%; management (ability of the participant to reach a correct diagnosis from the information obtained from the clinical interview, physical examination, and complementary tests; knowledge of treatments; ability to devise a treatment plan), 40%; attention to family members, 4%, and preventive activities, 6%.
The choice of clinical situations was agreed by consensus on the basis of the following priority criteria: prevalence, clinical severity, importance of prevention and early diagnosis, complexity of the case, evaluation of the physician's ability to reach a decision, and feasibility of evaluation.9 The most appropriate instrument for evaluation was chosen depending on the clinical situation: standardized patient, mannequin, clinical image interpretation, set of open questions requiring brief answers, or multiple-choice questions. The table of specific components contained the competence areas to be evaluated and their corresponding clinical situation and instrument. Two clinical cases with their script, practical instructions and scoring checklist (along with additional advice) were supplied by the Institute of Health Studies. All other clinical cases were developed by the test committee.
Logistics and psychometric analyses were managed by the staff of the Family and Community Medicine Teaching Unit in Huelva.
The test consisted of 10 clinical situations, and candidates to be evaluated were allowed 6 minutes per station. This period reflects the mean duration of each patient-physician contact at primary care centers in the region of Andalusia. The stations represented a variety of situations commonly encountered in primary care: emergencies, scheduled appointments and minor surgery (Table 1). The exercise was divided into 2 groups of stations (rounds), and no breaks were allowed between stations. The total time allowed to complete the test was 104 minutes.
The OSCE was carried out at the La Orden health center in the city of Huelva on May 16, 2003. This center was chosen because it provided an appropriate physical environment for the test.
Participants were chosen from among trainers of residents at the Family and Community Medicine Teaching Unit in Huelva, and participation was voluntary. Other participants in the exercise were 5 standardized patients, 7 observers and 3 persons responsible for logistics.
The number of stations was set at 10 on the basis of available resources, and because this was the minimum number of stations that yields reliable results.
The highest score possible for each case was 100 points, distributed between the various components of competence to be evaluated. The total score was calculated as the mean of the scores for each case. At the end of the exercise all participants were given a questionnaire to solicit their opinions and feelings about the exercise.
The data were analyzed with SPSS software (version 11.0). Reliability of the test in general and of each component was calculated as Cronbach´s alpha. Correlations of the scores with age and with years as a trainer were calculated as Pearson's correlation coefficient. The results per component of competence were found for each participant, and mean scores and standard deviations were also calculated. The scores per case were also calculated, as were the results for each participant.
Results
Thirteen of the 22 certified tutors at the teaching unit (8 men, 5 women) participated in the OSCE. Mean age was 42.8 years (SD, 3.6 years), and mean duration of teaching and tutoring activities for residents at the time of the test was 8.6 years (SD, 4.6 years). Of the 9 trainers who did not take part in the OSCE (4 men, 5 women), 8 had personal reasons (the exercise was held on a Friday afternoon) and one women was on maternity leave. Mean age of the 9 nonparticipants was 41.2 years (SD, 2.4 years) and mean time as a trainer and tutor was 4.4 years (SD, 2.9 years). There were no significant differences in age between participants and nonparticipants in the OSCE (P=.27), although the difference in the number of years as a trainer and tutor was significant (P=.03).
The global reliability coefficient for the test (Cronbach's alpha) was 0.7287. By component of competence, the highest scores were obtained for preventive activities, interpersonal communication and technical skill (Table 2).
The global mean score for all participants was 73 (SD, 6.2), with a range of 64.4 to 81.9 and an interquartile range of 11.7 (Figure 1).
Figure 1. Distribution of trainers' scores by case.
The best scores were obtained for the components of competence attention to family members, technical skill and interpersonal communication (Table 3). The worst scores were for preventive activities and physical examination, including clinical interview and management. Table 4 shows the mean score and standard deviation for each case.
Test scores and ages of the trainer showed a normal distribution (Shapiro-Wilk's test). Correlations were sought between scores and years as a trainer or age with Pearson's correlation coefficient. Neither correlation was significant.
Most participants (10 of 13) agreed partially or completely that the test and the time allowed for each station reflected conditions encountered in actual practice. Approval was also expressed with the items on the participants' evaluation form that inquired about how the stations were designed, interpretation of the cases, and whether this type of evaluation was useful in improving training for residents. All participants agreed that this type of exercise measured professional capacity better than classical written examinations. The open item on the evaluation form provided space for participants to express emotional features of the test, and elicited comments such as "I was really nervous," "I felt really stressed and a bit nervous," "My self-esteem has hit bottom," "I really liked it," "Overall, I felt comfortable," and "I really enjoyed it."
Discussion
Reliability is the best documented psychometric parameter in evaluations with simulated patients. This indicator is defined as a measure of the reproducibility of a process across a series of replications.10 A minimum of 10 cases are needed to achieve good reliability. In the present study, Cronbach's alpha was similar to values reported in earlier research.2
The global findings for components of competence were satisfactory. However, it was surprising that the skill that yielded the lowest mean score was preventive activities (notwithstanding the large standard deviation), a result that reflected the wide variation in individual scores. This component of competence thus requires improvement. The second lowest mean score was obtained for physical examination skills. This may reflect a loss of skills with time, probably because the patient load leads to simplification of some steps in the physical examination.
On the other hand, the highest scores were obtained for attention to the patient's family (although the weighting of this element in the overall OCSE score was low), interpersonal communication, and technical skill. These results reflect the conscientiousness with which practitioners treat these elements, and possibly the efforts the teaching unit devotes to training the trainers in these areas.
Of the different situations encountered in the exercise, the lowest scores and greatest dispersion were obtained for cases 4 (basic CPR) and 5 (chronic obstructive pulmonary disease). The results for case 4 may reflect the fact that only 4 of the trainers who performed the OSCE worked in rural health centers where a physician is on duty around the clock. This requires them to keep their training in basic life support techniques up to date. In case 5 the greatest weighting was given to the preventive activities component, according to which physicians should give the patient brief counseling about the benefits of quitting smoking. Overlooking this element lowered the score for this case considerably. The stress of the evaluation exercise, and the short time allowed for each case (although the period was a reflection of the time available in real-life conditions) might explain this result. In any case we believe the global outcome and scores for the remaining cases can be considered good.
In general, participants were willing to accept the exercise although some trainers were nervous, and one participant noted that the test had affected self-esteem.
Our findings show that the teaching unit for family and community medicine should make efforts to improve the components of competence that yielded the worst results. Further training will be offered in preventive activities such as counseling and other elements of the motivational interview. Training in physical examination skills will also be improved with appropriate tools, including workshops, CDs, standardized patients, and videotapes of encounters with real patients. These measures will probably lead to improvements in residents´ training.
Objective structured clinical examinations can be useful for evaluating trainers with the aim of enhancing their training in areas where skills are weak. In the not-too-distant future OSCE exercises may serve as a certification and recertification tool for trainers of residents in family and community medicine.
Acknowledgments
We thank the test committee, the trainers who volunteered to participate in this study, Milagros Villamarín Casal for her dedication, Juan Pedro Moreno García and the Huelva-Costa Health District for their assistance, and José Luis Sánchez Ramos for his judicious advice, and the
