The socioformative rubrics in the OSCE to assess the level of achievement of the competencies comprising the profile of the physician graduate

Abstract

Introduction

One of the challenges for teachers in the health area is competency assessment. The OSCE is designed to assess the performance of students in specific clinical situations. The objective of this study was to assess the level of achievement of the competencies proposed in the graduates' profile by applying the OSCE with socioformative rubrics and assessing their relationship with the written final exam and the general average of the degree.

Method

A cross-sectional study was carried out with graduates of the career of medicine; in which an OSCE was applied as part of its degree exam. The rubrics were the instrument that was applied to assess competencies.

Results

The significant differences between the students who applied the OSCE in the month of January compared to May, were also observed in the results of the written exam, which indicates that the students of the 4th. OSCE developed in better way their skills during their training process. The correlational analysis highlights a significant positive correlation between the average of the OSCE and the average of the written exam of the students, which makes it clear that an adequate performance in the written exam that assesses the theoretical knowledge is related to an adequate performance in the clinical skills demonstrated in the OSCE.

Conclusion

The OSCE, is an objective and reliable test, suitable as part of the professional exam of the career of medicine; with the use of the rubric, it is possible to determine the level of achievement with which the student graduates.

Resumen

Introducción

uno de los retos para los docentes en el área de la salud es la evaluación de las competencias. La ECOE se diseñó para evaluar el desempeño de los estudiantes en situaciones clínicas específicas. El objetivo de este estudio es evaluar el nivel de logro de las competencias propuestas en el perfil de egreso a través de la aplicación del ECOE con rubricas socioformativas y evaluar su relación con el examen final escrito y el promedio general de la carrera.

Método

Se realizó un estudio transversal con egresados de la carrera de medicina. Se aplicó una ECOE como parte de su examen de grado. Las rúbricas socioformativas fueron el instrumento que se aplicó para evaluar las competencias.

Resultados

Las diferencias significativas entre los estudiantes que realizaron la ECOE en el mes de enero respecto a mayo, también se observaron en los resultados del examen escrito, lo que indica que los estudiantes del 4to. ECOE desarrollaron mejor sus competencias durante su proceso formativo. El análisis correlacional destaca una correlación positiva significativa entre la media de la ECOE y la media del examen escrito de los estudiantes, lo que deja claro que un desempeño adecuado en el examen escrito que evalúa los conocimientos teóricos se relaciona con un desempeño adecuado en las habilidades clínicas demostradas en la ECOE.

Conclusión

La ECOE, es una prueba objetiva y confiable, idónea como parte del examen profesional de la carrera de medicina. Con el uso de la rúbrica socioformativa se puede determinar el nivel de logro con el que egresa el estudiante.

Introduction

The curricular educational and academic models need to focus on the integral education of the physician by competences. One of the challenges for teachers in the health area is competency assessment, as they tend to evaluate skills, knowledge, and attitudes separately; likewise, it is done with instruments that lack relevance, validity, and reliability. Currently, there are more medical schools that are developing their curricula by competencies, from a sustainable perspective, which refers to update, transform, mobilize, and transfer the knowledge and skills: cognitive, practical, social, affective, and creative to the solution of complex problems. The main characteristic of their curricula is that they are renewed continuously to face the current challenges of health and medical training in the knowledge society through different approaches.^1–3

The socioformation emerged in 2002 from research in Latin America through collaborative work of professionals, researchers, and teachers from various countries seeking a comprehensive training and, in turn, offered a new educational dimension of society using new information technologies and with this, giving meaning to a knowledge society, more reflective, with greater entrepreneurship, addressing problems of the context, and solving the immediate and raised needs.⁴

One of the essential aspects of the socioformation are the competences, because they contribute a new perspective of the education from the challenges of the society of knowledge.⁵

In socioformation, the socioformative evaluation is proposed, which is oriented in the integral formation, articulating the diagnostic assessment, continuous and summative, starting from the self-evaluation, co-evaluation, and hetero evaluation in a collaborative work environment, and for this, it is necessary to use instruments that enable continuous improvement based on the socioformative taxonomy of 5 levels (preformal, receptive, resolutive, autonomous, and strategic). One of the most relevant instruments is the rubric.⁶

The rubrics evaluate the approach of problems of the context, in which knowledge, abilities, attitudes, strategies, and tendencies are put in play that are related to the constructive elements to face and solve the problematics raised, besides it accommodates the metacognition for the improvement from the feedback. Cebrián defines the rubric as a valid tool for the teaching-learning process, which helps to define and explain to students what is expected to be learned.⁷

The Faculty of Medicine innovated its curriculum by competencies, to contribute with the training of physicians committed to solving the current and future problems in the area of health through basic, professional, and specific competencies with sustenance in the educational model by competences focused on the learning. Being focused on learning, the curriculum places the student in a leading position to create and recreate their activities through the use and appropriation of the necessary theoretical-practical instruments. This structure of appropriate procedures for self-directing learning allows students to comprehensively address the health-disease process.⁸

One way to evaluate the clinical competence of physicians is through the Objective Structured Clinical Examination (OSCE), which is designed to assess the performance of students in specific clinical situations, where the students are tested their theoretical knowledge, clinical reasoning, skills, and abilities, as well as their attitude and interpersonal communication skills. It is a timed test, structured as a circuit of stations that resemble clinical scenarios, with specific tasks that allow evaluating the clinical competence of students.⁹ The stations include the presence of an evaluator or a standardized patient, who carries at the moment a performance rating scale, either in the form of a checklist, a perception list, or a rubric.

The objective of this study was to assess the level of achievement of the competencies proposed in the graduates' profile by applying the OSCE with socioformative rubrics and assessing their relationship with the written final exam and the general average of the degree.

Material and methods

A cross-sectional study was carried out with graduates of the career of medicine of the Faculty of Medicine and Biomedical Sciences of the Autonomous University of Chihuahua, in the city of Chihuahua, Chihuahua, Mexico in 2018; in which an OSCE was applied as part of its degree exam.

The professional examination of the career of medicine integrates two evaluations: (1) Written exam of general knowledge of medicine and (2) OSCE where the domains belonging to the competences (basic, professional, and specific) that make up the profile of the physician.

Participants

The study involved 227 graduates, 132 men, and 95 women, belonging to the 2011–2016 generation of the career of medicine of the Faculty of Medicine. There were no criteria for inclusion or exclusion because it is a mandatory exam for all graduates.

Development of the Objective Structured Clinical Examination (OSCE) consisted of the organization of 18 successive stations with clinical cases according to 6 areas of medicine: pediatrics, gynecology, internal medicine, family medicine, emergencies, and surgery; each area was evaluated in 3 stations. In each of the stations, the domains corresponding to the competences that make up the graduation profile were evaluated.

The duration of each station was 6 min, in which the student analyzed each of the clinical situations with a standardized patient, in the case of dynamic stations. In the static stations the student analyzed clinical cases, which for his diagnosis required interpreting laboratory and image studies. A station was also organized to evaluate if students identify solutions based on different reliable sources of information, including the review of the international literature in English.

The OSCE was developed in 4 different dates, this because of the number of participants and the lack of infrastructure to do it in a single application. They were used 2 places and 2 shifts per day, which allowed the evaluation of 52 students in January, 59 in March, 55 in April, and 61 in May 2018 randomly assigned. As for the place, 105 students presented it in classrooms and 122 in a clinical skills laboratory. The time interval of 4 months for each application, was due to the organization of the logistics of each one and the next factors: availability of the evaluators (teachers and residents), the financial resources to develop an examination of this nature, and the infrastructure needed to carry out. To avoid bias in the review, clinical cases were changed at each shift and date.

Regarding to the written exam, it was carried out the same date before the application of the OSCE. Therefore, the students performed the OSCE and the written exam the same day according to their assignation. Also, the written exam was taken from a database of aleatory selected questions.

The competencies that were evaluated were 7 and they integrated a total of 22 domains (performances), which were feasible to evaluate through the OSCE (Tables 1 and 2).

Table 1.

Overall domain averages.

Domain	General average
Applies structured examination (organized and coherent) about the patient's health status.	8.416
Examines the patient in a complete and systematic way, respecting the dignity and integrity of the patient.	8.184
It identifies a probable diagnosis and classifies the pathology in a general and/or specific way.	8.259
Identifies the need to follow up the patient and proposes the type of follow-up that will be carried out according to the pathology found.	8.330
Proposes the treatment or reclassifies and refers to another level of competence.	7.975
Informs the patient and/or relatives about the condition in a complete, clear, friendly, respectful manner, and to their satisfaction.	8.490
Analyses, integrates and contrasts anatomy and physiology, both general and local or regional, with their knowledge of the patient's disease.	8.079
Identifies and uses the procedures, techniques, and instruments used for physical examination.	7.859
Identifies the clinical–pathological alterations of the examined patient.	8.374
Promotes and cares about the doctor-patient relationship.	8.663
Develop verbal ability effectively	8.700
Overcome communication obstacles in conversational exchanges.	8.532
Demonstrates analysis and synthesis skills in various languages	8.438
Use analytical tools in the interpretation of research results and construction of alternatives that allow better decision–making.	8.623
Interacts with different social groups respecting their dignity and the rights of people.	8.680
Identifies the characteristics of the national health system and health care models	8.533
Determine, practice and promote healthy lifestyles.	8.548
Critically analyses the different components of a problem and their interrelationships considering the local, national, and international context	7.770
Identifies solutions based on different reliable sources of information, including the review of the international bibliography (in other languages)	7.762
It relates the elements health and disease with the person and their environment.	9.115
Classify, order and explain the causal factors of the main health problems.	9.036
It promotes the improvement of the quality of human life according to established welfare indicators.	8.097

Own source.

Table 2.

General averages of competences.

Competence	Date				Place		General
	January	March	April	May	Classroom	Clinical skills laboratory
Diagnosis and management of pathologies	8.04	8.0	8.26	8.54	8.25	8.14	8.23
Communication	8.52	8.4	8.34	8.82	8.57	8.49	8.55
Sociocultural	8.57	8.9	8.2	8.95	8.69	8.61	8.68
Health culture	7.78	8.6	8.76	8.36	8.46	8.39	8.38
Troubleshooting	8.16	8.05	7.2	7.57	7.87	7.74	7.75
Basic conceptual elements	9.19	8.98	9.1	9.11	9.12	9.12	9.11
Public health and health system	9.08	8.94	8.9	9.11	9.05	9.01	9.03

Own source.

The evaluation at each station was carried out by professors of basic and clinical sciences, as well as of medical residents using the rubrics as an instrument.

At the end of the OSCE, a survey of the students was carried out in order to evaluate the development of the professional exam through the OSCE; which assessed aspects such as: development and organization of the OSCE, the role of the evaluator during the OSCE, the participation of the student during the OSCE; in addition to a section of opinions and suggestions about the exam. This survey was evaluated using a Likert scale of 0–4.

Instrument

The rubrics are the instrument that was applied to assess competencies. The rubrics were previously validated through a pilot group of 46 Undergraduate Physicians, obtaining a Cronbach Alpha of 0.81. They included 5 levels of performance: Preformal, receptive, resolutive, autonomous, and strategic, which were evaluated from 0 to 10:

• Preformal: Does not own the competence. It has no notions of information (0–3 points).

• Receptive: The student is able to solve the problem if information is available. Thus, his performance is very operational. Requires advice and continuous supervision (4–5 points).

• Resolutive: Simple problems are solved, there are technical elements of the processes involved in the competence. Basic concepts are possessed. Requires advice on specific and complex situations (6–7 points).

• Autonomous: There is autonomy in acting, continuous advice from other people is not required, there is solid scientific argumentation. Problems of various kinds are solved with the necessary elements (8–9 points).

• Strategic: There is creativity and innovation. Strategies of change are proposed. Evolutionary and prospective analyses are done to better address problems (10 points).

Analysis of the information

The information of the evaluations was retrieved in a database to perform the following statistical analyses:

• 1.

  Descriptive analysis: The frequency and averages of the performance of the student was analyzed to identify the level of achievement of each of the competences evaluated in the OSCE by the student.

• 2.

  A correlational analysis was carried out to determine significant relationships between the final average of the OSCE and the general average of the career; as well as between the results of the written exam and the final average of the OSCE.

• 3.

  A comparative study was also carried out between the levels of achievement achieved in each domain and the date on which they performed the OSCE. Through Breakdown & one-way ANOVA Post-hoc Tukey for unequal samples.

Statistical analyses, presenting the data a normal distribution was performed for quantitative variables using descriptive and inferential statistics with P > 0.05, through the Statistical Package for the Social Sciences (SPSS v. 20) and Statistic v. 10 programs.

Results

Regard to the level of achievement of the competences comprising the profile of the physician, of the 7 competences that make up the physician profile, 6 obtained an autonomous level of achievement: Diagnosis and management of pathologies; Communication; Sociocultural; Culture in health; Basic conceptual elements; Public health and health systems. Only the Problem-Solving competence reached a level of resolving achievement (Table 2). These results integrate the analysis of the 4 dates which OSCE develop, where applied the same socioformative rubric. It is important to mention that the sample was divide in different groups because of the size of population.

Concerning to domains of the competencies with higher achievement levels, of the 22 domains evaluated, the one that obtained the highest average was: Relates the health and disease elements with the person and their environment, with an autonomous level of achievement. Instead, the domain with the lowest average was: Identifies solutions based on different reliable sources of information, including the review of the international literature (in other languages), which reached a level of resolutive achievement (Table 3).

Table 3.

Domain averages by date.

JANUARY
General	Average	Higher	Lower
		9.19	7.19
	Domain	Relate the elements health and disease with the person and their environment	Identifies and uses the procedures, techniques and instruments used for exploration
MARCH
General	Average	Higher	Lower
		8.98	7.66
	Domain	Relate the elements health and disease with the person and their environment	Analyzes, integrates, and contrasts anatomy and physiology, both general and regional, with his knowledge of the patient's disease
APRIL
General	Average	Higher	Lower
		9.18	7.89
	Domain	Relate the elements health and disease with the person and their environment	Identifies and uses the procedures, techniques, and instruments used for exploration
MAY
General	Average	Higher	Lower
		9.21	7.51
	Domain	Develop your verbal ability effectively	Critically analyzes the different components of a problem and their interrelationships considering the local, national, and international context

Own elaboration.

Regarding the dates, in the month of May, 12 domains obtained the highest averages, 4 in the month of April, 3 in January, and 3 in March.

A significant positive correlation was observed between the average of the OSCE and the average of the written exam of the students through the Pearson correlation coefficient (P = < 0.01). As well as between the average of the OSCE and the general average of the career. Another important finding is that a correlation was observed between the general average of the career and the average of the written exam (Table 4).

Table 4.

Pearson correlation coefficient.

	Written exam	OSCE final grades	Final career grades
Written exam	1	0.418⁎⁎	0.305⁎⁎
OSCE final grades	0.418⁎⁎	1	0.296⁎⁎
Final career grades	0.305⁎⁎	0.296⁎⁎	1

⁎⁎

. The correlation is significative at 0,01 level (2 tails). Own elaboration.

Through the comparative analysis between the levels of achievement in each domain and the date they performed the OSCE, the results reveal significant differences, mentioning on which date they reached autonomous and / or resolutive levels of achievement. The level of autonomous achievement was the highest in the development of the OSCE and the lowest level was the resolutive one.

In the months of March, April, and May, the students reached an autonomous level of achievement in: Promoting the improvement of the quality of human life attending to established welfare indicators; compared to the month of January, where the students only reached a level of resolutive achievement.

In the months of January, April, and May, the domain: Analyses, integrates, and contrasts the anatomy and physiology, both general and local or regional, with their knowledge about the patient's disease, reached an autonomous level of achievement, instead in March, they achieved a resolutive level.

In April, the highest average corresponds to the domain: Examines the patient in a complete and systematic way, respecting the dignity and integrity of the patient, reaching an autonomous level of achievement as in the months of March and May; however, in January, they only reached a level of resolutive achievement.

During the month of May, the domains: (1) Apply structured interrogation (organized and coherent) on the patient's state of health; (2) identify the clinical-pathological alterations of the examined patient, reached an autonomous level of achievement, and (3) propose the treatment or reclassify and refer to another level of competence; got the highest average in May. However, the third domain reached levels of resolutive achievement in the months of January, March, and April.

The statistical differences observed in the results of the OSCEs between the sessions of January, March, and April in relation to May, were also observed in the averages of the written exam carried out on those same dates (Table 5), which suggests that the students who carried out the OSCE in the month of May developed in better way their skills during their training process. General average obtained by groups according to the dates on which they took the written exam and the OSCE.

Table 5.

General average obtained by groups according to the dates on which they took the written exam and the OSCE.

	January	March	April	May
OSCE	7.44	7.40	7.53	8.12
Written exam	8.13	8.23	8.47	8.63

Source: own elaboration.

Survey analysis: Regarding the survey responses, the results of the means analysis are shown; students believe that clinical cases are consistent with their level of preparation (X¯  = 3.73 ± 3.14), an average that exceeds the upper limit (X¯ + 1σ = 3.656). On the other hand, compliance with the established schedules (X¯ = 2.82 ± 1.21), feedback by the evaluators (X¯ = 2.73 ± 1.15), the promotion of reflection (X¯ = 2.71 ± 1.11) presented values below the lower limit (X¯-1σ = 3.097).

In the opinions and suggestions section, the students felt that the OSCE was an appropriate strategy to evaluate the competences which the physician should graduate; Likewise, they commented that the clinical cases used for the evaluation were concrete and adequate according to the general medicine consultation. They also said that the OSCE allowed them to determine their strengths and the areas in which they should improve. On the other hand, they felt that the time in each station was not enough to adequately develop the skills that were evaluated in each of the stations and suggested that they be offered in advance instructions regarding the organization of the OSCE to avoid confusion when entering each of the stations; They also recommended greater feedback from the evaluator. Another important opinion on the part of the students, is the fact of having simulated patients and mannequins; since it made them nervous and considered that since they are not real patients it is more complicated to determine a diagnosis since there is no real clinical case.

Discussion

This study highlights that the OSCE, being an objective and reliable exam, is ideal as part of the professional examination for the career of Medical due to its flexibility in terms of the number of students that can be evaluated, the type of patients represented, and the format of the exam itself, including the length of the exam, the number and duration of stations. In addition, by using the rubric as an evaluation instrument, it is possible to determine the level of achievement of the competencies with which the students graduate, which allows feedback on their training process.

The socioformative rubrics is an instrument that explains to students what is expected of their professional performance and it has criteria on how it will be assessed, thus promoting reflection and metacognition. Likewise, it provides feedback on achievements and aspects to continue improving. It is considered as a strategy that unfolds the potential of students by providing information on each competence and contributes to improving teaching strategies.⁵

The averages of the students who presented the OSCE in the month of May, were the highest. This correlates with the averages of the written exam carried out on that same date (Table 5). These results indicate that students developed in a better way their skills during their training process. However, according to Callavier et al. (1993), it is also possible that it is due to a “copy effect” in which the followers exchange information with the rest of the participants, a situation in which 3 effects can occur:

1. Positive: It makes the last candidates to take the exam obtain better averages.

2. Negative: The possession of the information causes a negative effect among the applicants, presenting themselves to the exam too confident or with prejudices, which has a downward impact on their averages.

3. No effect: Does not affect the averages; the information that circulates among supporters is usually not relevant to improve their average.¹⁰

Therefore, divide application in different dates can be a limitation for this study.

The organization of the OSCEs, where the clinical cases used in each application were different, allows us to infer that had a greater impact the previous training of the students on the results of the evaluation than the information that circulated among them.

These results are comparable to those of a previous investigation which aimed to present the results of the students of the Faculty of Medicine who applied the practical phase of the professional exam with the OSCE. It was observed that the correlation of the average of the 5 years of the career with the result of the OSCE varied since they have different degrees of development of clinical skills, so that the grades influence other variables, in addition to that they are often overvalued.¹¹

The correlational analysis highlights a significant positive correlation between the average of the OSCE and the average of the written exam of the students, which makes it clear that an adequate performance in the written exam that assesses the theoretical knowledge is related to an adequate performance in the clinical skills demonstrated in the OSCE. It was also found a correlation between the average of the OSCE and the general average of the career, which is of importance since it allows to know if the competencies established in the career profile are related at the time of demonstrating them in the OSCE.

Regarding the surveys carried out on the students, it was observed that they consider to a greater extent that the clinical cases are consistent with their level of preparation. Instead, they consider to a lesser extent that there was compliance with the established schedules, feedback from the evaluators, and promotion of the reflection by the evaluators, which indicates that there are areas for improvement in future examinations; however, this evaluation is carried out by completing the OSCE and the feedback from the evaluator is given to the student in a written format later.

After the end of the circuit of stations, in most of the OSCE, a feedback session needs to be done, where the evaluators review the actions carried out with the student, in order to bring reflection and self-criticism. Getting involved in feedback is mandatory. This is a valuable moment for meaningful learning, since it allows to recognize the student's development during the evaluation, as well as the analysis of their performance, highlighting those actions that were performed correctly and helping the student to discover and understand what to improve and how to do it.¹²

One of the main recommendations of Harden and other authors to improve the validity, reliability, and objectivity of the exam is to be very careful in its design, planning, and administration.¹³

It is also important to use the rubrics, which provide information on the effectiveness of the teaching techniques used during educational activities. They also allow each course and the evaluation process to be analyzed by external peers, which is reflected in a continuous improvement process.¹⁴

This study leads to the conclusion that the OSCE, being a reliable evaluation process, is suitable as part of the professional examination of the medical career, which as previously mentioned, it correlates with the results of other evaluations. Thus, with the use of the rubric, it is possible to determine the level of achievement of the competencies which the student graduates. It also allows for feedback to improve the academic performance of the graduate and the institution.

Therefore, the OSCE is one of the best methodological strategies to assess competencies for their objectivity and compliance with the requirements of validity and reliability. In addition, it is more reliable when applied through rubrics, because they report greater accuracy of the performance levels achieved in each competition, considering the specificity of each clinical situation.

It was determined that the OSCE pertinently assesses the competences established in the graduate profile of the career of medicine which is based on the health problems in the different population groups and the challenges that faces to give them effective solution. In this way, the graduate physician has been able to reach mostly the level of autonomous and resolutive achievement of the domains that integrate the proposed competencies in the physician profile.