Test on the quality of higher education – SABER PRO – What do the results indicate?

The tests on the quality of undergraduate in Colombia, now called SABER PRO, have been suggested as standardized external evaluation instruments that are part of a set of processes undertaken by the National Government in an attempt to assess the quality of public education and do inspection and surveillance.¹ The test is administered to students who are about to complete their college education and is obligatory as a prerequisite for graduation.^2,3

The expectation is that Universities develop learning-focused curricula designed to develop the necessary competencies that the student must achieve to be able to perform in a particular field of knowledge. Consequently, the test is oriented to assess competencies.^1–3

If the test design guarantees appropriate validity and reliability, the results shall become a major input for self-evaluation and self-regulation processes of Universities, providing some guidance on the efficacy and relevancy of the academic syllabus offered. Additionally, those results should provide the foundation for a proper inspection and surveillance.

However, the test results may be associated with various factors, some relating to the institution and the academic syllabus, and others related to the student's social and demographic characteristics.⁴

In their article “Impact of medical schools and students on the results of the national test on the quality of higher education (SABER PRO)”, Drs. Gil, Rodríguez, Sepúlveda, Rondón and Gómez⁵ present an interesting analysis through a hierarchical model evidencing some institution-related factors that may account for the students’ test results, including the type of institution (public vs. private), having its own teaching hospital, number of groups in Colciencias categories A1, A and B and offering medical specialization programs. However, these characteristics only account for 29% of the differences among the various Universities. There are possibly additional variables that were not assessed in the study but are typical of the Institutions and may be related to the results. Likewise, other characteristics of the students that are not included in the SABER PRO registration forms may also account for the differences in the test scores. With regards to the particular overall competencies assessed, it is impossible to establish the impact of the characteristics of the elementary and high school education on the students’ test results.

The information contained in the study by Gil et al.⁵ offers valuable information because it identifies Institution-related factors that are associated with better test results. But even more important than the characteristics identified is the finding that these tests are proportionally of little value when accounting for the differences in the scores of the tests among the various institutions, and hence are not really helpful for government decision-making in surveillance and control. Nevertheless, the analysis of the various components may have contributed to strengthen specific career competencies.

Considering the changes introduced to the test in the last semesters,¹ including a rigorous evaluation of the overall competencies of the students that are about to graduate from college, using specific competency modules according to the particular discipline, it is impossible to make time-based comparisons and to determine the evolution of the test in one particular institution for decision-making purposes.

Probably later on, once the test improvements are consolidated, it will be possible to assess the results and reassess the characteristics associated with the students’ scores, including the variables suggested in the paper by Gil et al., and eventually any other variables identified to determine the significance and the interpretation of the results, to assist both the institutions and the government in making decisions that promote the continuous improvement of the education processes and of the Medical Schools in the country.

There is, however, one additional consideration into the future regarding the need to administer a test that goes beyond the standard written exam and includes the evaluation of competencies in the specific discipline, through clinical simulation scenarios. Written tests, regardless of how well designed they may be, fail to evaluate the development of competencies at higher levels.^6,7

Funding

None.

Conflicts of interest

The author states not to have any conflicts of interest.