Design of a perceived quality scale in orthopaedic surgery education

Abstract

Objectives

The experience of students contributes to proactively identifying the changes necessary in training approaches and activities. The main objective of the Institutional Evaluation Programme was to design and validate a tool that permits discerning the experience of students from traumatology teaching.

Methods

Lecturers from the Orthopaedic Surgery Teaching Unit and experts in quality evaluation methodology, prepared the initial items. In this study, a descriptive analysis was carried out first, followed by an analysis of internal consistency and reliability, construction validity, and predictive validity.

Results

The results (Cronbach's alpha = 0.58, Inter-Item Correlations >0.5, Eigenvalues >0.6, factor loadings, and the Student t-test values) confirmed its reliability and validity.

Conclusions

The developed scale is reliable and valid to assess the experience of students. The use of a tool with these characteristics systematically contributes to improve teaching quality.

Resumen

Objetivos

Las valoraciones del alumnado permiten identificar cambios necesarios en la organización y contenidos de las prácticas clínicas. El objetivo del Programa Institucional de Evaluación fue diseñar y validar un instrumento para capturar sus valoraciones de la enseñanza práctica en traumatología.

Método

Profesores de la unidad docente de Cirugía Ortopédica y expertos en calidad educativa elaboraron una propuesta inicial de dimensiones a explorar, y se diseñó un estudio de campo para determinar consistencia y validez de construcción y predictiva.

Resultados

Los resultados (alfa de Cronbach; 0,58; correlaciones ítem-total: > 0,5; Eigenvalues: > 0,6; saturaciones factoriales y T-test) verifican la consistencia y validez del instrumento.

Conclusiones

Este instrumento permite capturar la experiencia del alumnado con fiabilidad y validez. El uso sistemático de un instrumento como este contribuye a la calidad de la enseñanza.

Introduction

The advantages from having students present in hospitals benefit medical training. However, combining classroom education with hospital activity properly is necessary. In the case of surgical disciplines, having tools that allow learning the students’ opinions as a means for achieving teaching that is more dynamic and tailored to their academic needs is necessary.^1–3 This study's objective was to validate an instrument that assesses the quality perceived by students of traumatology teaching.

Method

Teaching unit

The Teaching Unit of Orthopaedic Surgery began in 1997. The teaching methodology followed in the theoretical classroom corresponds to lectures, while training credits were awarded by work placements in a hospital of 15 sessions lasting 5 h each. Teaching was carried out by integrating the students into all the trauma department activities. The students spent 5 h a day in the hospital, 25 h a week, during three weeks (worth three credits).

Research design

This was a validation of a new instrument study. The consensus conference technique was applied to establish the areas of evaluation for this new instrument and to propose initial items for carrying out this evaluation. First, five teachers from the Teaching Unit of Orthopaedic Surgery and three experts in quality assessment methodology participated. After several rounds, this group produced 32 initial items grouped into six areas. The comprehension of each element was checked with a reduced number of students. Ethical issues of concern were carefully considered and approved by the Department of Orthopaedics at the University of Salamanca.

Analysis

Data were collected for a prior assessment of facial validity, element-by-element analysis (floor or ceiling effects), internal consistency and reliability (using Cronbach's alpha and the Spearman–Brown split-half coefficient), construct validity (using the factor analysis technique of principal components and Varimax rotation), and analysis of predictive validity (using bivariate statistics).

Items were included if they had been answered by at least 85% of the respondents. A point of acceptance for the item-total correlation was established from 0.45 and higher. Furthermore, we considered the eigenvalues of each item on the principal components factor analysis, meaning acceptable values greater than 0.45.

Discriminant validity was calculated using bivariate statistics (t-test and ANOVA). The scale scores were used as dependent variables and the following variables as factors (predictors): the need for additional classes on a regular basis to follow the teaching of the subject matter, consistent studying of the subject matter throughout the course, the extent of the syllabus with respect to its number of credits, and an overall assessment of the Teaching Unit of Orthopaedic Surgery.

Results

In all, 227 students responded voluntarily to the questionnaire. When asked how to improve the scale (face validity), three students suggested including more questions about the practicals and two requested more space for open questions. In the subsequent analysis, 32 items were answered by over 85% of students. 91.2% of the respondents answered affirmatively to the question of whether they considered that the scale allowed assessing the quality of the teaching in the course. Alpha values deleting the scale items one by one ranged between 0.51 and 0.60. The “relationship with the teacher” and “positive rating of the subject contents during the work-placement in various departments” items were central to the scale. The estimated reliability for the whole scale was 0.74 (Table 1; online).

Factor analysis determined the presence of six dimensions with a percentage of variance that accounted for 65%. The item distribution in each dimension according to their factor loadings isolated the following dimensions (Table 1; online): Dimension 1, consisting of items related to the quality/efficiency of the course and the teachers, had the highest proportion of variance, encompassing 18% of the total variability. Dimension 2, consisting of items related to the fulfilment of the program, explained 12% of the total variance. Dimension 3, which accounts for 15% of the variability, groups together items related to tutorials and external help. Dimension 4, the results indicated a percentage of variance that accounted for 9% and contained items related to resources. Dimension 5, consisting of items related to responsibility issues, explained 7% of the total variance. Dimension 6, which accounts for 7% of the variability, groups items related to facilities.

With respect to discriminant validity, Table 1 shows the results of bivariate analysis to test the ability of the extra dimensions to discriminate in the factor analysis. In all cases analysed, the questions of scale achieved an adequate ability to discriminate between subjects who value the teaching positively and those who believe the contrary.

Table 1.

Discriminant validity results.

External criteria	Dimensions	F ANOVA	P-value
Do you consider the teaching of orthopaedic surgery given to be suitable in order to be a general practitioner?	Efficiency	5.2	0.003
	Fulfilment	7.7	0.001
	Resources	4.1	0.009
	Responsibility	2.8	0.044
Consideration of the extent of the topics of the subject matter with regard to the number of credits	Fulfilment	5.8	0.001
	Resources	5.3	0.002

External criteria	Dimensions	t-Test	Sig
Do you consider the overall evaluation of the teaching unit of orthopaedic surgery to be positive?	Efficiency	6.0	0.001
	Fulfilment	12.9	0.001
	Resources	8.1	0.001
	Responsibility	8.8	0.001
Have you attended external complementary classes?	Efficiency	1.1	0.046
	Tutorials	1.6	0.001
Did you consistently study the subject matter during the course?	Responsibility	3.1	0.008

F ANOVA – one-way analysis of variance; t-test – Student t-test.

Discussion

The scale presented has been designed to capture the student perspective in a framework for assessing teaching quality within a university department. Its factorial structure contains 18 items that respond to the dimensions considered most relevant, and its data are consistent, reliable, and acceptable for this type of scale.⁴ Furthermore, the scale's predictive ability can be considered suitable for the purpose it was designed.

There is general consensus that teaching should focus on the student to become more effective.^5,6 This is particularly true for clinical teaching.^7–9 This approach, which stresses the importance of learning, requires a focus on work by teachers and students where interactivity is a basic premise.

Involving students is a necessary part of the practicals that require some tools, among others, and allow us to learn from their views on the teaching.^1–3

This kind of study has some limitations. The first is that students, while respecting the voluntary nature and anonymity of answering, responded within the classroom at the request of their own teachers. The second is that the answers may be influenced by each student's academic performance.

Conflict of interests

The authors report no declarations of interest. The authors alone are responsible for this paper's content and writing.

Acknowledgements

This research was conducted at the University of Salamanca. The authors thank all the participants their collaboration in the study.