Validity and internal consistency of the Maslach Burnout Inventory in dental students from Cartagena, Colombia

Abstract

Objective

To determine the internal consistency and content validity of the Maslach Burnout Inventory-Student Survey (MBI-SS) in dental students from Cartagena, Colombia.

Materials and methods

Scale validation study in 886 dental students from Cartagena, Colombia. Factor structure was determined through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Internal consistency was measured using the Cronbach's alpha coefficient. Analyses were performed using the Stata v.13.2 for Windows (Statacorp., USA) and Mplus v.7.31 for Windows (Muthén & Muthén, USA) software.

Results

Internal consistency was α = 0.806. The factor structure showed three that accounted for the 56.6% of the variance. CFA revealed: χ² = 926.036; df = 85; RMSEA = 0.106 (90% CI, 0.100–0.112); CFI = 0.947; TLI = 0.934.

Conclusions

The MBI showed an adequate internal consistency and a factor structure being consistent with the original proposed structure with a poor fit, which does not reflect adequate content validity in this sample.

Resumen

Objetivo

Determinar la consistencia interna y la validez de contenido del Inventario Maslach para burnout-Encuesta para estudiantes (IMB-EE) en estudiantes de Odontología de Cartagena, Colombia.

Material y métodos

Estudio de validación de una escala en 886 estudiantes de Odontología de Cartagena (Colombia). Se determinó la estructura de factores a través de análisis factorial exploratorio y confirmatorio, y la consistencia interna, con el coeficiente alfa de Cronbach. El análisis estadístico se realizó empleando Stata v.13.2 para Windows (StataCorp., Estados Unidos) y Mplus v.7.31 (Muthén & Muthén, Estados Unidos).

Resultados

La consistencia interna general fue α = 0,806. La solución factorial mostró tres factores que explican el 56,6% de la varianza. El análisis factorial confirmatorio mostró: χ² = 926,036; gl = 85; RCEMA = 0,106 (intervalo de confianza del 90%, 0,100-0,112); ICA = 0,947; ITL = 0,934.

Conclusiones

El IMB muestra adecuada consistencia interna y una estructura factorial coherente con la propuesta original, pero con ajuste pobre, lo cual no refleja adecuada validez de contenido en esta muestra.

Introduction

Burnout is a term that is historically used to refer to a syndrome characterised by emotional exhaustion and cynicism (development of indifference and/or aloofness towards other people), which often occurs in individuals who “work with people” in some way or another. One fundamental aspect of burnout syndrome (BS) is the increased perception of emotional exhaustion, and these two characteristics of BS are related to a certain extent. A third aspect of BS is the tendency to assess oneself in a negative manner, particularly with regard to working with others. Workers feel unhappy with themselves and their occupational achievements.¹

Since healthcare professionals and dental students in particular are included in this category of human care, the intensity of these symptoms may be increased and can affect different aspects of their personal and professional lives. Mafla et al.² determined in 2014 that 7% of dental students in Colombia meet the criteria for BS. Thus, measuring this construct becomes an important aspect for ensuring adequate psychological wellbeing.

The measurement of this construct was originally justified for professionals working in human care institutions. The instrument comprised 25 items and was structured around three main factors (emotional exhaustion, personal fulfilment and depersonalisation), with a fourth optional factor (participation).¹ The fact that BS was initially only identified in human services workers is largely due to an artificial incident resulting from the universal use of the Maslach Burnout Inventory (MBI). Namely, this questionnaire can only be applied to the aforementioned population group due to the fact that it includes dimensions that are defined in terms of interactions with the survey respondents.

Even though the MBI was previously the most universally accepted instrument, there are currently other alternatives such as the Oldenburg Burnout Inventory (OLBI),³ the Burnout Measure (BM)⁴ and the Copenhagen Burnout Inventory (CBI).⁵ The OLBI has two subscales (exhaustion and disengagement) and is based on the argument that the MBI has psychometric limitations due to the fact that the items are all framed in the same direction (positive or negative). The BM was exclusively designed for measuring exhaustion. The items are worded in a general fashion and can thus be applied to any population group. However, studying the psychometric properties of this measure has revealed underlying theoretical issues.⁶ On the other hand, the CBI allows for context-free BS assessment (specific population group).

Due to growing interest in measuring this phenomenon and the early onset of common mental health disorders in young people, and taking into account that health sciences students dedicate at least 40–60% of their time and schedule to professional training and human care, respectively, which can lead to emotional exhaustion and thus a burnout, a version was designed for screening BS in students (MBI-Student Survey [MBI-SS]). This version contains 15 items with affirmative statements, the wording of which was modified in some cases to adapt it to the student context, and is available in various languages, including Spanish.^7,8

In 2013, Rostami et al. assessed the psychometric properties of the MBI-SS among 238 female students from a university in Isfahan. Internal consistency was assessed with the Cronbach's alpha coefficient, according to the subscales, and revealed values between α = 0.84 and α = 0.90; factor structure determined through exploratory factor analysis (EFA) showed three factors: emotional exhaustion, cynicism and academic efficacy. The authors concluded that this instrument presented adequate validity and internal consistency for measuring the construct.⁹ Nevertheless, the study was only conducted on women, and its validity was determined using EFA. It is therefore necessary to report confirmatory factor analysis (CFA) in order to make suitable inferences regarding the content validity of this phenomenon.

Various research papers report the use of the MBI-SS for assessing BS in dental students in different languages. However, in Colombia there are currently no studies that describe psychometric properties including CFA of the MBI-SS in Spanish. As such, the objective of this study is to determine the content validity and internal consistency of the MBI-SS among dental students in Cartagena, Colombia.

Material and methods

Screening scale validation study with no reference criteria. Given that there are only three Dentistry faculties in Cartagena, which is considered a captive, geographic and spatially well-defined population, it was decided to perform a census. In this sense, the students of all three universities completed the instrument anonymously after familiarising themselves with the research objectives, taking into account the ethical considerations described in Resolution 008430 for health research in Colombia.

The sample of this study belongs to another broader study undertaken in Colombia in 2011: The STRESSCODE Study,² led by the Cooperative University of Colombia (Pasto campus), whose protocol was approved by the university's Ethics Committee. However, the dental research committees of the three Dentistry faculties taking part in this study approved the protocol at a later time. Likewise, the participants agreed to participate by signing a written informed consent form.

Population characteristics

All undergraduate dental students from the three Dentistry faculties in Cartagena (Colombia) were invited to participate. The inclusion criteria were: students who agreed to participate voluntarily and who were registered and academically active in the second semester of 2011 and the first semester of 2012 during the conduct of the STRESSCODE Study.

The STRESSCODE Study is a multicentre study performed between 2011 and 2012 on 5647 dental students in Colombia. By applying a battery of tests, it revealed the prevalence of stress, psychological distress, burnout, overall self-regulation and related factors. However due to technical and research-related considerations, this study only describes the validity and internal consistency of the MBI-SS in Cartagena students who met the aforementioned inclusion criteria.

Instrument

The MBI is a scale comprising 15 items expressed in the form of statements that investigate the presence of BS-related elements, such as “I feel emotionally drained by my studies” or “I have become more cynical about the potential usefulness of my studies”, measured on three factors: emotional exhaustion, cynicism and reduced academic efficacy. To apply the scale, the total time required is 5–8 min.

The frequency with which these aspects appear was determined by assessing the score of their Likert-type responses. They vary on a scale of 0 (never) to 6 (daily) and is considered dimensional given that there is no cut-off point. As such, scores may vary from 0 (lowest intensity) to 90 (highest construct intensity).

The Spanish language version used in this study derives from the version used in the STRESSCODE Study, to which a prior transcultural adaptation process was applied, comprising an English-Spanish translation followed by a Spanish–English translation by qualified scientific personnel.²

Statistical analysis

Descriptive statistics were initially performed. Proportions were calculated for the qualitative variables, while the mean ± standard deviation or median [interquartile range] were calculated for quantitative variables, depending on the distribution symmetry. Internal consistency was estimated with the Cronbach's alpha coefficient overall, as well as in each of the subscales (domains).

In order to assess content validity, the factorability of the matrix was determined with Bartlett's test of sphericity and the Kaiser–Meyer–Olkin measure of sampling adequacy. Subsequently, EFA was performed to reveal the factor structure. The number of factors to be extracted was determined using the values themselves and a Cattell sedimentation graph. To define factor conformity, factor loads with a value of ≥0.40 were taken into account. Oblique rotation (promax) was also used, given that it was thought that the potential factors would be correlated with each other.

To proceed with CFA, multivariate normality was verified using Mardia's test. Although multivariate normality was not detected, CFA is robust for these types of scenario when there are large sample sizes (>200) and no multivariate normality.¹⁰ CFA was performed to confirm the factor structure previously determined in the EFA, and thus the content validity through Partial Least Squares with Mean and Adjusted Variance (PLSMAV). To assess the model fit, the following statistical measures were determined: χ², degrees of freedom (df), root mean square error of approximation (RMSEA) and confidence interval of 90% (90% CI), comparative fit index (CFI) and Tucker–Lewis index (TLI). These measures were assessed (model fit) using the criteria proposed by Hu and Bentler: p > 0.05, RMSEA ≤ 0.06, CFI ≥ 0.95 and TLI ≥ 0.95.¹¹

The descriptive statistical analysis and EFA were performed on the Stata package, v.13.2, for Windows, (StataCorp., College Drive Station, Texas, USA); CFA was performed using Mplus v.7.31 for Windows (Muthén & Muthén, Los Angeles, California, USA).

Results

The MBI-SS was applied to a total of 886 subjects; of these, 59.3% were female, 44.1% were undertaking semesters in the clinical academic year, the majority of whom belonged to stratum 3, and 54.39% came from a private university. The mean age was 20.42 ± 2.8 years. Fig. 1 depicts the study entry flow diagram. The study response rate varied from 88% to 100% (mean: 92.5%), while the MBI-SS scores were between 0 and 90. The highest scores on the scale corresponded to means of 5.11 and 4.93, which were found under the “I have learned many interesting things during the course of my studies” and “I feel stimulated when I achieve my study goals” items. The lowest were found under the “I doubt the significance of my studies” and “I have become more cynical about the potential usefulness of my studies” items, with a mean of 1.23. The scores obtained in each domain of the scale are shown in Table 1 By domain, the lowest score corresponded to cynicism.

Fig. 1.

Study flow chart. CFA: confirmatory factor analysis.

Table 1.

Descriptive statistics of each of the reactive components of the MBI-SS scale (Cartagena de Indias, February–August 2011, surveyed students).

Item	Mean ± SD	TISC	ICO
I feel emotionally drained by my studies	3.650 ± 1.800	0.286	0.805
I feel exhausted at the end of a day at university	4.370 ± 1.680	0.102	0.818
I feel tired when I get up in the morning and I have to face another day at university	3.660 ± 1.950	0.336	0.801
Studying or attending a class is really a strain for me	2.490 ± 1.960	0.519	0.788
I feel burned out from my studies	2.250 ± 1.950	0.572	0.784
I have become less interested in my studies since my enrolment at the university	1.310 ± 1.690	0.569	0.784
I have become less enthusiastic about my studies	1.390 ± 1.690	0.618	0.781
I have become more cynical about the potential usefulness of my studies	1.230 ± 1.680	0.515	0.788
I doubt the significance of my studies	1.230 ± 1.820	0.465	0.792
I can effectively solve the problems that arise in my studies	3.980 ± 1.980	0.210	0.810
I believe that I make an effective contribution to the classes that I attend	3.900 ± 1.800	0.310	0.803
In my opinion, I am a good student	4.540 ± 1.640	0.455	0.793
I feel stimulated when I achieve my study goals	4.930 ± 1.590	0.436	0.794
I have learned many interesting things during the course of my studies	5.110 ± 1.500	0.419	0.796
During class I feel confident that I am effective in getting things done	4.700 ± 1.480	0.468	0.792

ICO: internal consistency if the item were omitted; TISC: total item-score correlation; SD: standard deviation.

Content validity

Bartlett's test of sphericity, applied in the EFA, showed χ² = 6924.232, df = 105 and p < 0.001; the Kaiser–Meyer–Olkin measure of sampling adequacy was: 0.8529. Factor structure indicates three factors: factor I was named “emotional exhaustion” and showed an eigenvalue of 3.96, which explained the 26.41% total variance; factor II, named “cynicism”, reached an eigenvalue of 3.45, accounting for the 23% total variance; and factor III, named “reduced academic efficacy”, had an eigenvalue of 1.08, thus explaining the 7.26% total variance. Table 2 how is the factor solution obtained.

Table 2.

Exploratory factor analysis (EFA) results. Item and uniqueness loads (Cartagena de Indias, February–August 2011, surveyed students).

Item	Factor I	Factor II	Factor III	Uniqueness
I feel emotionally drained by my studies	0.7394			0.4330
I feel exhausted at the end of a day at university	0.6909			0.4615
I feel tired when I get up in the morning and I have to face another day at university	0.7030			0.4573
Studying or attending a class is really a strain for me	0.6459			0.3484
I feel burned out from my studies	0.6175			0.3443
I have become less interested in my studies since my enrolment at the university		0.7569		0.3251
I have become less enthusiastic about my studies		0.7724		0.2749
I have become more cynical about the potential usefulness of my studies		0.8075		0.3525
I doubt the significance of my studies		0.6621		0.4991
I can effectively solve the problems that arise in my studies			0.5585	0.6692
I believe that I make an effective contribution to the classes that I attend			0.6812	0.5259
In my opinion, I am a good student			0.8128	0.3477
I feel stimulated when I achieve my study goals			0.7996	0.2757
I have learned many interesting things during the course of my studies			0.7443	0.2771
During class I feel confident that I am effective in getting things done			0.7938	0.3775

The CFA found the following fit indices: χ² = 1137.38, df = 87, RMSEA = 0.117 (90% CI, 0.111–0.123), CFI = 0.934 and TLI = 0.920. Nevertheless, after applying two modification indices, the following fit indices were obtained: χ² = 926.036, df = 85, RMSEA = 0.106 (90% CI, 0.100–0.112), CFI = 0.947 and TLI = 0.934.

Internal consistency

Overall, the scale found α = 0.806 (95% CI, 0.787–0.819). By factors, the measures were represented as follows: factor I, 0.839 (95% CI, 0.825–0.854); factor II, 0.859 (95% CI, 0.839–0.866), and factor III, 0.871 (95% CI, 0.850–0.8744).

Discussion

Various studies have used the Spanish version of the MBI-SS to quantify BS, indicating a prevalence that ranges from 7% to 28%.^2,12,13 However, few studies have explored its validity and internal consistency in Spanish and for populations in Colombia.² In this sense, the results of this study provide the first ever approach to the content validity and internal consistency of this scale, in Spanish, among dental students from Cartagena, Colombia.

In validation studies, it is necessary to take population characteristics into account, especially when the test is applied to the general population; the age and gender of the participants could act as a barrier to generalising the results obtained,¹⁴ due to the fact that the risk of suffering from BS may vary according to these sociodemographic characteristics. This is one of the main reasons for developing the MBI-SS, given that the context to which the general version of the scale was traditionally applied was unsuited to students and implied measurement bias.

In this sense, the results of the study are only applicable to students registered at the three dentistry faculties of the city of Cartagena undertaking semesters in the clinical academic year, from stratum 3, for the most part, and private universities, which limits the application of the instrument to other health areas or to populations with characteristics that differ from those of the sample included here.

The MBI-SS is easy to apply due to the lower number of questions, and the scale used for construct quantitation facilitates quick and reliable measurement. These questions explore the major elements that may define the presence of BS, which is reflected by an adequate internal consistency according to Cronbach's alpha coefficient, calculated overall as well as for the three domains proposed, thus also agreeing with other studies that have validated the MBI-SS in other languages.¹⁰ It is important to take into account that reliability is a necessary step for initiating the validity estimation process, but a reliable instrument is not necessarily valid and, in turn, a valid instrument is not useful if it is not sufficiently reliable, thereby reflecting the need to estimate internal consistency in validation studies in order to express interrelation between construct items.^15–17

The results of this study show that the MBI-SS on undergraduate dental students in Cartagena offers a duplicable factor structure compared to the version that was originally proposed^1,11,18 and studies on the content validity of this scale in other languages, such as Portuguese.¹⁹ The three factors found explain more than half of the variance; this may assert that the construct is present in the items included in the scale. This process is important for gathering data with adequate validity and reliability, and must be performed before beginning any study. However, on evaluating the fit indices provided, the CFA revealed that the model presents poor fit. In this sense, it is necessary to continue investigating some items that in this population do not completely reflect the nature of the construct despite the fact that its factor structure was successfully duplicated in the exploratory factor analysis.

A research study in Isfahan to determine the validity of the Persian version of the scale showed a three-factor structure, but with two fewer items in the cynicism factor, which were included under the emotional exhaustion factor. However, these authors did not perform CFA, which is an essential approach to determine the validity of a scale.^9,20 This discrepancy between the findings can also be explained by the fact that this study only included female students with different academic levels (masters and doctorates), which may signify greater sources of variance in construct measurement.

The justification behind the inclusion of modification indices (MI) in the CFA models is that, by identifying “areas of tension” in the model, statistical strategies may be applied to improve the content validity of the instrument, which is reflected in the improvement of the model fit indices. The modification made to the model is based on the similar wording of the “I feel emotionally drained by my studies” and “I feel exhausted at the end of a day at university” items; moreover, due to belonging to the same domain, these items may share the error covariance. Moreover, this phenomenon could also be observed in the “I have become more cynical about the potential usefulness of my studies” and “I doubt the significance of my studies” items. The usefulness of including the MI may thus be reflected in the statistically significant improvement obtained for the model on comparing the two models in CFA.^10,20

In this sense, in the case of the MBI-SS, two MIs were applied, resulting in a statistically significant improvement in the model fit; however, the scale still presents a poor fit with respect to its content validity. These findings do not match those reported in other studies, which indicate a good fit on employing CFA techniques.¹⁹ The differences lie fundamentally in the criteria used to assess the model fit obtained, which note a CFI value of ≥0.90 and a RMSEA of <0.10.^21,22 Nevertheless, in this research study, stricter assessment criteria were used that were universally consistent with statistical scientific literature on CFA and structural equation modelling, which consider model fit “acceptable” if the RMSEA is ≤0.06, χ² with p > 0.05 and CFI and TLI ≥ 0.95. Although the model had poor fit, the results of the χ² test are expected to be significant due to the fact that its use in CFA techniques is sensitive to sample size; on increasing the sample size, this will tend to be significant. In light of this, other measures of fit, such as the CFI, RMSEA and TLI, must be taken into account for assessing the model.⁶

Measuring BS is important as it permits the early detection of potential cases, thereby attenuating its serious consequences on physical and mental health. In this sense, historically, few instruments have been proposed for measuring this construct, so the MBI is the most used model; more than 90% of the world's research on burnout has been performed with this instrument.⁵ Although various versions have been proposed, including the MBI-SS, with the purpose of improving the detection of the construct in different population groups, numerous authors have noted the marked shortcomings of the instrument's design, which may compromise its results when applied to populations other than the one for which it was originally developed.³ Exhaustion is the central aspect of this syndrome and the MBI, on considering the other two factors, erroneously characterises the concept of burnout given that these dimensions represent a coping strategy and consequence of the syndrome, respectively. This may partially explain the poor fit observed on assessing content validity. Although these hypotheses stem from research studies with the original instrument for professionals, it must be taken into account that the student version is an adaptation.

This created the need to obtain new instruments for measuring BS, such as the Copenhagen Burnout Inventory⁵ and the one-dimensional student burnout scale.²³ The former is intended for measuring the construct among professionals, while the latter is for students (originally designed in Spanish). The one-dimensional student burnout scale still has no scientific evidence on its content validity, but EFA results show that there are two diverging dimensions with respect to the MBI-SS: behavioural indicators and attitude indicators.

These findings highlight the importance of performing more in-depth studies on the content validity of the MBI-SS and of exploring other instruments for this population that may reflect adequate content validity. Future efforts will have to be made to improve this aspect of its psychometric performance, and to thus improve the quality of the measurements performed with the instrument and, as a result, the possibility of generalising the results derived from such research.

Ethical disclosures

Protection of human and animal subjects

The authors declare that the procedures followed were in accordance with the regulations of the relevant clinical research ethics committee and with those of the Code of Ethics of the World Medical Association (Declaration of Helsinki).

Confidentiality of data

The authors declare that they have followed the protocols of their work centre on the publication of patient data.

Right to privacy and informed consent

The authors have obtained the written informed consent of the patients or subjects mentioned in the article. The corresponding author is in possession of this document.

Conflicts of interest

The authors have no conflicts of interest to declare.

Acknowledgements

To the undergraduate students belonging to the research groups, who generously provided their assistance in the data collection process. To the management boards of the University of Cartagena for the academic and administrative support provided during collection of data for this research.