Validation of the Spanish version of Soane's ISA Engagement Scale

Abstract

The interest in the study of engagement in the academic field can be seen through the increasing number of results in Google Scholar and in Scopus, going from barely 20 results between 2000 and 2005 to more than 500 in Scopus and more than 1100 in Google Scholar between 2011 and 2015. Soane et al. (2012) propose a unified theoretical framework as the basis of the psychological mechanism of engagement, grounded on the approach of Kahn (1990). The aim of this paper is to analyze the psychometric properties of the Spanish version of the ISA engagement scale in a sample of 477 employees of the administration and services sector in a Spanish public university. Keeping the original design of the English version of the scale, the proposed factorial structure is validated with the good fit of the data according to the revised goodness of fit indices; reliability and the results of the analysis of construct validity.

Resumen

A través del creciente número de resultados en Google Académico y en Scopus puede verse el interés en el estudio del engagement [compromiso] en el terreno académico, pasando de apenas 20 resultados entre los años 2000 y 2005, a más de 500 en Scopus y más de 1100 en Google Académico entre 2011 y 2015. Soane et al. (2012) proponen un marco teórico unificado como la base del mecanismo psicológico de engagement, con raíces en la aproximación de Kahn (1990). El objetivo de este artículo es analizar las propiedades psicométricas de la versión española de la escala ISA en una muestra compuesta por 477 empleados del sector servicios de una universidad pública española. Probado el buen ajuste de los datos muestrales recogidos según los índices de ajuste revisados, los índices de fiabilidad y los resultados de los análisis de validez de constructo efectuados, se valida la estructura factorial propuesta, manteniendo el diseño original de la versión inglesa de la escala.

May 5th, 2015 passed like any other day, quiet with no major incidents. Newspapers did not announce any national or international news of particular relevance. In a headline published in the digital edition of the provincial newspaper La Voz de Almería that same May 5 we can read in a small entry: “we are strong and at the same time convinced that we will avoid relegation”, words pronounced by a UD Almería footballer, club owned by Alfonso García Gabarrón. Though certainly he did not make express reference to the term “engagement”, once translated into Spanish [compromiso, implicación] it is a good description of the word by its relationship with commitment or implication. Schaufeli (2013) states that it is unknown when “engagement” was used for the first time referred to workplace. However, it is attributed to the Gallup Inc. company in the 90s. This North American organization, based in the District of Columbia and currently chaired by Jim Clifton, made in 1988 a list of “strong” workplaces with a sample of 100,000 employees, using the instrument called Q¹² (formerly, “Gallup's Q¹² employee engagement survey”) (Gallup Inc. (n.d.). This scale, which had its initial development by the professor at the University of Nebraska, Donald O. Clifton between the years 1950 and 1970 and its final draft in 1998, has been reprinted and translated into multiple languages, including Arabic, English, Vietnamese, and Spanish (see Figure 1). The items that make up this scale may be consulted in the paper by Harter, Schmidt, Agrawal, and Plowman (2013) entitled “The Relationship Between Engagement at Work and Organizational Outcomes. Q¹² Meta-Analytics”.

Figure 1.

Example of a results report from Q12 scale Gallup Inc.

The interest in the study of engagement in the academic field can be seen through the increasing number of results in Google Scholar and in Scopus, going from barely 20 results between 2000 and 2005 to more than 500 in Scopus and more than 1,100 in Google Scholar between 2011 and 2015 (see Figure 2). Among the reasons for its academic and professional emergence, Schaufeli (2013) proposes the psychologizing of organizations, being critical the contribution of employees to their jobs and organizations. Human capital is becoming more and more important because of the need to produce more, to obtain more output with less input; in short, it is not surprising that companies and institutions are interested in engagement at a time marked by change not only at organizational levels but also in economic and social fields.

Figure 2.

Number of results from the search for “employee engagement” in Google Scholar and Scopus.

Although the meaning of this construct may seem clear, a review of the literature reveals the vagueness of the concept (Schaufeli, 2013). There is no exact definition of the term. In that same decade of the 1990s in which Gallup Inc. published its study, Kahn (1990), professor at Boston University, in an article entitled “Psychological conditions of personal engagement and disengagement at work”, which appeared in the Academy of Management Journal, defines engagement as “the harnessing of organization members’ selves to their work roles”, presenting it as a construct composed of three facets: physical, cognitive, and emotional, acting simultaneously and creating an “engaged” state. There is empirical evidence supporting this classic conceptualization (e.g., Rich, Lepine & Crawford, 2010). “Engaged” employees are psychologically present, fully “there”, attentive, connected, integrated, and focused on their role performances (Rich et al., 2010). Among the most current definitions, the proposal by Shuck and Wollard (2010) highlights the organizational level, defining engagement as “an individual employee's cognitive, emotional, and behavioral state directed toward desired organizational outcomes”.

Beyond the set of behaviors described above and the different proposals of definition, the conceptualization of the engagement construct is not uniform; Shuck (2011) identifies at least four approaches: Kahn's (1990) need satisfying approach, burnout-antithesis approach as two opposing concepts within a single continuum (Maslach & Leiter, 1997), or as two different concepts but negatively related (Schaufeli, Salanova, González-Roma, & Bakker, 2002); satisfaction-engagement approach, wherein the Q¹² scale of Gallup Inc. is based (Harter, Schmidt, & Keyes, 2002); and Sack's (2006)multidimensional approach to engagement and its evaluation, distinguishing between job engagement (psychological presence in one's job) and organizational engagement (psychological presence in one's organization). The vision proposed by Maslach & Leiter (1997) implies that an employee with high levels of burnout could not present, inevitably, high levels of engagement, even though it would be possible in the approach maintained by Schaufeli et al. (2002). The instrument named UWES (Utrecht Work Engagement Scale), widespread in the academic world and whose technical manual is available in Schaufeli and Bakker (2003), is based on the latter conceptualization. In our country, the aforementioned approach by Schaufeli et al. (2002) has been developed under the broad umbrella of the so-called Positive Psychology, embroiled in the MacGuffin controversy (see, for example, the controversies of Pérez-Álvarez, 2012; Pérez-Alvarez, 2013; Vázquez, 2013), as its defended roots and values (happiness) overflow the consideration as a scientific object, decontextualizing the subjective experience of the individual and transferring it to the masses. In short, it is emphasized the inadmissibility of happiness as a principle of life and as a scientific object (Pérez-Alvarez, 2013). The same publication, American Psychologist Review, which gave birth to Positive Psychology in 2010, published an article entitled “Beyond Positive Psychology? Toward a contextual view of psychological processes and wellbeing” by McNulty and Fincham (2012) and later another article was published entitled “The complex dynamics of wishful thinking: the critical positivity ratio” by Brown, Sokal, and Friedman (2013). Both articles directly attack the principles and assumptions they defend.

The original authors of the ISA Engagement Scale propose a unified theoretical framework as the basis of the psychological mechanism of engagement, with roots in the classical theory of Kahn and whose development is based on three conditions: work-role focus, activation, and positive affect. Soane, Truss, Alfes, Shantz, and Gatenby (2012) describe three dimensions of the construct (intellectual, social, and emotional engagement), which meet the aforementioned three conditions to get the state “engaged” although with a greater focus on social aspects of engagement than is acknowledged in either the Gallup Q¹² or UWES measures of employee engagement. Technical details of the ISA scale can be found in the article by Soane et al. (2012), entitled “Development and application of a new measure of employee engagement: the ISA Engagement Scale”. Currently, since there is no validation of the ISA scale except in samples in English language, it has been proposed this validation which summarizes and gives support to existing regulations, partly motivated by the arrival in Spain of evaluation certificates of people such as ISO 10667 standard performed by AENOR about quality in the evaluation process of people, groups, and organizations approved by the Committee ISO/TC 230 in December 2010 and published in the Official State Gazette of Spain (BOE) dated July 10, 2013.

The aim of this article is just to analyze the psychometric properties of the Spanish version of the ISA Engagement Scale in a sample of employees from the public administration and services and to provide the final version of the scale validated by the original design of Soane et al. (2012).

Method

Sample

To perform the validation, we used a sample of 477 employees from the administration and services sector in a Spanish public university who were assigned to various types of administrative services (maintenance and construction, information and maintenance of official records, support to management, employment service, research administration, administrative support to departments, library, prevention of occupational hazards or economic management, among other areas and services). Since 51 participants did not fully answer the questionnaire or gave the same response (for example, 1) to all the questions, these subjects were removed from the analysis, so that the final sample consisted of 426 subjects (89.3% of the total).

Of these subjects, men make up 49.5% and women 50.5% of the sample (211 and 215, respectively). Regarding age, the most representative age range of participants was between 46 and 55 years of age (53.1%), followed by the range between 36 and 45 years old (32.9%). The most representative academic level was university degree (62.4%), followed by high school or second level of vocational training (25.6%). Most employees had morning shift (88.7%) and the remaining (11.3%), afternoon shift. With reference to their legal status, the sample is divided into civil servants (96.7%) and ordinary employees of the public administration (3.3%).

Instruments

Spanish version of Soane et al.’s ISA Engagement Scale (see Table 1). This scale is based on the development and application of Soane et al. (2012), whose measure includes intellectual engagement, social engagement, and emotional engagement. Each of the above facets represents a dimension of the engagement variable, with three items each. Response options were presented in a Likert scale with 7 categories, ranging from 1 (strongly disagree) to 7 (strongly agree), following the original design of the scale and those with higher scores indicating a higher level of engagement. The psychometric characteristics of the original scale and its factor structure were reviewed in two studies: in the first study the sample consisted of 278 employees of a manufacturing industry and in the second, a total of 683 employees of a retail distribution company. Its design was carried out applying exploratory factor analysis (EFA), principal component analysis (PCA), and orthogonal Varimax rotation (Kaiser, 1974), followed by confirmatory factor analysis (CFA). The results of internal consistency of the original scale, measured through Cronbach's alpha index, were the following: .90 for intellectual engagement, .92 for social engagement, .94 for emotional engagement, and .91 for the overall construct.

Table 1.

The Spanish Version of ISA Engagement Scale.

Engagement intelectual

1. Me centro mucho en el trabajo [I focus hard on my work]

2. Me concentro en mi trabajo [I concentrate on my work]

3. Presto mucha atención al trabajo [I pay a lot of attention to my work]

Engagement social

4. Comparto los mismos valores de trabajo que mis compañeros [I share the same work values as my colleagues]

5. Comparto los mismos objetivos laborales que mis compañeros [I share the same work goals as my colleagues]

6. Comparto las mismas actitudes ante el trabajo con mis compañeros [I share the same work attitudes as my colleagues]

Engagement afectivo

7. Me siento válido y provechoso con mi trabajo [I feel positive about my work]

8. Me siento lleno de energía y fuerza con mi trabajo [I feel energetic in my work]

9. Estoy entusiasmado con mi trabajo [I am enthusiastic about my work]

Note. All items use a Likert scale with response categories from 1 (strongly disagree) to 7 (strongly agree).

Utrecht Work Engagement Scale (UWES) (Schaufeli & Bakker, 2003). This instrument, like ISA Engagement Scale, aims to obtain a measure of engagement. There are several versions of this scale, which has been validated and translated from Dutch into several languages, including Spanish, English, French, Russian, and Finnish, among others. In our sample, the reduced version, UWES-9, was applied, translated into Spanish by Schaufeli and Bakker (2003), with a total of 9 items, each item requiring a 7 point Likert response scale, ranging from 0 (never) to 6 (every day). These items measure the dimensions of vigor, dedication, and absorption, and form the engagement construct. “In my work I feel full of energy” can be found as an example of an item. Schaufeli and Bakker (2003) studied a sample from 9 different countries and found that both unifactorial structure and a three-factor structure remain unchanged. In its Spanish version, the results of internal consistency of the instrument, measured by Cronbach's alpha index, were the following: .72 for the vigor dimension, .84 for the dedication dimension, .77 for the absorption dimension, and .90 for the overall scale. These values, although above the .70 value recommended by Nunnally and Bernstein (1994), are lower than those obtained in the UWES-15 and UWES-17 versions (see Table 2). It should be noted that, as Schmitt (1996) shows in an exemplary way, the assumption of considering as acceptable a certain level of this index (e.g., .70) is not always right.

Table 2.

Results of Internal Consistency of the UWES-9, UWES-15 and UWES-17 Instruments (Schaufeli & Bakker, 2003).

Dimension	UWES-9	UWES-15	UWES-17
Vigor	.84	.86	.83
Dedication	.89	.92	.92
Absorption	.79	.82	.82
Total scale	.93	.92	.93

Note. Data were obtained from a sample of 9,679 subjects in the UWES-9 and UWES-15 scales, and 2,313 subjects in the UWES-17 (Schaufeli & Bakker, 2003).

Performance questionnaire (adapted from Goodman and Svyantek's scale [Goodman & Svyantek (1999)]). This instrument aims to measure the dimensions of in-role performance (e.g., “we reach work objectives”) and extra-role performance (e.g., “we perform functions that are not required but they improve the image of the organization”). The dimensions are measured with a total of 6 items maintaining a Likert scale of 7 response categories, ranging from 0 (strongly disagree) to 6 (strongly agree). The results of internal consistency obtained in our sample were: .92 for the dimension of in-role performance, .86 for the dimension of extra-role performance, and .94 for the overall instrument.

Maslach Burnout Inventory (Maslach & Jackson, 1986) (reduced version adapted to Spanish by Mañas, González-Romá, Peiró, & Subirats (1998). This questionnaire consists of 9 items, which measure the dimensions of emotional exhaustion, reduced personal accomplishment, and depersonalization which form the burnout construct. The response format questionnaire is a Likert scale of 5 categories, ranging from 1, strongly disagree, to 5, strongly agree. An example of this type of items could be “I feel that I have become more insensitive and hard with my colleagues”. The internal consistency results obtained in our sample were the following: .82 for the emotional exhaustion dimension, .80 for the dimension of reduced personal accomplishment, .79 for the depersonalization dimension, and .84 for the overall scale.

Procedure

The first step prior to data collection was to contact the heads of the administration and services in this public university, inviting them to participate in the project. Following the approval and discussion of objectives and in order to facilitate the management of the scale to all staff of the different services, the scale was applied in an online service based on the open source software Lime Survey, providing the required instructions beforehand in order to complete the scale. Participation was voluntary, and confidentiality and anonymity in the treatment of the information was guaranteed with the use of codes.

The Spanish version of ISA scale was performed following the methodological guidelines recommended by the International Test Commission (ITC) and by the European Federation of Psychologists’ Associations (EFPA) (ITC, 2005; Muñiz & Bartram, 2007; Van de Vijver & Hambleton, 1996). In this way, the original instrument was first translated into Spanish by a translator expert on this scientific terminology. The translation was discussed by the members of the research team taking part in the project. They conducted a pretest taken by 10 participants to check that there was no difficulty in the exposition of the items and, after making some corrections to this first translation, a reverse translation was performed by another translator and finally this last translation was compared to the original version of the ISA scale. After final approval, we proceeded to its handling according to the procedure referred to in the preceding paragraph.

Data Analysis

The information was treated by creating a database with IBM SPSS Statistics, version 22. Subsequently, this base was exported to R, version 1.65, for structural equation modeling (Fox, 2006). The following packages were installed: “Lavaan” 0.5-16 (Rosseel, 2012a), “e1071” package for the study of univariate normal, “MVN” for mmultivariate normality, “foreign” for the reading of data from SPSS, and the “q graph” and “semPlot” packages to obtain graphical models.

The factorial structure of the Spanish version of the scale will be determined and validated by using R (Rosseel, 2014). After exporting the SPSS database to R, the confirmatory model (CFA) of the engagement variable was calculated according to the criteria set by Hu and Bentler (1999). An adjustment by the Maximum Likelihood Robust method (MLR), which is suitable for this case, was used (Chou, Bentler, & Satorra, 1991; Jöreskog & Sörbom, 1996).

The reliability of the scale was evaluated using Cronbach's alpha (Cronbach, 1951), both for each dimension of the construct and for the scale as a whole, and the item-total correlations were additionally presented after being corrected.

Finally, a study of convergent and discriminant validity of the instrument was conducted using a correlational analysis between the dimensions of the ISA scale and the performance questionnaire, as well as the Maslach Burnout Inventory, the UWES instrument, and a total of seven linear regression analyses.

Results

As Stanley S. Stevens said in his Handbook of Experimental Psychology, whose first edition dates back to 1951, “When description gives way to measurement, calculation replaces debate” (Stevens, 1951). None of the items met normality criteria. Asymmetry values were between -1.59 and -0.87 (M = -0.7) and kurtosis values between 0.42 and 3.73 (M = 0.30). With reference to the item responses, means of each item were between 5.17 and 6.09 (M = 5.59, SD = 0.99), being 1 the minimum and 7 the maximum.

Subsequently, the model was adjusted by the Maximun Likelihood Robust method (MLR), keeping the original design of the ISA scale on a three-factor model composed of the engagement construct and its three dimensions as latent variables (intellectual engagement, social engagement, and affective engagement). The chi-square statistic was not significant (p-value > .05) and had a successful adjustment. Additionally, the ratio χ²/df = 1.33 was calculated, being thus lower than the value of 2 recommended in Blunch (2008). Based on relative fit indices, CFI (Comparative Fit Index) = .994 and TLI (Tucker Lewis Index) = .991, we get a good fit of the data; both indices can give a result ranging from 0 (poor fit) to 1 (perfect fit), and those results above .900 are considered good results (Hu & Bentler, 1999). Results of absolute fit indices, RMSEA (Root Mean Square Error of Approximation) = .028 (the 90% confidence interval is lower than 0 and higher than .045) and SRMR (Standardized Root Mean Residual) = .026 showed a good fit of the data. Scores below .08 in both indices show a good-fitting model (Hu & Bentler, 1999). Additionally, scores on the original ISA scales are compared to those of the Spanish validation of the scale (see Table 3).

Table 3.

Model Fit Indexes Compared to the Original Scale.

	χ^2	df	χ^2/df	TLI	CFI	RMSEA	SRMR
Spanish version	31.99	24	1.33	.99	.99	.028	.026
Original scale	64.00	24	2.67	^*	.98	.080	.040

Note. df = degrees of freedom; TLI = Tucker-Lewis index; CFI = comparative fit index; RMSEA = root mean square of approximation; SRMR = standardized root mean square residual.

*

GFI (goodness of fit index) was used in the original scale, obtaining a score of .95.

The adjusted model can be seen in Figure 3. All items load to their respective dimensions are between .79 and .93, and the covariances between them range from .45 to .64. Standardized values in R were taken both for latent variables and for observed variables, which is called “completely standardized solution” (Rosseel, 2012b).

Figure 3.

Confirmatory Model (CFA) of Three Factors of Spanish Validation of the ISA Scale.

Note. EI = intellectual engagement; ES = social engagement; EA = affective engagement.

The results of Cronbach's alpha index for each dimension and those of the scale as a whole were the following: .92 in the intellectual engagement dimension, .93 in the social engagement dimension, .91 in the affective emotional engagement, and .91 in the scale as a whole. The corrected item-total correlations are between .63 and .78 (see Table 4), showing adequate levels of homogeneity (Nunnally & Berstein, 1994).

Table 4.

Corrected Item-total Correlations.

	Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Item 7	Item 8	Item 9
Item-total correlations	.65	.66	.63	.72	.73	.69	.71	.78	.78

Note. Corrected item-total correlations of the Spanish validation.

N = 426.

In Table 5 you can see the correlations obtained between the Spanish validation of the ISA scale and the performance questionnaire, the Maslach Burnout Inventory and the UWES instrument, with the aim of testing the convergent and discriminant validity of the scale. At the same time, a total of seven regression analyses were carried out using the dimensions of the ISA scale as predictor variables and the Maslach dimension scales, the performance questionnaire, and the UWES instrument were used as dependent variables. A high correlation is seen in the UWES instrument (between r = .47 and r = .88), mainly in the affective engagement dimension (between r = .77 and r = .88). The highest correlation is also observed with Maslach Burnout Inventory (negative in this case) in the same dimension (between r = -.40 and r = -.58). In the performance questionnaire, the highest correlation is found in the intellectual engagement dimension (r = .55).

Table 5.

Results of the Analysis of Correlation and Regression.

	EI	ES	EA	R^2	ΔR^2
UWES Vigor
r	.58^**	.54^**	.80^**
β	.14^**	.09^*	.66^**	.66	.66
UWES Dedication
r	.58^**	.54^**	.88^**
β	.09^**	.03	.81^**	.77	.77
UWES Absorption
r	.63^**	.47^**	.77^**
β	.27^**	-.01	.61^**	.64	.63
Maslach Depersonalization
r	-.22^**	-.34^**	-.40^**
β	.04	-.16^**	-.33^**	.18	.17
Maslach Emotional exhaustion
r	-.28^**	-.36^**	-.47^**
β	0	-.14^*	-.39^**	.23	.22
Maslach Lack of personal realization
r	-.51^**	-.35^**	-.58^**
β	-.27^**	.02	-.43^**	.38	.37
Performance
r	.55^**	.34^**	.49^**
β	.39^**	.03	.25^**	.34	.34

Note. Bilateral correlations. EI = intellectual engagement; ES = social engagement; EA = affective engagement.

ΔR² = adjusted coefficient of determination.

*

p < 0.05, ^**p < 0.01.

With reference to regression analyses, a high linear relationship can be seen between the dimensions of the UWES instrument and the affective engagement dimension of the ISA scale (β = .66, β = .81, β = .61, in the dimensions of vigor, dedication, and absorption respectively) with high results of the coefficient of determination (both adjusted and unadjusted). Regressions are lower in the dimensions of intellectual and social engagement of the ISA scale, the relationships found between the vigor and absorption dimensions of the UWES instrument with the social engagement dimension not being significant. With reference to Maslach scale, the highest linear regressions (negative) were found again in the affective engagement dimension of the ISA scale (β = -.33, β = -.39, β = -.43, in the dimensions of depersonalization, emotional exhaustion, and reduced personal accomplishment respectively), with the highest coefficient of determination (R² = .38), which was obtained in the last-mentioned dimension. In the performance questionnaire, the highest linear regression was found in the intellectual engagement dimension of the ISA scale (β = .39) with a moderate value in the coefficient of determination (R² = .34).

Discussion

Tested the fit of sample data collected under the revised adjustment indices, reliability indices and the results of validity analysis, the proposed factorial structure is validated keeping the original design of Soane et al.’s (2012) English version of the scale.

After more than 25 years since the publication in 1990 of Kahn's article entitled “Psychological conditions of personal engagement and disengagement at work” we have not yet had in the literature a fully accepted and uniform definition for the concept of engagement. The rapid proliferation of articles and research on the subject in academic, professional, and even informative publications, the different approaches from which can be addressed its study and analysis (remember the four mentioned in Shuck's article (Shuck, 2011), entitled Four emerging perspectives of employee engagement: An integrative literature review), and a long etcetera of factors has somehow caused the lack of a common basis on its study.

Despite the fact that for years there have been data in the academic literature about engagement that demonstrate its influence on other variables such as individual performance (e.g., Alfes, Truss, Soane, Rees, & Gatenby, 2010), the organizational citizenship behavior and task performance (activities that are directly related to the implementation of core labor tasks) (e.g., Rich et al., 2010), or to the intention to leave (e.g., Hallberg & Schaufeli, 2006), we have not found the validation of this scale in non-English language samples in the years following its publication. The importance of engagement, as shown by Rich et al. (2010), is that it predicts a wide range of behaviors, and it may occur that the number of tasks that an employee considers part of his functions in an organization get expanded indistinctly, thus explaining, based on the approximation of Kahn (1990), the link between engagement and performance. There are indeed adaptations to the Spanish language of the UWES scale in its three versions (UWES-9, UWES-15, and UWES-17), but this is based on the approximation of Schaufeli et al. (2002). There are also adaptations of the scale Q¹² Employee Engagement Survey, owned by Gallup Inc. and based on the satisfaction-engagement approach.

The limitations of the validation proposed are focused on a sample which is not representative of the entire population, as it is centered in the public sector and nearly all of the participants, 96.7%, have a labor regime of civil servants. It is required to make generalizations to perform further validation of the scale in other productive sectors, such as the primary (extraction of natural resources) and the secondary (industrial processing activities), as well as in other occupational samples.

In summary, compared to other instruments, given the inclusion of a social component, the ISA Engagement Scale could be a measure more useful for organizations that are concerned about the social aspects of engagement.

Conflict of Interest

The authors of this article declare no conflict of interest.