Structural Equations Model (SEM) of a questionnaire on the evaluation of intercultural secondary education classrooms

Abstract

This research includes the design of a questionnaire for evaluating cultural coexistence in secondary education classrooms (Berrocal, Olmedo & Olmos, 2014; Olmedo et al., 2014), as well as the comparison of its psychometric properties in a multicultural population of schools in southern Spain. An attempt is made to create a valid, reliable and useful tool for teachers to measure conflict situations in the classroom, as well as understanding the nature of the conflict from the point of view of all those involved. The metric aspects show a maximized content and construct validity (Muñiz, 2010) using a Structural Equation Model (SEM) and Confirmatory Factor Analysis (CFA) analysis, checking and modifying its model by Wald and Lagrange indicators (Bentler, 2007), to obtain the most adjusted model to the theoretical and goodness criteria.

Resumen

Esta investigación incluye el diseño de un cuestionario para la evaluación de la coexistencia en las clases de educación secundaria (Berrocal, Olmedo & Olmos, 2014; Olmedo et al., 2014) y la comparación de sus propiedades psicométricas en una población multicultural de centros educativos en el sur de España. Se busca crear un instrumento válido, fiable como instrumento, útil al profesorado para evaluar las situaciones de conflicto en el aula, pero entendiendo la naturaleza del conflicto desde el punto de vista de aquellos que están involucrados en él. Los aspectos métricos muestran una alta validez de contenido y constructo (Muñiz, 2010) empleando un modelo de ecuaciones estructurales (SEM) y un análisis factorial confirmatorio (CFA), para comprobar y modificar su modelo de acuerdo con los indicadores de Wald y Lagrange (Bentler, 2007), para obtener el modelo más ajustado a la teoría y los criterios de bondad.

It is very common to find conflicts in the current school classrooms; it is actually quite normal for schools’ staff to have difficulties in its proper management. These situations lead to behaviours and attitudes which are negative for intercultural coexistence. School life is made up of experiences, values and relations and could be understood as a collective and dynamic construction of human interactions in school communities (Kriesberg, 1998; Bakerman & Maoz, 2005). Therefore, these experiences and values should emerge from an agreement about minimum principles such as respect, understanding, justice or freedom, and made all them from diversity.

This coexistence perspective focused on schools with a wide and diverse cultural contexts, usually determined by geography locations next to border zones, should be understood as “the conditions that serve as the fundamental prerequisites for the evolvement of advanced harmonious intergroup relations […], the very recognition in the right of the other group to exist peacefully with its differences, and to the acceptance of the other group as a legitimate and an equal partner with whom disagreements have to be resolved in non-violent ways” (Bar-Tal, 2004; p. 256).

Strategies for intercultural adaptation have been revelled as the best ways to work conflicts out and improve the schools coexistence (Crawshaw, 2002; Jain, 2012; Lima & Castro, 2005; Tomé et al., 2010). However, a new approach is required, based on teachers and students’ cultural models and demands, to supply intercultural adaptation strategies and empower those which contribute to a positive life improvement (Olmedo, Berrocal & Olmos, 2014).

It is interesting to know the actions which are most widely taken to improve coexistence at school in a specific geographical area; likewise, it is important to analyse which of them have an influence on relevant variables to perform and develop students’ motivation, group cohesion level or frequency of violent incidents, among others. It is not enough to study only the actions written on school planifications or work plans in order to analyse these aspects. It is necessary to have adequate incorporation of information about the educational community’ opinion in relation to what is being actually done inside the classroom, especially from the point of view of students as real witnesses of these incidents.

This paper presents outcomes for validation of this questionnaire about assessment of coexistence in Secondary classrooms. It have been developed and supported on a National Research Project (I+D+I: Assessment and Development of Intercultural Adaptation Strategies to Improve Coexistence in Secondary Education Schools/EDU2010-22130) funded by the Ministry of Education and Science of the Spanish Government. Starting the making process from a wide literature review of studies focused on methodologies or instrument to collect data about coexistence, such as references by Gutiérrez (2002, 2007& 2008) focused on The Third Space development (Expósito, Olmedo, Pegalajar & Tomé, 2014) on educational interventions to resolve classroom conflicts (Moreno Olmedilla, 1998), about social behaviour (Armas, 2007), proposals and strategies to prevent bad behaviours (Ortega, 1999), who offers a vision of what school coexistence is and how it might be addressed.

This revision shows questionnaires as the most widely used instrument to collect data about coexistence opinions or information at mandatory secondary schools (Berrocal, Olmedo & Olmos, 2014; Díaz-Aguado, 2010; Olmedo et al., 2014), based on its efficiency and validity. Also, it is an inexpensive instrument, immediately applicable and that requires a few resources and easily procedures in comparison with other techniques. However, test or questionnaire scores may contain mistakes by multiple reasons that have to be minimized through adequate procedures (Muñiz, 2010). The underlying theoretical background for the process of elaboration of this questionnaire process takes the best theoretical and metric principles of reliability and validity in order to maximize inferences reached from its use.

The main aim of this paper is to build a questionnaire and calculate its reliability and validity through an analysis based on Structural Equation Modelling (SEM) within a Confirmatory Factor Analysis (CFA). The afore mentioned elements enable to contrast the model constructed a priori, establishing the whole set of relations between elements that form it, to give greater flexibility to the researcher in establishing a hypothesis about the construct structure (Bentler, 2007). Therefore, it is a kind of inferential analysis that uses multivariate regression to relate patterns of responses to a set of latent factors not directly observed, but according to the Substantive Theory (Rizopoulos, 2006) that exist in continuous dimensions of the people evaluated. This Structural Equation Model provides some technical procedures and criteria for validation of measurement models under two conditions (González-Montesinos & Backhoff, 2010):

• •

  Conditional independence, understood as a set of latent factors. For instance, skills, attitudes or perceptions, that influences a group of observed variables measured by the questions that make up a scale. The answers are mutually independent, but conditioned by the latent variable which determines them.

• •

  Latent factors can be quantified by a dimensional structure based on a Substantive Theory that postulates the existence of psychological constructs that perform a causal influence on people's responses to a reactive group.

Method

Participants

This study included a sample of 20 selected multicultural schools in the south of Spain, with at least 15% of immigrant students. It is one of the areas with greatest flow of immigration, given the 450 km border between two continents with different ethnicities; Africa and Europe. The procedure included a sample selection by cluster process to select groups/classes with a minimum of 15% of immigrant students. We collected data from the whole amount of 767 students. Of the students, 12.31% were immigrants and 87.68% were autochthonous. These students were 393 females (51.2%) and 374 males (48.8%), and all of them were aged between 11 and 16 years.

Instruments

The instrument for collecting data is Questionnaire for Assessment Coexistence shared experiences in Intercultural Secondary Classrooms (QACISC) for students (Olmedo et al., 2014; Berrocal, Olmedo & Olmos, 2013). It consists on 48 items grouped in six blocks. The first 10-item set is intended for social and demographic identification and the rest of them is a Lickert scale. Scores are encrypted from 1 to 4 (1, “nothing important”; 2, “not important”; 3, “important”, and 4, “very important”) to prevent intermediate values being positive or negative scores on each item.

Results

Items analysis

Table 1 shows all response categories chosen by percentage of students and their standard deviations, all them higher than .87 as a sign of discrimination items. The corrected item-total correlation (r_i-t) is positive on all items, with values between .340 and .724. This indicates that all contribute to gauge what the test measures and further in the same direction.

Table 1.

Descriptive results (QACISC) for sample N=767

Ítem	Scale (%)				Mean	DT	ri-t
	1	2	3	4
1	14.6	41.2	30.4	13.7	2.44	0.903	0.426
2	22.7	49.8	22.9	4.6	2.09	0.795	0.437
3	8.3	18.9	44.9	27.6	2.92	0.890	0.595
4	12.8	22	37.7	27.4	2.79	0.983	0.626
5	15.3	26.2	41.2	17.3	2.60	0.983	0.547
6	16	24.9	37.4	21.6	2.65	0.993	0.579
7	27.4	37.9	23.1	11.5	2.18	0.961	0.379
8	21.4	24.3	27.5	26.9	2.60	1.102	0.581
9	20.9	38.5	31.3	9.3	2.29	0.897	0.530
10	18.9	33.6	32.9	14.6	2.43	0.955	0.527
11	26	38.4	27.5	8.1	2.17	0.907	0.443
12	29.6	33	24.4	12.9	2.21	1.008	0.340
13	11.6	15.9	28.5	44	3.05	1.028	0.468
14	14.1	6.3	12.1	67.5	3.34	1.087	0.626
15	12.1	11.7	22.6	53.6	3.18	1.048	0.597
16	8.9	29.9	43	18	2.71	0.862	0.536
17	11.6	30.9	39.4	18.1	2.65	0.908	0.597
18	13	26.5	35.9	24.6	2.72	0.978	0.627
19	7.8	26.8	43.7	21.7	2.78	0.870	0.593
20	13.7	35.1	38.4	12.8	2.50	0.882	0.541
21	21.4	45.5	27.4	5.7	2.17	0.832	0.488
22	17.7	40.3	30.8	11.1	2.36	0.897	0.559
23	15.8	37.8	35.6	10.8	2.42	0.879	0.545
24	8.2	13.6	32.5	45.7	3.16	0.946	0.678
25	7.3	7.7	16.9	68.1	3.46	0.918	0.700
26	8.7	7.6	15.6	68.1	3.43	0.961	0.658
27	11	5.7	10.3	73	3.45	1.015	0.635
28	9.4	14.2	39.4	37	3.04	0.939	0.689
29	14.6	20.9	37.3	27.2	2.77	1.008	0.674
30	11	5.4	11.9	71.8	3.45	1.005	0.664
31	8	8.2	21.9	61.9	3.38	0.933	0.672
32	6.9	12.1	35.9	45.1	3.19	0.902	0.724
33	8.1	17.9	38.1	36	3.02	0.928	0.660
34	8.3	20.3	38.7	32.6	2.96	0.932	0.621
35	11.2	8.9	34.1	45.8	3.15	0.991	0.669
36	8.5	17.6	35.7	38.2	3.03	0.948	0.689
37	8.5	26.4	37.1	28.1	2.85	0.929	0.551
38	9.7	9.7	32.4	48.2	3.20	0.964	0.644

DT: standard deviation; r_i-t: corrected item-total correlation.

Scale: 1, conduct “nothing serious”; 2, “not serious”; 3, “serious”, and 4, “very serious”.

The scores on skewness/kurtosis indices and the Kolmogorov-Smirnov, of the sample evaluated in each of the statements in the Questionnaire, does not fit a normal distribution (table 2). All items, except 1, 2, 7, 9, 10, 11, 12, 21, 22 and 23, show negative skewness, which means that students tend to score higher values on the scale. Regarding kurtosis indices are negative for all items except: the 14, 25, 26, 27, 30, 31, 32 and 38, indicating that scores cluster are below the normal distribution curve.

Table 2.

Asymmetry and kurtosis and the Kolmogorov-Smirnov Z (QACISC)

Item	Asymmetry (ET=.88)	Kurtosis (ET=.176)	Z	P^*
1	0.166	–0.738	6.722	.000
2	0.379	–0.268	7.531	.000
3	–0.553	–0.383	7.258	.000
4	–0.397	–0.854	6.448	.000
5	–0.234	–0.835	6.842	.000
6	–0.237	–0.972	6.364	.000
7	0.390	–0.815	6.386	.000
8	–0.127	–1.300	5.160	.000
9	0.163	–0.776	6.096	.000
10	0.048	–0.942	5.521	.000
11	0.291	–0.777	6.121	.000
12	0.338	–0.996	5.748	.000
13	–0.735	–0.691	7.250	.000
14	–1.336	0.173	11.222	.000
15	–0.982	–0.393	8.832	.000
16	–0.215	–0.606	6.804	.000
17	–0.159	–0.766	6.339	.000
18	–0.258	–0.939	6.017	.000
19	–0.303	–0.583	6.860	.000
20	–0.048	–0.716	6.226	.000
21	0.271	–0.510	6.987	.000
22	0.171	–0.727	6.466	.000
23	0.043	–0.717	6.008	.000
24	–0.899	–0.193	7.476	.000
25	–1.600	1.376	11.173	.000
26	–1.545	1.075	11.192	.000
27	–1.637	1.139	12.054	.000
28	–0.757	–0.311	6.838	.000
29	–0.392	–0.919	6.508	.000
30	–1.626	1.142	11.825	.000
31	–1.401	0.835	10.133	.000
32	–0.954	0.090	7.370	.000
33	–0.647	–0.475	6.429	.000
34	–0.540	–0.599	6.431	.000
35	–0.993	–0.078	7.312	.000
36	–0.674	–0.515	6.301	.000
37	–0.331	–0.810	6.000	.000
38	–1.040	0.055	7.768	.000

SE: standard error; Z: Kolmogorov-Smirnov Z.

*

Bilateral.

Exploratory factorial analysis

The exploratory factorial analysis is made by an extraction method of calculating the Kaiser-Meyer-Olkin index and Bartlett's test of sphericity (KMO=0.959; x²=703 for 18 578.691 d.f.; P=.000). The result of this initial analysis suggested the existence of five factors explaining 63.283% of total variance; the first one explains the majority of the whole variability.

The factorial construct validity of the questionnaire, among the various methods of orthogonal (varimax, quartimax and equamax) and oblique rotation, is verified through an equamax method that simultaneously simplifies and changes factors. The criterion proposed to determine the required correlation with the variable factor is Comrey (1985) 0.3. It indicates that any variables below this value are less than 10% of their variance in common with the factor, and those which are above are considered to have a weight factor in the component. All this information is shown on table 3 with communalities of all the items above .30, grouped in five factors.

Table 3.

Matrix of correlations between factors (QACISC). Sample, N=767

	Factor 1	Factor 2	Factor 3	Factor 4	Factor 5
Factor 1	1.000	0.358	0.320	0.588	0.439
Factor 2	0.358	1.000	0.507	0.442	0.525
Factor 3	0.320	0.507	1.000	0.514	0.214
Factor 4	0.588	0.442	0.514	1.000	0.385
Factor 5	0.439	0.525	0.214	0.385	1.000

Factor 1: general, psychological and physical violence; Factor 2: structural violence; Factor 3: problems with insubordination and vandalism towards the teacher and classmates; Factor 4: behaviour while in class; Factor 5: school absenteeism and dropout.

The fourth factor (6/837% of the total variance) refers to behaviour towards classwork, and includes items such as p16/ Forgetting homework, p17/ Avoiding work in class, p18/ Not paying attention to the explanations of the teacher, p19/ Talking while the teacher explains, p20/ Forgetting materials, notebooks, p21/ Putting things away too early, p22/ Making insulting comments about the task, p23/ Getting up without permission.

The fifth factor (4.809% of the total variance) refers to behaviour related to school absenteeism and dropout, including items such us p1/ Being late to class, at the entrance, between classes or after breaks, p2/ Ask to be let go to the toilet continuously, or p12/ Leaving the classroom on class changes.

The first factor (25.274% of the total variance) refers to general, psychological and physical violence, and includes (see table 4) items such as p13/ Poor hygiene, p14/ Skipping classes, p15/Frequent truancy, p24/ Systematically disobeying rules and instructions, p25/ Disrespect to the teacher, p26/ Confronting teacher when he/she reproaches, p27/ Threatening the teacher, or p31/ Fighting with a partner (pushing, hitting, threats, assaults).

Table 4.

Matrix factor allocation, commonality and variant (QACISC). Sample, N=767

Item	Factors					Commonality
	1	2	3	4	5
1	.350	–.189	.170	.066	.695	.567
2	.146	.240	–.044	.128	.686	.643
3	.366	.455	–.014	.190	.431	.674
4	.327	.681	.117	.220	.109	.665
5	.094	.762	.087	.269	.084	.512
6	.220	.734	.105	.274	.013	.654
7	–.249	.528	.187	.182	.320	.635
8	.351	.696	.052	.241	.010	.646
9	.030	.729	.159	.186	.163	.530
10	–.057	.711	.154	.283	.174	.471
11	–.130	.557	.154	.144	.376	.339
12	–.261	.415	.276	.051	.437	.708
13	.449	.290	.198	.035	.097	.657
14	.757	.129	.230	.084	.205	.580
15	.693	.063	.198	.091	.334	.647
16	.176	–.055	.146	.596	.446	.700
17	.195	.397	.108	.637	.011	.619
18	.183	.452	.148	.652	–.022	.597
19	.077	.300	.175	.668	.175	.628
20	.103	.104	.135	.702	.266	.557
21	–.166	.200	.189	.604	.420	.525
22	.025	.273	.231	.637	.180	.660
23	.021	.132	.290	.479	.444	.781
24	.522	–.007	.430	.295	.320	.681
25	.768	.115	.328	.207	.154	.816
26	.680	.059	.374	.175	.215	.599
27	.804	.054	.320	.151	.061	.593
28	.338	.128	.418	.305	.406	.796
29	.304	.372	.475	.376	–.024	.686
30	.786	.038	.333	.148	.127	.698
31	.605	–.002	.490	.137	.217	.621
32	.450	.152	.633	.276	.124	.649
33	.222	.177	.678	.270	.153	.703
34	.236	.024	.615	.145	.437	.603
35	.533	.014	.629	.139	.065	.675
36	.311	.214	.623	.287	.146	.643
37	–.001	.074	.670	.158	.415	.567
38	.490	.173	.617	.110	.070	.643
Variance explained (%)	25.274	16.007	11.466	6.837	4.809

Factor 1: general, psychological and physical violence; Factor 2: structural violence; Factor 3: problems with insubordination and vandalism towards the teacher and classmates; Factor 4: behaviour while in class; Factor 5: school absenteeism and dropout.

The second factor (16.007% of the total variance) refers to structural violence and refers to items such as p3/ Noises and shouts in class, p4/ Damage of equipment, p5/ Throwing things in the classroom, p6/ Damage of school supplies, p7/ Eating candy or chew gum, p8/ Listening to music through headphones or mobile devices, p9/ Disorganizing furniture, p10/ Playing in the classroom, or p11/ Lack of order when entering or exiting the class.

The third factor (11.466% of the total variance) refers to problems with insubordination and vandalism towards the teacher and classmates, grouping items related to p28/ Interrupting the teacher with inappropriate questions, jokes and insulting comments; p29/ Ignoring the teacher when he/she is speaking; p30/ Insulting teachers; p32/ Disrespecting classmates; p33/ Taking away, hiding, spoiling, playing with classmates’ materials; p34/ Arguing with peers in class; p35/ Showing sexist behavior; p36/ Hampering peers’ work; p37/ Talking without being given the floor, and p38/ Xenophobic behavior.

These five factors are in accordance with the dimensions defined by Torrego et al. (2006), describing different behaviours of students in the classroom that can generate conflict. The internal consistency, reliability and homogeneity, denotes acceptable outcomes. For instance, the overall Cronbach's alpha coefficient is .717 for the whole sample, obtaining higher scores on some factors (F1, α=.934; F2, α=.895; F3, α=.927; F4, α=.877) and lower in F5 (α=.5221).

Confirmatory factor analysis to verify models by structural equation models

The confirmatory factor analysis aims to improve the questionnaire through its three structural model configutarion, based on a criterion of maximum likelihood according to the multivariate normality criteria of items. Each of these models will be judged overall by a set of indices for adjustment (Jöreskog & Sörbom, 1984). Modifications of the theoretical model a(M1) are generated from analysis of variation of parameter estimates, residual values and modification tests.

• •

  Model (M1): This is a model which originates in exploratory factor analysis as a theoretical model. Parsimonious adjustment PNFI is close to 1 (0.731), and comparative setting indexes CFI, TLI and NFI are 0.809, 0.796 and 0.783, respectively. Although these are good scores, it is necessary to study some other indexes. The most interesting one is the root mean square error of approximation (RMSEA), which is slightly above the critical limit: 0.085. Therefore, the approach has been fitted to the adjusted model.

• •

  Model (M2): This second model arises from the modification of the first model (figure 1) removing inappropriate items or items pruning (Batista-Foguetet et al., 2004). It preserves 25 items and four of five factors from M1: general, psychological, physical, and structural violence; problems with insubordination and vandalism towards the teacher and classmates, and behaviour towards classwork. It is interesting to realize the level of RMSEA and setting indexes CFI, TLI, NFI as well as that of PNFI.

  
  

  Figure 1.

  Comparing parsimonious models 1 and 2.
• •

  Model (M3): The lack of setting verified by x², requires the use of Wald and Lagrange modification indexes (Bentler, 2007) to implement the contrast of the Lagrange multipliers on parameters absent on the model, with an approximate value of their estimates: Lagrange test suggests the introduction of new relations in the model from a new second order factor called coexistence factor, which subsumes the fourth factor. The Wald test in turn suggests eliminating four items and applying relationships between certain latent errors. All that provides a setting model to theory, goodness fitted and set of values CFI RMSEA of 0.940 and 0.073 (table 5 and figure 2).

  Table 5.

  Comparative summary of fit of goodness and specifying model

  Model	CMIN	P	Absolute adjustment indices			Incremental adjustment indexes
  			LO 90	HI 90	RMSEA	PNFI	NFI	CFI	TLI
  Model 1: 5 factors, 38 items	4290.1	0.00	0.087	0.092	0.085	0.731	0.789	0.809	0.796
  Model 2: 4 factors, 25 items	1737.9	0.00	0.081	0.088	0.084	0.780	0.888	0.875	0.870
  Model 3: 4 factors, 1 second order factor, 21 items	834.6	0.00	0.068	0.077	0.073	0.732	0.926	0.940	0.923
  
  

  Figure 2.

  Model 3 is more complex and hierarchical.

Reliability

The reliability of the questionnaire was determined by verifying the internal consistency by a=.954. This index increases with the removal of any item.

Discussion

The theoretical background of this research work by Olmedilla (1998), Ortega (1999), Torrego (2000), Torrego and Moreno (2003), Gutiérrez (2007, 2008), Santos Guerra (2003), or Weapons (2007), provide a model about what coexistence should be like in schools and how it could be addressed. The questionnaire, which was designed with capabilities for its contextual adaptation, facilitates information and helps to improve coexistence and reduce the frequency of violent incidents. But a questionnaire might only be an important tool for scientific research in the field of education, if it is tested on its psychometrical characteristics in order to optimize the extent of interest variables.

The analysis of the items shows a positive total item right correlation (r_i-t) in all items, with values between .340 and .724. The exploratory factor is performed by the extraction method, having previously used the fit of goodness and AFE indicators. There are five factors which explain 63.283% of the total variance, with an Equamax orthogonal rotation and a limit on correlation degree between the variable and the factor proposed by Comrey (1985) of 0.3.

The confirmatory factorial analysis (CFA) is performed through a progressive verification of the three structural equation models, globally judged by a set of fit indexes (Jöreskog & Sörbom, 1984). M1 and M2 have a parsimonious index with PNFI close to 1. M3, which is more complex and hierarchical, is considered a more appropriate setting to the theory on good scores of goodness RMSEA=0.073 and CFI=0.940 (Byrne, 2006; Arbuckle, 1997; Saris & Stronkhorst, 1984; Long, 1990). These structural equations allow assessment of each item and establish workable modifications, sequentially, until arriving at M3, in which there is a second order factor called the coexistence factor, and four factors of the first order: general, psychological, physical and structural violence; problems with insubordination and vandalism to teacher and classmates; bad behaviours while in class; absenteeism and dropouts. These data arise from a sample of students from schools with high levels of immigration, calculating their correlations and specifying a model with exploratory factorial analysis (EFA), thus having been thoroughly demonstrated as to ensure its validity.

The internal consistency, estimated by the Cronbach's alpha index, is considered adequate for each one of factors. However, it should be taken in to account that model modification has been developed on data only from a specific sample. Therefore, capitalization on chance (McCallum et al., 1992) by cross-validation should be studied to extend outcomes beyond the population.