Measuring the educational environment in ambulatory settings

Abstract

Background and objectives

Students’ perceptions of their educational environment (EE) have been studied in undergraduate and postgraduate curricula. The Ambulatory Care Learning Educational Environment Measure (ACLEEM) is an inventory that was recently developed to measure the EE in postgraduate ambulatory settings. The aim of this study was to assess the psychometric properties of the inventory.

Methods

A mixed methodology was used to develop the ACLEEM including: Grounded theory (8 focus groups); a two-round Delphi technique to identify consensus; and a pilot study. The inventory was refined to 50-items after the pilot study and it was prospectively administered to a large cohort of clerks and residents in Chile during 2010-2011. Psychometric measurements included factor analysis followed by Varimax rotation for construct validity, Cronbach's alpha coefficients for internal consistency and Generalizability theory for test reliability.

Results

Four-hundred and eleven students responded: 151 clerks (83.9% of the target population) and 260 residents (74% of the target population) from 31 postgraduate programs. The factor analyses showed an eight factor instrument. ACLEEM was found highly reliable with a Cronbach's alpha of 0.94 and D-study revealed a reliable outcome for residency programs with at least 15 respondents with a G coefficient of 0.831. The EE perceived by residents and clerks was positive without differences between groups: 152.52 ± 23.36 (76.26%) and 150.61 ± 24.62 (75.30%), respectively (p=0.761).

Conclusions

The 50-item ACLEEM inventory is a multidimensional and valid instrument requiring only 15 respondents for reliable results. We recommend using it to measure the EE in the ambulatory postgraduate Spanish-speaking programs.

Resumen

Introducción y objetivos

Las percepciones de los estudiantes sobre el ambiente educacional (AE) han sido estudiadas en programas de pregrado y posgrado. El cuestionario Ambulatory Care Learning Educational Environment Measure (ACLEEM) es un instrumento desarrollado para medir el ambiente educacional en programas de posgrado ambulatorios. El objetivo de este estudio fue evaluar las propiedades psicométricas de este instrumento.

Métodos

Se utilizó metodología mixta para desarrollar el instrumento ACLEEM, incluyendo: teoría fundada (8 grupos focales) y panel Delfi de 2 rondas (consenso). Se realizó un estudio piloto y luego se refinó el cuestionario a 50 ítems, que fue administrado prospectivamente en una cohorte de internos y residentes chilenos durante los años 2010-2011. Las evaluaciones psicométricas incluyeron análisis factorial seguido de rotación Varimax (validez de constructo), coeficiente alfa de Cronbach (consistencia interna) y teoría de la generalizabilidad (confiabilidad).

Resultados

Se obtuvieron 411 respuestas: 151 internos y 260 residentes (tasa de respuesta: 83,9% y 74%, respectivamente) correspondientes a rotaciones de internado y 31 programas de posgrado en Medicina. El análisis factorial mostró un instrumento de 8 categorías. ACLEEM presentó una alta confiabilidad (alfa de Cronbach 0,94) y un estudio de generalizabilidad que mostró resultados confiables en programas con al menos 15 encuestados (coeficiente G de 0,831). El ambiente educacional percibido por los residentes e internos fue positivo, sin diferencias entre ambos grupos: 152,52 ± 23,36 (76,26%) y 150,61 ± 24,62 (75,30%), respectivamente (p = 0,761).

Conclusiones

El cuestionario ACLEEM de 50 ítems es un instrumento multidimensional y válido, que requiere solo 15 encuestados para contar con resultados confiables. Recomendamos su uso para medir el ambiente educacional en programas de posgrado ambulatorios de habla hispana.

Introduction

Educational environment (EE) can be defined as “the impressions, beliefs and expectations held by members of the school community about their school as a learning environment, their associated behaviour, and the symbols and institutions that represent the patterned expression of behaviour”¹. In other words, the EE is the learning climate within a school seen through the eyes of all of its members. Moreover, it is known that the EE is an important aspect that must be considered in postgraduate medical education. Particularly, its impact is well recognized and accepted because of its real influence over students’ achievement, satisfaction and success^2–4.

In the last years, many new instruments have been developed and validated to evaluate the EE in healthcare professions. Specifically related with postgraduate medical education, Soemantri et al. found 9 instruments that have been used in this level, but none of them have been designed particularly for the ambulatory setting⁵. The World Health Organization in the document “Primary Health Care: Now more than ever” evidenced the importance of the primary health care (PHC), considering it as the most important level in the healthcare systems⁶. This is also supported by scientific evidence showing that health systems with stronger PHC has better health outcomes, reduce inequities and has less healthcare costs^7–9. These are some of the reasons why international agencies such as the Pan American Health Organization have recommended medical schools to orientate their curricula more strongly towards components of PHC¹⁰. This setting is clearly different from the in-hospital location where the teaching characteristics are substantially different¹¹.

Given the agreed relevance of ambulatory care and in the absence of a specific instrument to evaluate the EE in this setting, it becomes an imperative to develop a valid and reliable instrument to measure EE in postgraduate ambulatory medical education. Our team started to carry out this task and the results related to the qualitative research process and the development of the 50-items inventory that we called “Ambulatory Care Learning Educational Environment Measure” (ACLEEM) are published elsewhere¹².

The aims of the present study were to prospectively administer the ACLEEM inventory to clerks and postgraduate residents at the Pontificia Universidad Católica Medical School (PUCMS) and to analyze the psychometric properties (validity and reliability) of the instrument in order to measure the EE in postgraduate ambulatory medical education.

Methods

Instrument

The ACLEEM inventory was developed in a three-stage process that included: grounded theory, a Delphi technique to identify consensus and a pilot study. Three quota samples of approximately 60 stakeholders were formed, one as Focus Groups and 2 as Delphi panels. Stage 1: Eight focus groups were carried out including 58 residents (Latin-American Spanish speakers) from 16 postgraduate programs. The results were analyzed and 173 items were offered in the first round to a National Delphi panel. Stage 2: Sixty-one residents and teachers identified 64 items that were considered important by the panel (>3 points in a 0-4 Likert scale). In the second round, the Delphi panel reduced the number of important items to 54 items. Stage 3: The 54-item inventory was then piloted with 63 residents. The refined version of the ACLEEM consists of 50 items with each one of them scored on a five-point Likert scale with 4 = Strongly agree, 3 = Agree, 2 = Unsure, 1 = Disagree and 0 = Strongly disagree. Because items 24 and 27 contained negative statements, we reverse coded the scores for these questions. Hence, higher scores indicate a more positive result. The details of these stages are published elsewhere¹² (Table 1).

Table 1.

Ambulatory Care Learning Educational Environment Measure (ACLEEM).

1.	Working in the OPC enables me to develop my problem solving skills.
	(El trabajo en el consultorio/policlínico me ayuda a desarrollar mis destrezas para resolver problemas)
2.	The teaching staff in the OPC have good clinical skills.
	(Los profesores clínicos de consultorios/policlínicos tienen buenas destrezas clínicas)
3.	The teaching staff in the OPC are up to date in their knowledge and skills.
	(Los profesores clínicos del consultorio/policlínico están actualizados en conocimientos y destrezas clínicas)
4.	My teachers in the OPC use teaching methods that are appropriate for each subject matter.
	(Mis profesores clínicos en el consultorio/policlínico utilizan metodologías de enseñanza adecuadas para cada uno de los contenidos impartidos)
5.	I feel that my clinical teachers are appropriately qualified to carry out their teaching duties.
	(Siento que mis profesores clínicos están apropiadamente capacitados para realizar sus actividades docentes)
6.	My clinical teaching staff are interested in improving the quality of the teaching activities in the OPC.
	(Mis profesores clínicos demuestran interés en mejorar la calidad de la docencia en el consultorio/policlínico)
7.	I can develop my interpersonal skills in the OPC.
	(Puedo desarrollar mis habilidades interpersonales en el consultorio/policlínico)
8.	I get my evaluations in a timely manner from the teachers in the OPC.
	(Conozco el resultado de mis evaluaciones por parte de los profesores del consultorio/policlínico de manera oportuna)
9.	I feel that the assessment methods used in the OPC are compatible with the teaching methodology.
	(Siento que los métodos de evaluación utilizados en el consultorio/policlínico son compatibles con la metodología de enseñanza)
10.	I have a clear idea about the objectives and learning outcomes of my educational activities in the OPC.
	(Tengo claros los objetivos/logros educacionales de aprendizaje de mis actividades docentes en el consultorio/policlínico)
11.	I feel that the learning objectives and outcomes of the OPC are achieved appropriately.
	(Siento que los objetivos/logros educacionales de aprendizaje del programa de mis rotaciones ambulatorias se cumplen a cabalidad)
12.	I am allowed to participate actively in external educational events and medical meetings.
	(En el consultorio/policlínico me dan las facilidades para participar en eventos educacionales y congresos)
13.	My teachers in the OPC use teaching and learning activities effectively.
	(Mis profesores clínicos del consultorio/policlínico utilizan las oportunidades de enseñanza y aprendizaje en forma efectiva)
14.	The allocated teaching time in the OPC is respected by the clinical teachers.
	(El horario de tiempo protegido para la docencia es respetado por mis profesores clínicos del consultorio/policlínico)
15.	My clinical teachers provide me with feedback about my strengths and weaknesses.
	(Mis profesores clínicos me proveen de retroalimentación (feedback) respecto a mis fortalezas y debilidades)
16.	My clinical teachers are enthusiastic about teaching.
	(Mis profesores clínicos son entusiastas al enseñarme)
17.	Working in the OPCs gives me learning opportunities in a wide variety of diseases.
	(La rotación por distintos consultorios/policlínicos me permite aprender una gran variedad de enfermedades)
18.	In the OPC I learn to treat patients with conditions that are specifically related to ambulatory care.
	(En el consultorio/policlínico aprendo a tratar enfermedades específicamente ambulatorias)
19.	My clinical teachers in the OPC appropriately emphasise the doctor-patient relationship.
	(Mis profesores clínicos del consultorio/policlínico ponen el énfasis apropiado en la relación médico-paciente)
20.	In the OPC I learn from the experience of my clinical teachers.
	(En el consultorio/policlínico aprendo de la experiencia de mis profesores clínicos)
21.	My clinical teachers are good professional role models for me.
	(Mis profesores clínicos son buenos modelos profesionales para mí)
22.	The clinical facilities in the OPC are suitable for working with patients in my specialty.
	(Los box de atención de los consultorios/policlínicos son adecuados para la atención de los pacientes de mi especialidad)
23.	I have the opportunity to follow up my patients appropriately in the OPC.
	(Tengo la oportunidad de realizar seguimiento a mis pacientes del consultorio/policlínico)
24.	I have insufficient time with each patient in the OPC.
	(El tiempo que tengo para la atención ambulatoria de cada paciente es insuficiente)
25.	My activities in the OPC are clearly programmed.
	(Mis actividades en el consultorio/policlínico están claramente programadas)
26.	I am able to refer my patients for evaluation by multidisciplinary teams.
	(En la atención ambulatoria puedo derivar a mis pacientes para evaluación por equipos multidisciplinarios)
27.	There are insufficient clinical supervisors for the number of residents in the OPC.
	(Los supervisores clínicos son insuficientes para el número de residentes que trabajamos en el consultorio/policlínico)
28.	I can obtain clinical supervision when I need it.
	(Puedo obtener supervisión clínica en el consultorio/policlínico cuando lo necesito)
29.	I feel that I have the appropriate level of responsibility for my patients in the OPC.
	(Siento que tengo el apropiado nivel de responsabilidad con mis pacientes del consultorio/policlínico)
30.	I feel that my clinical supervisors consider my opinions in clinical decision making about my patients.
	(Siento que mis supervisores clínicos consideran mis opiniones en la toma de decisiones respecto a mis pacientes)
31.	I feel that I treat my patients in the OPC according to the treatment protocols for their conditions and illnesses.
	(Siento que trato a mis pacientes del consultorio/policlínico de acuerdo a los protocolos clínicos existentes para el manejo de sus enfermedades)
32.	I am able to learn the required practical procedures in the OPC.
	(Tengo oportunidades para aprender los procedimientos prácticos requeridos en el consultorio/policlínico)
33.	I feel that I am learning to become confident in my speciality in the OPC.
	(Siento que la enseñanza que recibo en el consultorio/policlínico me ayuda a desarrollar las competencias relacionadas con mi especialidad)
34.	I feel that the clinical rotations in the OPC are preparing me properly for my professional future.
	(Siento que me preparan adecuadamente en las rotaciones ambulatorias para mi futuro ejercicio como profesional)
35.	In the OPC I manage clinical problems taking into account the social and emotional aspects of my patients.
	(En el consultorio/policlínico manejo los problemas clínicos considerando los aspectos sociales y emocionales de mis pacientes)
36.	I am able to learn to adjust my work to the resources available in the OPC.
	(Aprendo a trabajar adaptándome a los recursos disponibles en el consultorio/policlínico)
37.	I am able to carry out health education activities in the OPC.
	(Tengo la posibilidad de realizar actividades de educación en salud en el consultorio/policlínico)
38.	I feel that my time in the OPC is preparing me to address the health needs of the country.
	(Siento que en el consultorio/policlínico me preparan para atender las necesidades de salud del país)
39.	The teachers in the OPC respond to my personal concerns appropriately.
	(Cuento con el apoyo del profesor encargado de la rotación ambulatoria para resolver mis problemas personales)
40.	The workload allows me to balance the clinical care of my patients with my educational activities.
	(La carga asistencial me permite compatibilizar la atención de mis pacientes con las actividades docentes)
41.	I can keep my work and personal life in balance when I am working in the OPC.
	(Puedo compatibilizar el trabajo con mi vida personal cuando trabajo en el consultorio/policlínico)
42.	My working hours in the OPC permit adequate rest and eating times.
	(Mi jornada laboral ambulatoria considera tiempos de descanso y alimentación adecuados)
43.	I feel part of the team in the OPC.
	(En el consultorio/policlínico me siento parte del equipo de trabajo)
44.	I receive support from other OPC residents when I need it.
	(En el consultorio/policlínico recibo apoyo de otros residentes cuando lo necesito)
45.	I feel that other members of the healthcare team are willing to help me when I need it.
	(Siento que otros miembros del personal de salud tienen buena disposición a ayudar cuando los necesito)
46.	I have adequate access to computers and Internet in the OPC.
	(En el consultorio/policlínico cuento con acceso adecuado a computadores con conexión a internet cuando lo requiero)
47.	The OPC provides lockers to keep my personal belongings safe.
	(Los consultorios/policlínicos disponen de un lugar seguro para guardar mis pertenencias)
48.	There are adequate bathroom facilities in the OPC.
	(En los consultorios/policlínicos cuento con un baño adecuado para el uso de los residentes)
49.	The OPCs have adequate supplies and instruments to render quality professional care.
	(En el consultorio/policlínico cuento con los insumos e implementos necesarios para prestar una atención de calidad)
50.	The clinical files and/or information systems of the OPC give me adequate access to patient information.
	(Las fichas clínicas y/o sistema informático del consultorio/policlínico me permiten un acceso adecuado a la información del paciente)

Subjects and procedure

The inventory was administered to clerkships (students in their last 2 years of medical school) and residents of 31 different programs of the PUCMS (family medicine, psychiatry, paediatrics with their sub-specialties, surgery with their sub-specialties, orthopaedics, dermatology, otorhinolaryngology, ophthalmology, neurology and internal medicine with their sub-specialties), during 2010 and 2011. The ambulatory facilities were primary and secondary care teaching clinics owned and/or managed by the PUCMS. The instrument was administered by members of the Center of Medical Education of the PUCMS with the support of the Postgraduate Department and Directors of each program. The project was approved by the Ethics Committee of our institution. The inventory was administered in both printed and online formats. Response was taken as informed consent and the results were anonymized.

Statistical analyses

To evaluate the construct validity of the 50-item questionnaire, we used an exploratory factor analysis followed by a Varimax rotation. Factors were chosen using the following 2 criteria: 1) the Kaiser-Guttman criterion, in which all factors with an Eigen value >1 were included^13,14; and 2) the Cattell criterion, where the inflexion point of the scree plot curve is the cut off, and all factors above are accepted¹⁵. Data were analyzed by the SPSS statistical program.

We calculate Cronbach's alpha to test internal consistency¹⁶. In addition, we used Generalizability theory (G-theory) to test the reliability of the questionnaire as a complement of the Cronbach's alpha¹⁷. This theory allows estimation of the size of the relevant influences that affect the measurement. The subsequent estimation of the reliability of the instrument is based on a variety of reliability indices. To perform G-theory, and because the number of students by gender, year and level of study was different, we used the Program urGenova¹⁸. For the D-study (Decision Study) we used the Genova program as well. Results were expressed as mean ± standard deviation of the mean. Data between groups (clerks and residents) were analyzed by a 2-tailed t test. A p value of less than 0.05 was considered statistically significant.

Results

The inventory was responded by 151 clerks (83.9% of the target population) and 260 residents (74% of the target population) of which 46.72% were female and 3.9% were non-Chilean nationals. The respondents had a mean of 27.41 years of age. The women's mean age was 27.66 (CI 95%, 27.45-28.75) and for men was 27.22 (CI 95%, 27.01-28.31), and the difference was non-statistically significant (p=0.41). No difference was found between Chilean and non-Chilean respondents.

Construct validity and internal consistency

Construct validity of the subscales was performed using the exploratory factor analyses followed by Varimax rotation of the data and resulted in 11 factors with an Eigen value > 1. The first factor had an Eigen value of 14.479 (accounting for 28.96% of the variance) and the next 10 factors had Eigen values > 1.037. The 11 factors together explained 59.88% of the variance. The inflexion point of the curve was observed between factors 8 and 9 in the 11-factor scree plot (fig. 1). Therefore, we forced a factor analyses with 8 factors. Distribution of the 8 factors accounts for 50.83% of the variance. Eight domains or subscales of resident's perceptions were defined (Table 2).

Figure 1.

Full scree plot of the factors identified in the exploratory factor analysis of the ACLEEM.

Table 2.

Mean scores of items separated by domains of the ACLEEM questionnaire. Comparison between residents and clerks.

Domain	Items	Residents			Clerks			p value^1
		RR (%)	Mean	SD	RR (%)	Mean	SD
1. Teachers	2	99	3.55	0.521	100	3.35	0.675	0.002
(11 items; maximum score 44)	3	99	3.52	0.612	100	3.38	0.641	0.030
	4	99	3.12	0.800	100	3.13	0.822	0.878
	5	99	3.31	0.800	100	3.13	0.936	0.044
	6	100	3.12	0.924	100	3.15	0.905	0.778
	13	99	3.17	0.790	100	2.96	0.832	0.010
	16	99	3.28	0.783	100	3.27	0.702	0.933
	19	99	3.27	0.728	100	3.19	0.905	0.322
	20	99	3.37	0.784	100	3.36	0.668	0.893
	21	99	3.41	0.678	99	3.12	0.819	0.0001
	39	99	2.97	1.038	98	2.87	1.048	0.367
	Total domain	99	36.11	6.092	98	35.01	6.460	0.087
	(%)		(82.0)			(79.5)
2. Clinical activities and patient care	23	99	3.34	0.868	99	2.64	1.177	0.0001
(11 items; maximum score 44)	25	100	3.29	0.784	100	2.89	0.920	0.0001
	29	100	3.36	0.780	100	3.25	0.902	0.184
	30	100	3.43	0.651	100	3.05	0.985	0.0001
	31	100	3.35	0.668	100	3.04	0.871	0.0001
	32	100	2.84	1.008	99	2.17	1.236	0.0001
	33	100	3.47	0.572	98	3.07	0.723	0.0001
	34	100	3.43	0.645	99	3.25	0.741	0.009
	43	100	3.13	0.954	99	2.79	1.137	0.002
	44	99	3.54	0.672	99	3.15	0.822	0.0001
	45	100	3.52	0.654	98	3.19	0.836	0.0001
	Total domain	98	36.70	5.098	97	32.76	6.519	0.0001
	(%)		(84.4)			(74.4)
3. Protected time	14	99	2.78	1.120	100	2.92	1.030	0.210
(5 items; maximum score 20)	37	100	2.71	1.155	99	2.97	1.068	0.019
	40	100	2.64	1.115	99	2.87	0.981	0.035
	41	100	2.70	1.089	99	3.05	1.002	0.001
	42	100	2.37	1.307	99	3.06	0.998	0.0001
	Total domain	99	13.20	4.084	99	14.89	3.896	0.0001
	(%)		(66.0)			(74.4)
4. Infrastructure	22	100	2.98	1.190	100	3.11	0.913	0.209
(6 items; maximum score 24)	26	100	3.25	0.849	98	3.03	0.951	0.015
	47	100	2.21	1.464	99	2.99	1.156	0.0001
	48	100	3.17	1.076	99	3.21	0.931	0.649
	49	100	3.08	1.026	99	3.07	0.875	0.971
	50	100	3.19	0.959	99	3.29	0.879	0.290
	Total domain	100	17.88	4.530	98	18.68	3.755	0.056
	(%)		(74.5)			(77.8)
5. Clinical skills	1	100	3.63	0.521	100	3.54	0.619	0.102
(6 items; maximum score 24)	7	99	3.52	0.637	100	3.32	0.836	0.013
	17	100	3.03	0.994	100	2.92	1.099	0.298
	18	100	3.37	0.741	100	3.41	0.646	0.533
	35	100	3.43	0.668	99	3.25	0.725	0.014
	36	100	3.54	0.522	98	3.30	0.760	0.001
	Total domain	99	20.51	2.654	98	19.72	3.149	0.011
	(%)		(85.4)			(82.1)
6. Assessment and feedback	8	99	2.68	1.115	99	2.55	1.282	0.331
(5 items; maximum score 20)	9	99	2.85	0.963	100	2.88	0.909	0.768
	10	100	2.83	1.084	100	2.93	0.974	0.354
	11	100	2.73	0.988	100	2.66	0.916	0.463
	15	99	2.61	1.138	100	3.01	0.894	0.0001
	Total domain	99	13.73	4.110	99	14.04	3.523	0.444
	(%)		(68.65)			(70.20)
7. Information. communication and technology	12	100	2.67	1.145	100	2.42	1.157	0.033
(3 items; maximum score 12)	38	100	2.62	1.079	98	2.81	1.105	0.085
	46	100	1.96	1.550	99	2.31	1.362	0.016
	Total domain	100	7.25	2.719	98	7.58	2.754	0.239
	(%)		(60.41)			(63.16)
8. Clinical supervision^2	24	100	2.00	1.353	100	2.25	1.254	0.065
(3 items; maximum score 12)	27	99	2.17	1.282	99	2.04	1.175	0.313
	28	100	3.00	1.045	100	3.18	0.792	0.076
	Total domain	99	7.17	2.540	99	7.46	2.182	0.242
	(%)		(59.75)			(62.16)
Global questionnaire	97	152.52	23.36	93	150.61	24.62	0.761
(50 items; maximum score 200) (%)		(76.26)			(75.30)

RR: relative risk; SD: standard deviation.

Reliability analyses

Response rates varied from 99.27% (items 23 and 39) to 100%. The internal consistency of the 50-item inventory was measured with a Cronbach¿s alpha of 0.94. G-theory was carried out and the following facets were considered for the analysis: Type of Undergraduate University (Traditional, Private or Foreign) (T), PUCMS or Other University (U), Gender (G), Year of Residency or Clerkship (Y), Nationality (Chilean or Foreigner) (N), Specialty or Clerkship Program (S) and Residency or Clerkship (R)¹⁸.

We adjusted two types of models. The first group of models considered the total of each person (p) in each domain (D). In this class we have the following models: Dx(p:(UxGxY)), Dx(p:(UxYxS)), Dx(p:(TxGxY)), Dx(p:(UxG)), Dx(p:(TxG)), Dx(p:(GxY)), Dx(p:(GxN)), Dx(p:(GxS)), Dx(p:(UxR)), Dx(p:(NxR)), Dx(p:(NxS)), Dx(p:(YxR)), Dx(p:(SxT)), Dx(p:(GxR)), Dx(p:(UxS)), Dx(p:(TxR)), Dx(p:G), Dx(p:S), Dx(p:U), Dx(p:R), Dx(p:Y) and Dxp. Our main objective at this stage was to analyze how much importance had the facets in the overall variability.

The second group of models included the results at the item level (i). The corresponding models were: (i:D)x(p:(UxGxY)), (i:D)x(p:(UxYxS)), (i:D)x(p:(TxGxY)), (i:D)x(p:(UxG)), (i:D)x(p:(TxG)), (i:D)x(p:(GxY)), (i:D)x(p:(GxN)), (i:D)x(p:(GxS)), (i:D)x(p:(UxR)), (i:D)x(p:(NxR)), (i:D)x(p:(NxS)), (i:D)x(p:(YxR)), (i:D)x(p:(SxT)), (i:D)x(p:(GxR)), (i:D)x(p:(UxS)), (i:D)x(p:(TxR)), (i:D)x(p:G), (i:D)x(p:S), (i:D)x(p:U), (i:D)x(p:R), (i:D)x(p:Y) y (i:D)xp.

In order to adjust the models at the domain level, the overall scores of each domain were expressed as percentages, because each domain included a different number of items, ranging from 3 items (maximum score of 12 points) and others with 11 items (maximum score of 44 points). For the majority of these models the facets were not important due to low variability associated to the facets. A simple model: Dx(p:S), reflects well the situation, in which 26.8% of the variance is due to the variability among persons whom respond the questionnaire, 18.96% of the variance can be explained by the domains, 9.6% to the interaction between specialties and domains, 5.9% to the variability among specialties and 38.7% the variance corresponds to the interaction among persons (inside the specialty programs) with domains confounded with the error.

If we consider the models with item, we conclude that the majority of the facets are irrelevant and we consider that the final model (i:D) x (p:S) reflects well the structure of the data. In this model we can appreciate that half of the variance corresponds to the interaction between person and item (both nested because we need to remember that the persons are inside the specialties and items inside the domains) confounded with the error. Again, the variability among persons is in the first place, with 16% followed by the interaction person-domain with 10.4% and in third place the variance explained by the domains with 8.3%.

We can conclude from these two models that the major components were explained by persons (first place) and the domains (second place), and an important percentage of the variance is confounded with error.

Based on previous analyses, we were focused on following the model selected at the Domain level. The D-study was performed by using the Genova program to investigate the reliability of the instrument for absolute and relative decisions, and the number of students whom had to respond the questionnaire to produce a reliable measurement of the EE. Due to the fact that the Genova program requires balanced designs, we randomly selected a sample of 14 students from each one of the 10 most numerous specialty programs. The most important results of the D-study are shown in Table 3.

Table 3.

D Study Results Using G Coefficients of the ACLEEM questionnaire.

Number of students	G coefficients
4	0.567
5	0.621
7	0.696
8	0.724
10	0.766
15	0.831
20	0.868
30	0.908
40	0.929

We have shown here that with reasonable sample size (15 respondents) it achieves high generalizability coefficients, above 0.83. However, with 10 respondents the G-coefficient is 0.766. On the other hand, we can see that to have an excellent reliability (G-coefficient>0.9), the instrument requires 30 respondents.

Finally, the mean values of the domains 2 (Clinical Activities & Patient Care) and 5 (Clinical Skills) were significantly higher among residents compared to clerks. On the other hand, the third domain (Protected Time for Non-Clinical Activities) was significantly higher among clerks compared to residents. No differences were observed in the EE perceived by residents and clerks based on the 50-item global score (Table 2).

Discussion

Quality assurance of postgraduate educational programs and residency training is increasingly important¹⁹. EE is one of the aspects to evaluate the quality of training programs providing information about several domains like atmosphere, feedback and supervision in hospital and ambulatory settings²⁰. Several inventories have been developed to measure the EE and ACLEEM is the first one particularly developed to measure aspects related to the EE in ambulatory settings.

The ACLEEM was evaluated in this study to determine its psychometric properties to measure the learning environment in sixteen specialization programs, with a total of 411 respondents. We reached a greater proportion of clerks (151 respondents corresponding to 83.9% of the total universe) and 260 residents (74% of the total universe) from 31 postgraduate programs, which gave us a large representation of the results. According to Kane, the first stage of the research project, development of the ACLEEM questionnaire, could be considered a criterion and content based approach²¹. Content validity issues were analyzed in detail in the first publication which described the development of the ACLEEM¹². However, it is important to take into account the limitation that even when residents from Argentina, Colombia and Ecuador participated in focus groups, giving relevant feedback about the content and meaning of the statements, some words in Spanish could be interpreted in a different way by residents from other Spanish-speaking countries¹².

The construct validity was carried out by using an exploratory factor analysis followed by a Varimax rotation. Factors were chosen using the Kaiser-Guttman criterion and the Cattell criterion (inflexion point of the scree plot curve). According to the results observed, the statistical analyses suggested a multidimensional instrument with eight factors based on the criteria used in this study. Revising the preliminary 3 domains emerged from grounded theory with the 8 factors arising from factor analysis we found the following agreements: The domain “Clinical Teaching” originated 2 new factors “Teachers” and “Assessment and feedback”; “Clinical Training” was separated into 3 new factors: “Clinical activities and patient care”, “Clinical skills” and “Clinical supervision”. Finally the items of the Support¿ domain were mainly spread into “Clinical activities and patient care”, “Protected time” and “Infrastructure”.

The 51% of the variance explained by the eight factors is higher or similar to other EE evaluation instruments, such as the PHEEM (1 factor, around 30% of the variance) and the DREEM (5 factors, 52% of the variance)^17,22. The high internal consistency is a remarkable aspect of the ACLEEM with Cronbach¿s alpha of 0.94.

A sound factor analysis with 5 subjects per item (250 residents) was achieved with a number of 260 residents demonstrating the construct validity¹⁴ internal consistency and reliability including Generalizability theory²³. A D-study was successfully carried out and only 15 respondents are required for reliable results (G coefficient of 0.831). Based on our results, we can asseverate that the ACLEEM is suitable to measure the EE of different specialization programs (absolute decision), but not to compare them (relative decisions).

The ACLEEM can be administered at the end of any clinical rotation in an ambulatory setting by someone who was not involved in the rotation teaching activities. The results can be useful for teachers in clinical rotations to improve their teaching practices; for residence program managers to supervise the teaching activities and design improvements in the curriculum; and for students allowing them to identify topics that are shared as positive and negative areas within a rotation. If we analyse the mean scores of the domains, the EE was perceived higher among residents in the postgraduate level in the domains related to patient care and clinical skills including practical procedures compared to clerks. On the other hand, clerks perceived that they have more protected time for non-clinical activities reflecting a higher workload for residents. The results of the ACLEEM can be considered at three levels: (i) individual items, (ii) domains and (iii) overall ACLEEM. Theraw scores obtained for the items making up each of the eight domains are summed for each participant, and then the mean of this summed score is taken to give domain summary scores. To obtain the overall ACLEEM score, the domain summary scores are summed. Examination of the individual items by looking at the mean score obtained across all participants for each item enables the identification of specific strengths and weakness within the EE. This criterion was extrapolated from DREEM and PHEEM^24,25 where the developers reported the achieved total and subscale scores as a percentage of the maximum score possible, but made no recommendations regarding interpretation of the DREEM. Subsequently, two of the developers provided guidance as to how to interpret scores at each of the three levels²⁶. Aligned with this criterion we propose for the ACLEEM questionnaire to consider individual, domains and overall mean scores of ≥ 3.0 (75% of the maximum score) are regarded as especially strong areas, mean scores of ≤ 2.0 (50%) need particular attention and items, domains and overall mean scores between 2 and 3 (50-75%) are areas of the EE that could be improved. Under this criterion, the perceived EE observed among residents and clerks could be considered really positive, because the global mean score was >75% in both groups.

Regarding further research, we can mention the following challenges. First, as the ACLEEM was built in a Spanish speaking setting, it should be translated into English or other languages to be validated as well. Secondly, as this is a single-school study, external validity of the instrument can be analyzed by applying it in other Chilean schools or other countries. Thirdly, despite the ACLEEM was developed considering multiple medical specialties, its performance in different postgraduate programs could be compared to explore potential differences.

Finally, we conclude that the ACLEEM is a multidimensional, valid and highly reliable instrument to measure the EE in postgraduate ambulatory settings. It is remarkable how stable the findings are, given their applicability in different specialization programs. We recommend using it to measure the educational environment within each ambulatory postgraduate program with Spanish-speaking residents.