Measuring the Institutional Efficiency Using DEA and AHP: the Case of a Mexican University

Abstract

There is a general interest in the study of schemes for the measurement of the efficiency of universities, which generates demand but at the same time is controversial because of the complexity of the problem. This problem is associated with the highly combinatorial characteristics that occur when facing the selection of the proper combination of the attributes, namely inputs and outputs. This investigation proposes an approach to measure the institutional efficiency in higher educational institutions combining Analytic Hierarchy Process (AHP) with Data Envelopment Analysis (DEA). Both methods are frequently used independently, on a global level in areas such as government, business, industry, health care and education. The use of the two methodologies as an evaluation tool is novel and very useful in institutional efficiency studies where results already exist, in order to obtain and confirm important equivalences. The use of the proposed approach is demonstrated using the Queretaro State University - Universidad Autónoma de Querétaro (UAQ) - as a case study.

Resumen

Hay un interés general en el estudio de los esquemas para la medición de la eficiencia en las universidades, que genera demanda y al mismo tiempo controversia debido a la complejidad del problema, asociada al carácter altamente combinatorio que se presenta para la seleccionar la combinación adecuada de los múltiples atributos (inputs y outputs).

En esta investigación se propone un enfoque para medir la eficiencia institucional en la educación superior combinando el Proceso de Jerarquía Analítica (PJA) con el Análisis Envolvente de datos (AED). Actualmente, ambas metodologías son usadas ampliamente de manera independiente, a nivel mundial en áreas, tales como gobierno, negocios, industria, atención de salud y educación. El uso conjunto de las dos metodologías constituye una herramienta novedosa y es muy útil para estudios de eficiencia institucional, ya que los resultados que arroja permiten obtener y confirmar equivalencias importantes. La utilización del enfoque propuesto es ilustrada con el caso de la Universidad Autónoma de Querétaro (UAQ).

1

Introduction

Efficiency is the achievement of an objective, utilizing a minimum amount of resources, Koontz and Weihrich (2004). Keeping the aforementioned definition in mind, we can consider institutional efficiency as a situation where the institution makes appropriate use of resources to achieve the goals proposed in its planning. Public education policies in several countries are changing the traditional arguments which prefer equity toward achieving goals of educational efficiency. The situation here at UAQ is not the exception. To this end, it is necessary to use techniques that enable an objective evaluation of the educational performance. UAQ, like so many other institutions, is compromised with issues of academic excellence and for the improvement of the current educational systems. Consequently, it is very important to evaluate the performance of faculties, in areas of research, teaching and administration through indicators and performance models that are of similar complexity with today's educational demands. Assessment of higher education is a common practice in several countries, as can be seen in strategies for improving the quality of higher education in Europe, Martin (2006). In fact, there are already performance indicators in place in certain areas and their results have impacted the decisions of students and employers Colbert et al. (2000).

In UAQ, which was established in the 17th century, there are currently 133 educational programs (EP) being taught, from the level of higher university technician to doctorate degrees in 13 faculties or Dependencias de Educación Superior (DES). Carrying out an evaluation of performance of any organization requires an understanding of its goals and objectives, Johnes (1992). In the case of the UAQ, each of the 13 DES has its own peculiarities; therefore, the complexity of making a proper and just evaluation which satisfies all parties and reflects the actual behavior of each.

2

Analytic Hierarchy Process (AHP)

The AHP elaborated by Thomas Saaty (1977, 1980, 1982) was designed to solve complex problems concerned with multicriteria.

Several papers have compiled the AHP success stories (Kumar & Vaidya, 2006; Ho, 2008; Shipai & Timor, 2010).

The AHP requires that the decision makers supply assessments regarding the relative importance of every opinion which specify a preference for each alternative in the decision making process. The AHP's output is a classification sorted by priorities of the alternatives of decisions made, based on global preferences expressed by the decision makers.

Since its introduction, AHP has been widely used, for example in manufacturing systems (Ic & Yurdakul, 2009), supplier selection (Labib, 2011), energy selection (Kahraman & Kaya, 2010), university evaluation (Lee, 2010), risk (López & Salmeron, 2011; Tian & Yan, 2013) and many others.

For the development of this research the indicators extracted from Programa Integral de Fortalecimiento Institucional (PIFI 2008–2009)¹ - Integrated Program of Institutional Strengthening were used. The aforementioned results were used to evaluate the performance of the 13 DES of the UAQ on the basis of multiple criteria.

The results obtained from the Integral Evaluation of the PIFI 2008–2009, see Table 1, were selected using the following indicators: Capacity (1.1, 1.2, 1.3, 1.4, 1.5) as related to the full time professors and the “academics bodies” which are composed of full time professors working in the same research field; Competitiveness (1.7, 1.8, 1.9) as related to educational programs; Institutional Self-Evaluation (3.1, 3.2, 3.3, 3.4, 3.5, 3.6); Updating of the Planning in Institutional Scope (4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 5.1, 5.2, 5.3) and Enrollment. All and all this totaled 26 Criteria and 13 Alternatives (DES), which constitute the inputs for the AHP with the Global Goal of obtaining the best DES for UAQ.

Table 1.

Results obtained from the Integral Evaluation of PIFI 2008–2009, for UAQ

Each of the criteria was compared in pairs so as to determine its relative importance. Then, the DES were compared each other in pairs in regard to each of the 26 criteria.

3

Data Envelopment Analysis (DEA)

The Methodology of Data Envelopment Analysis (DEA) was first characterized in Charnes, Cooper and Rhodes (1978) as a way of comparing the efficiency of Decision Making Units (DMUs) that have multiple inputs and outputs. A DMU can be a company offering a service, manufacturing or, as in this case, an institution of higher education.

DEA has been widely used to evaluate the relative performance of a set of DMUs based on multiple criteria.

Because this requires very few assumptions, DEA has opened possibilities for institutional evaluations which can generally be very difficult to carry out, because of the complex nature of the relations between multiple inputs and outputs.

DEA, unlike other methods, use financial and non-financial elements. This method is also particularly appropriate to assess the efficiency of public universities because they operate outside the market. Criteria such as profitability and income are “not satisfactory”. This is, because public universities are not geared towards making a profit. Furthermore, in these institutions the primary source of financing does not come from the sales of goods and services.

In this research it is assumed that if a DES, named DES1 is able to produce or generate Y1 output's units with X1 input's units, then other DES must also be able to do the same if they are operated efficiently. Similarly, if DES2 is able to produce Y2 output's units with X2 input's units, then the other DES must also be able to do the same. DES1 and DES2 can be combined to generate a DES (virtual) composed of inputs and outputs of them. This virtual DES is used like a standard of performance for the DES.

In particular, several studies have been done to analyze the efficiency in institutions of higher education. Among the most relevant articles that apply DEA, are the following:

The comparative analysis of Rhodes and Southwick (1986) that studied the efficiency of the public and private universities of the USA. McMillan and Datta (1998) used DEA to assess the relative efficiency of 45 Canadian Universities. Ng and Li (2000) examined the effectiveness of the reform implemented in the mid 80's in China. Abbot and Doucoulagos (2003) used DEA to determine the performance in the research and teaching in public universities of Australia.

Bougnol and Dulá (2006) applied DEA to assess performance in higher education.

Figueiredo de Franca, de Figueiredo and Lapa (2009) presented a methodology of DEA to assess the impact on the asymmetry of information on the efficiency with an application to the higher education systems in Brazil.

Sav (2012) compared private for-profit colleges to publicly owned colleges in terms of their operating efficiency and productivity using DEA.

Concerning higher education in Mexico, few studies have been conducted using DEA. Two of these investigations are by Siegler (2004) and Güemes-Castorena (2008).

4

Formulation of AHP Model for DES of the UAQ

The methodology used for this model is as follows:

• 1.

  Selection of the Global Goal, that is “selection of the best DES”, which by its academic results (capacity and competitiveness, institutional self-evaluation and updating of the planning in the institutional scope) would be in the best position.

• 2.

  Selection of Criteria, which in this case correspond to the 26 criteria; 25 PIFI indicators (1.1, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8, 1.9, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 4.1, 4.2, 8 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 5.1, 5.2, 5.3) and the Enrollment of Students from each DES.

• 3.

  Selection of Decision Alternatives (all DES of the UAQ, 13 in total); - Legal Sciences, Natural Sciences, Chemical Sciences, Social Sciences, Psychology, Economic- Administrative, Medicine, Nursing, Fine Arts, Philosophy, Languages and Literature, Information Technology, and Engineering. The resulting hierarchy is shown in Figure 1.

  
  

  Figure 1.

  Hierarchy of the Model of the DES-UAQ

4.1

Computationals Results of AHP

The ExpertChoiceTM software was used to record the calculations. The results shown in Figures 2 and 3 were obtained. The 13th DES showed the best results and the 9th DES the worst.

Figure 2.

Comparative results obtained from each of the DES with regard to the 26 criteria, in addition to its relative position with regard to the overall goal of selecting the best DES

Figure 3.

Relative position of the DES of the UAQ with regard to the overall goal of selecting the best DES

5

Formulation of DEA Model for DES of the UAQ

Inputs and Outputs

In a model of DEA, undesirable inputs and outputs may be present. It is possible to have undesirable outputs as the number of defective products. Therefore, it is desirable to reduce their number to improve performance, Zhu (2009). Problems arise in conventional models of DEA because it is assumed that the outputs should be increased and the inputs decreased in order to improve performance or reach the border of best practices. There are situations in educational practice where certain inputs need to be increased or some outputs decreased to improve institutional performance, these are then called “undesirable”. In this case the undesirable inputs are: PTC Doctorado and PTC SNI. Seiford & Zhu (2002) developed an approach to deal with these input/output undesirables in envelopment models with variable returns to scale. DEA classification invariances were used in order to ascertain the efficiencies and inefficiencies which are invariant to the data transformation. The inputs and outputs for this model are those shown in Table 2.

Table 2.

Inputs and Outputs for DEA

INPUTS (Number of)	OUTPUTS (Number of)
PTC Doctorate degree (Doctorado)	Graduate studies in PNPC
PTC SNI	CA
	PE

The meaning of these is as follows:

PTC Doctorado: Full time professors with a doctorate degree.

PTC SNI: Full time professors that belong to Sistema Nacional de Investigadores (SNI²) - National System of Researchers.

PNPC:Programa Nacional de Posgrados de Calidad (PNPC) - National Program of Quality Graduate Studies, which is jointly administered by Secretaría de Educación Pública - Secretariat of Public Education through Subsecretaría de Educación Superior - Sub-secretariat of Higher Education and Consejo Nacional de Ciencia y Tecnología (CONACYT) - National Council of Science and Technology. The program has established that its mission is “to promote the continuous improvement and the quality assurance of the national graduate studies, which offers support to increase scientific capacity, technological, social, humanities, and innovation of the country”.

A postgraduate program which pertains to PNPC means that it is recognized by the academic community and society in regards to its quality; this recognition is the result of evaluation and monitoring processes conducted by a committee of researchers nominated by CONACYT. The aim of PNPC is to guarantee the quality of higher education institutions in Mexico.

CA:Cuerpo Académico (CA) - Academic Body is a set of professors/researchers who share one or more common lines of study, whose objectives are intended for the generation and/or application of new knowledge. In addition, because of the high degree of specialization that is reached in participating in the research and teaching, they provide a high quality of education. The academic bodies support academic institutional functions and integrate part of the system of higher education within the country.

PE Acreditados:Programas Educativos Acreditados (PE) - Accredited Educational Programs: These are educational programs in which academic bodies composed of professors within the institutions of higher education throughout the country evaluate and certify the functions and the academic programs that are taught. They then delivered recommendations regarding improvement to the managers of these institutions, which are contained in evaluation reports.

5.1

Model of DEA

The model considered herein is that of variable returns to scale with undesirable inputs. In order to increase the institutional efficiency two inputs were increased (PTC Doctorado and PTC SNI) and these were not to be reduced.

Variable Returns to Scale oriented to Input Model:

Subject to:

Denoting with XijI as the inputs to be increased and XijD as the inputs to be decreased, in order to improve performance of a DMU.

XijI was multiplied by -1 and then an appropriate u_i was found to obtain

The following model is based on the previous one, using the transformation above.

Variable Returns to Scale oriented to Input Model with Undesirable Inputs:

Subject to:

Where XijI is increased XijD is decreased for a DMU to improve the performance.

5.2

Computationals Results of DEA

As can be seen in Table 3, the thirteen DES that exist in UAQ with their respective values of inputs and outputs are presented. For this research DEAFrontierTM software was used, which is a complement to Excel as developed by Joe Zhu. The results in Figure 4 were obtained.

Table 3.

Inputs (PTC Doctorado, PTC SNI) and Outputs (PNPC, CA, PE Acreditados) in Excel

Figure 4.

DES-UAQ Efficiency

6

Analysis of the Models

As can be seen in Figure 5, when comparing the two methods, the first three DES and the last three have similar rankings, which is understandable because within the first three, their academic positioning is highly consolidated. There are 10 graduate programs featured at UAQ which are part of PNPC. Of these, 6 are offered in DES13, 2 belong to DES3 and 2 to DES2. The final three DES have a limited number of PTC with doctorate degree. and any program within the PNPC.

Figure 5.

Ranking of DES-UAQ using DEA and AHP

The difference in rankings given, for example in DES10, 5° in DEA and 10° in AHP, is that DES 10 has a better position, using DEA, because it is the fifth DES in relation to PTC with doctorate degree and it is the fourth as related to PTC SNI. However, when the analysis is done with all the indicators utilizing AHP, its academic advantage is diminished. 16

In the case of DEA, the number of full time professors with doctorate degrees and SNI make substantial contributions to increasing the academic competitiveness. These factors could be relevant in determining the best DES using DEA instead of utilizing AHP which requires more time to be implemented.

7

Conclusions

In this research, an approach to measure institutional efficiency combining AHP and DEA has been established. The majority of results obtained using AHP correlated with those of DEA also reflected a widespread perception about how the performance of a university might be evaluated.

The modeling of AHP and DEA combined offers decision makers an opportunity to learn more about the educational systems in order to define policies that permit academic authorities to make better decisions in the short and long term.

When planning, and developing programs within the universities, it is necessary to generate and analyze the indicators of academic performance in order to improve academic competitiveness, stimulate educational innovation and strengthen academic ability. The combination of AHP and DEA can be used to facilitate this process.

According to Güemes-Castorena (2008), the allocation of resources in the higher education system in Mexico correlates to the enrollment, the professors and administrators. However, they are poorly correlated to the SNI and academic efficiency.

If authorities want budget allocations from certain government programs to be developed into successful and equitable action, it would be necessary to promote efficiency and higher levels of academic performance in the institutions.

Using a combination of AHP and DEA can facilitate the task of laying the foundations and criteria for the allocation of financial resources.

This research not only provides evidence that the integrated DEA-AHP is better than the stand-alone DEA, but it can also contribute to analyze the institutional efficiency and the planning processes of higher education.

Acknowledgements

The authors wish to acknowledge Consejo Nacional de Ciencia y Tecnología (CONACYT) and Universidad Autónoma de Querétaro for providing the resources for this research.