Quality indicators are a key element to evaluate and monitor any Quality management system and should be designed in order to detect at an early stage, variances in systems and implement correction measures and/or prevent errors which could have an impact on patient safety.1 To carry through the support activities for the diagnostic and follow up the different health conditions, laboratories should establish quality specifications in all processes that take place both inside and outside the lab, from the moment a doctor asks for a blood test until the results of such are received.

The Catalonian Health Institute Working Group on Quality was created in 2004, composed of the responsible for quality in public laboratories in Catalonia, of the Institut Català de la Salut (ICS) which caters for a population of seven million people. The main purpose of the Group was to select and define the most appropriate Quality Indicators to monitor laboratory processes and to fix their quality specifications or acceptability limit, in the interests of patient safety.

The aim of this paper is to describe the methodology used in the selection and definition of quality indicators and their specifications for extra-analytical processes in public laboratories in Catalonia. We also present their modification taken from the evaluation of the corresponding data from 2004 to 2013.

The composition of the working group has varied over time due to the reorganization of the laboratories within the Catalan public health system (ICS). Finally, the study group was made up of 12 laboratories, 4 of which worked in the area of primary health care, 2 in hospital care and the other 6 in both primary and hospital care areas. All of them have a management System of Quality control available according to the ISO 9001 norm, except one, which conforms to the 15189 norm.

Comparison of the different types of processes was a previous step to the study of indicators, agreeing on classify them in key, strategy and support processes.

In the first stage,2,3 indicators for the different processes were set out, the frequency of measurement, the calculation formula and their data source and preliminary specification.

To classify the indicators the guidelines of the Investigation and Health Quality Agency and the Clinical and Standards Institute (CLSI) were followed, which evaluated the incidence of the laboratory errors in different aspects of Quality management, including Patient safety.4,5

In general, frequency was defined as monthly for the indicators of key extra-analytical processes and annually for indicators of strategic and support processes. Following the guidelines of the consensus meeting in Stockholm,6 a desired specification was the average of results from all the laboratories (state of the art), with some exceptions as in the case of sentinel indicators, important for patient safety, for those which require zero error.

During the study period (2004–2013), and taking into account only programmed analysis (not urgent), the different laboratory members of the group have reported, in the shared “Excel” template, the laboratory average annual value for each indicator, and calculating the yearly average of all the participants.

These results have been compared and analyzed by means of periodic meetings, regarding the pre-established specifications of the different laboratories. This has made possible to contrast the validity of the indicators, to take corrective measures in light of unsuitable values or to reconsider the specification. Finally, the values attained to transform the indicators of key processes to the Six-sigma scale to establish the grade of control of the corresponding processes.7,8 In general, the processes with sigma values of ≥4 and with a minimum acceptable value of 39,10 were considered well controlled, except for the sentinel indicators (errors in the identification of a patient), in which a sigma value >6 is considered acceptable.

This study reflects the evolution of quality indicators and their quality specifications over a period of 10 years’ experience (2004–2013), in a context of significant organizational changes.

The definition of the indicators used, including name, description, calculation formula, units and typing are shown in Table 1 (indicators of extra-analytical key processes), Table 2 (indicators of support processes) and Table 3 (indicators of strategic processes). Table 4 shows the modification of the indicator during this period and the specification assigned, and its evolution over time (indicators of extra-analytical key processes), as well as Table 5 (indicators of support processes) and Table 6 (indicators of strategic processes). These tables also show the inclusion or elimination of some indicators.

The suitability or validity of the indicator is fundamental to monitor the process adequately, as is the correct selection and definition of the indicator. It is a hard task, due to the magnitude of certain processes, some of which cannot be compared exactly because of the difficulty of data collection and the slightly differences in data production. For many indicators, counting is not automated and implies intervention and effort on the behalf of the professionals involved.11 The aspects taken into account when evaluating the suitability or validity of the indicators are in the same line as the conclusions drawn by the Conference of consensus organized in Padua in 2013 by the work group “Laboratory Errors and Patient Safety” of the Internal Federation of Clinical Chemistry and Laboratory Medicine (IFCC).11

The aim of our working group has been to determine the usefulness of the indicator in the monitoring of a process, its correct definition (accepting a shared understanding by the members), and a feasible and precise quantification (verifying the possibility of the addition of data by the majority of the participants with little discrepancy between laboratories). It was also important to agree on its specification, to work toward improvement but also to take into account its practicability.

The ongoing monitoring and evaluation by the group has consolidated the current indicators. The fact that the different laboratories belong to the same health system, within a common culture and regulatory field and also with quality management systems initially implanted to give response to a common ISO norm has helped to unify criteria, different to other work groups, in which the members come from different countries.12 Inevitably, differences exist but these are basically between hospital and primary care (different software, different operating systems, etc.).

The evaluation of the validity of each quality indicator was done taking into account the following aspects:

• (a)

  Usefulness of the indicator in process monitoring: Most of the initial indicators are still considered useful. Only two indicators have been rejected for its uselessness: (1) copies of analytical reports, they lose importance with improvement of laboratory informatics systems, making easier the electronic requests and reporting of results. (2) Noncompliance in infrastructure, which could be useful individually but was not useful for the Group as the differences in this aspect are difficult to harmonize. Some have merged to form a new indicator, as in the case of the indicators of written and verbal complaints, which became the indicator of complaints of noncompliance. Whether they are written or verbal, the important fact is to know if this is a serious problem connected to a process. The indicators of noncompliance of computerized petitions and results were grouped together with the indicator of noncompliance of Information Technology (IT) connections between Laboratory Information System (LIS) and the client.

  Some indicators related to supply adequacy have been added, as this aspect was considered important in terms of patient safety, as well as scope and efficiency (in the organization and resource management group). Other indicators to monitor processes which did not previously exist, became necessary with the evolution of laboratories (petition unification in the IT system of clinical management of hospitals of the region (Specialized care information system – SAP)) and also in order to make a quantification more objective and comparable with other work groups (indicator of hemolysis samples according to the hemolytic index ≥0.5g/L).11

• (b)

  Indicator description: The lack of detail in the description of some indicators produces doubts in the provided results, mainly in those related to support and management processes where there is a certain degree of subjectivity (complaints and suggestions, nonconformities from suppliers of supplies and reagents or from subcontracted laboratories, quality objectives and projects). If there was not a clear definition and therefore no comparable results, the indicator has been withdrawn (evaluation of training, noncompliance of infrastructure). The definitions of all indicators have been revised; especially points which could lead to errors (see Tables 1–3). These definitions have then been included in the shared template used to collect indicator data among the Group.

• (c)

  Data collection and sources: Another factor to be taken into account is the system applied by each laboratory. When data is difficult to obtain there is little participation from the members of the Group, giving less validity to the results and/or making them subject to uncertainty and a possible incorrect quantification. The automation or semi-automation has a positive contribution to the veracity of the data (indicators of request related to erroneous demographic data or tests): when the diagnosis is carried out by scanner it is more reliable than when done by a member of staff looking at each request individually. In the same way, if the staff have to register data manually, less data is registered (verbal complaints, IT connections, noncompliance of infrastructure, noncompliance from suppliers, unidentifiable samples and management incidences of samples between laboratories). As an accurate register of incidences was not made, this was dropped. The aim is to pinpoint how each laboratory collects data for each indicator, including the information in the template of the joint register, which will prove the reliability of the data. If the different members of the group had the same software tools available, this would obviously help to unify data collection, but currently this is not the case.

• (d)

  Calculation formula: It is not always easy to get an indicator whose calculation formula directly reflects what it is supposed to be measuring. Sometimes approximations are given (indirect indicators) when it is difficult to access more direct data. This occurs in the different indicators depending on the sample type, where the denominator is the total of requests, representing the number of patients, in order to standardize the data collection and improve the comparability of results between the laboratories of the group.

  This is because not all laboratories have quantification according to sample type, or they have a different way of recording them, resulting in an underestimation of sample numbers related to tests which are requested less frequently (stools, swabs, citrate or heparin tubes). In the future, the denominator to calculate them could be the analytical activity, defined as the number of determinations of the most frequently measured test in each sample type (i.e., creatinine for the tube to collect serum, complete blood count for the EDTA tube, etc.), as this data is easy to get hold of in all laboratories.

  Whenever possible, indicators, which are still recorded in absolute values (complaints, noncompliance of suppliers, etc.), will be qualified with the denominator being the activity of the laboratory estimated by the total requests.

• (e)

  Indicator specification:Table 4 shows (indicators of extra-analytical key processes), 5 (indicators of support processes) and 6 (indicators of strategic processes) and how they have evolved over time, being conditioned by the evaluation of accumulated data.For the majority of the indicators the specification is seen as a cut-off value (an annual average), as it is clearer. In the case of indicators related with achieving quality objectives or projects we have considerate it would be useful to use an interval, controlling the limit below the interval to reach the objective or project, and the upper limit precision in the definition of the objective or project to implement.

The Group assigned a value to establish preliminary specifications,2,3 which are later revised after five years of experience [2004–2008]7,8 to see if the original specifications were suited to reality. A further more thorough revision was carried out in 2013, in which specification was designed for all the indicators which still had none, and those for which there was a lack of data (incidences of allocation, tests not corresponding to the requests, unidentifiable samples, professional satisfaction, extraction centers, transport noncompliance and noncompliance from subcontracted laboratories) and the cut-off value was modified (fault repairs and maintenance hours of the analyzers, noncompliance of material suppliers and reagents and training, hours).

Revision and change of the requirements is representative of the actual functioning of the laboratories in the group along the years. Therefore, we can consider the new specification more robust and realistic than the initial, since incorporates the changes in the organization and the evolution of the laboratories during the period of evaluation. For example, the implementation of the electronic request in all the laboratories carried to restrict the specification for the indicator Data missing (demographic data or tests), whereas the difficulty in reducing the number of incidences related with the indicator Samples not received: urines (that is largely in the hands of the user and that affects especially to primary care) or with the “accidents at work” (in a lot of occasions linked to no predictable situations, like those which happen “in itinere”), carried to ease the specification for these indicators.

The indicators of erroneous data in patient identification and requests with incorrect data which are not detected in the laboratory are considered as sentinel indicators. A specification of 0 DPM (defects per million), >6 in the sigma scale is proposed for these indicators. For indicators of incidences in samples that do not arrive to the laboratory, or that are insufficient, coagulated or hemolyzed, specifications with sigma values above 4, are regarded as a good process control. However, it must be pointed out that here are processes whose specification approaches the minimum acceptable rating (global incidences in external samples and tests delivered with delay).

When taking into consideration the indicators used in international clinical laboratories and regarding the work of Plebani et al.,11 in which indicators of maximum priority correspond to key processes, our Group has an almost exact alignment in pre-analytic indicators, but more differences are found in post-analytical indicators. We are considering including some of them in the future, monitoring errors (report errors or results), postponing others due to the difficulty of data collection (delayed warning of critical values).

The number of quality indicators should be limited, an excess could mean that what is really important is not given sufficient priority. For this reason, in the future, in agreement with what has been established by the Quality indicators and patient security Group of the IFCC,11 a degree of priority will be given to each indicator and the participation index will be quantified for the different indicators.12 Those indicators for which data collection is difficult will be excluded, as well as those with little data, as they could reduce the validity of the results. A few global representative indicators should be selected, which would efficiently give information about the system: total number of requests, total number of samples, total number of results/report errors, total number of delayed results/reports, etc.13

Quality indicators used by the Group are dynamic. Their definition and/or specification has been redesigned to fit in with changes in the process itself (remove the indicator of report copies that they are currently computerized), the availability of more valid automatic data (new functions of LIS), using calculation formulas which are more efficient (more representative denominators) or more useful for organizational needs (introducing demand indicators). Quality specifications of the processes should be defined according to how they are controlled, and should be modified periodically by analyzing the data over time and/or depending on real possibilities of changing the process.

The harmonization in the definition of the indicators and their corresponding specifications, and the continuous monitoring of them, allows the laboratory to effectively and efficiently control his processes, especially those related to the risks management. The indicators assessment allows to know the prevalence of the failures/errors, analyze their causes and take actions to avoid their appearance, all is in order to increase patient safety. This profit is obtained, for example, with the improvement reflected by the results of the indicators, on the identification of the patient, the term of delivery, the number of no received samples, or the percentage of hemolyzed samples, allowing that the laboratory reports are complete, in time and with more reliable results. Probably, all this has also an economic impact on the health system, as it leads to a decrease in the number of medical visits, the number of new extractions or the number of complementary tests need for diagnostic. However, the quantification of the economic impact that results of this improvement is beyond of the aim of this study.

The last 10 years’ experience of the Catalonian Health Institute Working Group on Quality Indicators to design indicators and establish quality specifications has proved very useful to improve the control of processes in clinical laboratories integrating the group. We believe it would be useful to extend this working method to other laboratories.

The authors declare no conflict of interest.