The National Quality Control Program in Mycology (PNCCM) of Argentina was established in 1996 to improve the quality of the mycological diagnosis, to help establish and to set up standardized procedures and continuous training of laboratory staff. The aim of this study was to assess the effectiveness of the PNCCM in the 1996–2018 period. Data from the National Mycology Laboratory Network (NMLN) and PNCCM database was used to estimate the increase in the number of controlled laboratories and jurisdictions, the percentage of participation, the improvement in the quality of results and the adherence to the program. Satisfaction surveys were performed to assess user satisfaction. The number of controlled laboratories increased from 29 to 146; participation increased from 49% to 93% and general adherence was 72% in the evaluated period (1996–2018). Improvement in the quality of the results was 15% for low complexity samples; 7% for intermediate complexity samples and 14% for the identification of high complexity strains. Up to 84% of the users consider the PNCCM to be “very good” and 16% “satisfactory”. These results show the importance of the PNCCM, which is widely accepted by mycological diagnostic laboratories from Argentina.
En 1996 se creó el Programa Nacional de Control de Calidad en Micología (PNCCM) de Argentina con el objetivo de mejorar la calidad del diagnóstico micológico, colaborar en el establecimiento de procedimientos estandarizados en aquellos laboratorios que carecen de ellos y contribuir a la capacitación continua del personal.
El objetivo de este estudio fue evaluar la efectividad del PNCCM en el período 1996-2018.
Se utilizaron los datos de la base de la Red Nacional de Laboratorios de Micología (RNLM) y del PNCCM para estimar el aumento en el número de laboratorios controlados y el porcentaje de participación, la mejora de la calidad de los resultados y la adhesión al programa. Para evaluar el grado de satisfacción de los usuarios, se analizaron las encuestas de satisfacción de los participantes. En el período evaluado, el número de laboratorios controlados aumentó de 29 a 146, la participación aumentó de 49% a 93% y la adherencia general de los participantes fue del 72%. La mejora de la calidad de los resultados de los laboratorios fue del 15% para muestras de baja complejidad, 7% para muestras de complejidad intermedia y 14% para la identificación de cepas de alta complejidad. El 84% de los usuarios considera que el PNCCM es muy bueno y el 16% que es satisfactorio. Estos resultados evidencian la importancia del PNCCM, que es ampliamente aceptado por los laboratorios que realizan diagnóstico micológico en nuestro país.
During the last years, the diagnosis of mycosis has become a challenge for mycology laboratories2. Causes such as the increase in fungal infections and the emergence of new fungal pathogens and the development of resistance to antifungal drugs generated the need to incorporate more complex and specific diagnostic procedures1,3–6,8,12–15.
Moreover, there is a trend toward the implementation of quality standards such as the ISO 15189 International Standard to ensure accurate, reliable and timely laboratory results and the continuous improvement of laboratory performance7. Participation in external quality assessment programs is one of the requirements of this standard, being essential to verify laboratory performance and, eventually, to improve it.
Argentina is a developing country with a surface area of 3761274km2 and an average population density of 10.7 inhabitants per km2 (https://www.ign.gob.ar). The country has large geographical areas with low population densities, where health facilities are generally scarce or limited; in contrast, city centers have higher population density and more health facilities (https://www.ign.gob.ar). To overcome this limitation, a laboratory network was created in 1996 in order to organize the referral of complex samples from low complexity laboratories to high complexity laboratories. In the same year, the Mycology Department (MD) of the National Institute of Infectious Diseases, introduced a National External Program of Quality Control in Mycology (PNCCM, for its acronym in Spanish), which offered quality assessments to the members of the network and other mycology laboratories. It aims to improve diagnostic quality, to help establish and to set up standardized procedures and training of laboratory staff. Participation in the program is not compulsory, and participating laboratories are grouped into three categories depending on their complexity.
The aim of this study was to evaluate the effectiveness of the PNCCM and the performance of the laboratories participating in the PNCCM in the period 1996–2018.
MethodsPNCCM organization: The program was organized into three components. The first component was the coordinating center (CC), located at the MD. The CC was responsible for the program direction, specimen preparation and, performance evaluation of the participating laboratories. The regional coordinators (RCs) constituted the second component (one per jurisdiction), and were responsible for specimen distribution. Jurisdiction refers to the City of Buenos Aires and the 23 provinces of Argentina. The third component encompassed all participating laboratories.
The object of study of the program was the assessment of hospital laboratory performance. Hospital laboratories were categorized by complexity (low, intermediate, or high) according to the medical practices offered and the associated fungal infections.
From its inception in 1996 until 1998, there was one testing event per year, comprising three specimens. Since 1998, there were two testing events per year. Then, since 2006 there have been two testing events per year, consisting of four specimens.
Specimen preparation: Selected specimens were clinical samples or fungal collection strains with different levels of difficulty for their identification. Clinical samples included hyperimmune sera, homogenized clinical samples, and simulated clinical samples inactivated with formalin. Fungal strains included both common and uncommon or emerging pathogens. Samples included an information sheet with epidemiological, geographical, and clinical data.
A batch of each specimen was produced and controlled. Evaluation of homogeneity of clinical samples and fungal strain batches involved a random sample of 10%. Viability, purity, and main characteristics were assessed for each specimen. Once homogeneity of the batch was confirmed, a new random sample of 10% was stored in order to assay its stability in case of unexpected troubles in the testing event.
The specimens were shipped into a triple packaging system according to the Guidelines for the Safe Transport of Infectious Substances and Diagnostic Specimens16.
Difficulty level of the samples: Specimens were divided into three categories according to the difficulty to be solved: Low difficulty level (L1) included fungal strains belonging to common dermatophyte species and Candida albicans, and clinical samples of skin, hair, and nails. Intermediate difficulty level (L2) included fungal strains belonging to Aspergillus fumigatus, A. niger and frequently isolated yeast species; as well as clinical samples for the determination of opportunist mycoses by direct examination (BAL, CSF).
High difficulty level (L3) included fungal strains belonging to Mucorales, dematiaceous, other hyphomycetes, and less frequent yeast species. Clinical samples included the diagnosis of endemic mycoses (coccidioidomycosis, paracoccidioidomycosis, and histoplasmosis) and chronic aspergillosis.
Sample analysis and response evaluation: To process and analyze the specimens, the laboratories were required to perform the same procedures they use for routine work. Laboratories submitted their results by filling out online standardized forms within two months of the shipment.
Fungal strains were evaluated based on the identification at genus or species level. Furthermore, the macro-and micromorphological, biochemical, physiological, and/or nutrition characteristics were submitted. Clinical samples and simulated clinical samples were evaluated by microscopic observation. The criterion for evaluation of the hyperimmune sera was the diagnostic result.
The data submitted by laboratories were stored and analyzed using a database management system designed in-house.
Reference responses: Each year, the six best-performing laboratories were selected. Their results were an internal check of the responses of each specimen. If at least three out of the six laboratories failed to identify a specimen, it was withdrawn from the testing event and not considered in the participants’ overall performance.
Reporting: Each participant received an individual report and a global report that summarized the results of all participating laboratories by complexity level.
When similar methodological errors arose in a significant number of laboratories, the evaluation report included additional information. RCs received a jurisdictional report summarizing the overall results. Specific information regarding individual laboratory identity and performance remained confidential.
Effectiveness of the PNCCM: Five indicators assessed the performance of the PNCCM. 1 – Number of laboratories and jurisdictions controlled. 2 – User adherence: percentage of laboratories participating in 2018 over the total number of laboratories enrolled since 1996. 3 – User satisfaction: five satisfaction surveys were carried out in the period 2006–2018. 4 – Improvement of laboratory results by sample complexity level: possible percentage increase in the number of correct results comparing two periods: the first from 1996 to 2003 and the other from 2012 to 2018. 5 – Laboratory performance: the average value of correct responses was calculated for each sample complexity level, and was used as a cut-off value to define whether each laboratory reached its level of complexity. A level 3 laboratory must reach three cut-offs (L3, L2, and L1). A level 2 laboratory must reach L2 and L1 cut-offs, and a level 1 laboratory only has to reach L1 cut-off.
This analysis included laboratories that responded at least six testing events from 1996 to 2018. In addition, the percentage of correct answers by specimen and sample complexity level was determined.
ResultsEffectiveness of the PNCCM1 – Number of laboratories and jurisdictions controlled: In 1996, 59 laboratories enrolled in the PNCCM, but only 29 answered the first testing event. Participants belonged to 14 out of the 24 jurisdictions. Since 2009, 148 laboratories have enrolled in the PNCCM, and 139 answered the testing events. From that moment on, the program was able to control all jurisdictions.
Figure 1 shows the increase in the number of participating laboratories by year. The percentage of participants who answered each testing event increased from 49% to 91% between 1996 and 2000 and thereafter fluctuated between 80% and 97% (Fig. 1).
2 – User adherence: Overall adherence of the participants during the evaluated period was 72% (166/229). In the public sector, adherence was 78% (140/179), while in the private sector was 52% (26/50).
3 – User satisfaction: The satisfaction surveys were answered by 26% of the participating laboratories, considering the PNCCM to be very good (84%) or satisfactory (16%).
4 – Improvement of laboratory results: The percentage of correct answers was 76% and 82% in the periods 1996–2003 and 2012–2018, respectively. Figure 2 shows the upward trend in the average correct answers per testing event. The overall average of correct answers for the testing events from 1996 to 2018 was 79%.
The overall improvement in the quality of the results for the evaluated period was 15% for low complexity samples, 7% for intermediate complexity samples, and 14% for improvement in the identification of level 3 strains. Serological techniques results showed no improvement in the evaluated period.
The percentage of correct answers for the identification of yeasts frequently associated with fungal infections in Argentina4,9,10,14 reached 80% (yeast of L1 and L2 samples). In the case of C. albicans, the percentage increased to 95%, while it markedly decreased when the species were emerging pathogens such as Candida haemulonii (13%), Cryptococcus gattii (28%), and Wickerhamomyces anomalus (33%).
Aspergillus spp. were correctly identified at the species level in 84% of the cases. Even emerging pathogens (level 3), such as A. terreus, were correctly identified at the species level by 48% of the laboratories in 2007 and by 68% of the laboratories in 2015.
Filamentous fungi uncommonly associated with human infections were correctly identified, at least at the genus level, 77% of the time. Moreover, less than 50% of the laboratories were able to identify other emerging pathogens, such as Scedosporium boydii and Purpureocillium lilacinum, which were only correctly identified by 44 and 29% of the participants, respectively.
Table S1 shows the percentage of correct answers by type of specimen and difficulty level (L1: low; L2: intermediate; L3: high, supplementary material).
5 – Laboratory performance: About 134 laboratories were included in the calculation of the cut-off value, which was 83% for level 1, 78% for level 2 and level 3, 74% for species identification and 88% for serological diagnosis. Figure 3 shows the distribution of the percentage of correct answers (score) according to the sample complexity level. Of 134 laboratories, 87 achieved the L1 sample cut-off value, 62 achieved the L2 sample cut-off value, and 54 achieved the L3 sample cut-off value for strains. Of 44 laboratories that performed serological diagnosis, 13 achieved the L3 sample cut-off value for antisera.
Over 82% of the laboratories answered correctly more than 70% of the samples sent, as shown by the black bars in Figure 3.
DiscussionArgentina does not have a mandatory external quality control program in mycology; however, the Ministry of Health and Social Development strongly recommends National Reference Laboratories to offer quality assessment programs for national laboratories networks and to evaluate their performance. In addition to the PNCCM, the National Reference Laboratory in Clinical Mycology offers continuous training, produces, and provides diagnostic reagents and reference strains for technology transfer. Through these activities, this institution aims to increase the diagnostic capacity of fungal infections in Argentina, and particularly the National Mycology Laboratory Network. Professionals widely accepted the importance of quality assurance, as shown by a growing interest in PNCCM since its inception until today.
In this study, we evaluate the usefulness of the PNCCM by analyzing its effectiveness through indicators and determining the performance of the laboratories. This analysis showed that, in the studied period, the number of controlled laboratories increased more than 4 times and the number of jurisdictions increased from 18 to 24. As a result, all jurisdictions have at least one laboratory to perform the diagnosis of fungal diseases in their area, increasing the coverage to 100% of the jurisdictions in Argentina. Quality surveys showed a high level of satisfaction. However, a low number of laboratories answered them. The degree of adherence was high, particularly in public laboratories that have shown an adherence of 78%. The majority of the laboratories have participated continuously since they enrolled in the program. On the other hand, some laboratories have left the PNCCM for reasons not related to the program, including laboratories that stopped providing services. We observed an improvement in the quality of laboratory results through time, which may be a consequence of the continuous training offered by the PNCCM. However, we detected a low percentage of correct answers in the identification of infrequent pathogens. It is necessary to keep training the laboratories in the identification of emerging pathogens, as was evidenced by the low percentages of correct responses (<45%) of C. gattii, W. anomalus, C. haemulloni, P. lilacinum, and S. boydii.
More than 75% of laboratories correctly identified A. fumigatus and A. niger. On the other hand, the improvement in the identification of A. terreus, an emerging pathogen of high complexity, suggests that laboratories have a good training in the resolution of this genus. Finally, the performance of the laboratories over the period 1996–2018 showed that 82% of them correctly identified over 70% of the specimens. This analysis evidenced which laboratories should be trained. Unfortunately, few reports about this subject are available and it was not possible to compare the effectiveness of the PNCCM with other proficiency testing programs.
This study showed that the PNCCM was effective in detecting methodological or interpretation errors, identifying the needs for training and evaluating the performance of the laboratories. The PNCCM created uniform quality standards for all laboratory tests to ensure the accuracy, reliability, and timeliness of patient test results, regardless of the laboratory involved. These requirements depend on the assay and the complexity of the laboratory.
Finally, in agreement with Stan et al.11, we believe that a proficiency testing program is a valuable tool for assessing laboratory performance as well as for improving the quality of the results.
Conflict of interestThe authors declare that they have no conflicts of interest.
The present study has not received specific aid from agencies from the public sector, commercial sector or non-profit entities.
We want to thank all the participants that voluntarily joined the PNCCM.
Laboratories participating in the Mycology Quality Control Program. See Appendix A for a complete list of members.
