The surge in digital pathology and the vast amount of data from whole slide images (WSIs) have made it essential to develop tools that can efficiently analyze and support cancer diagnosis. An innovative artificial intelligence-based pipeline has been developed to tackle this challenge, significantly speeding up and enhancing the accuracy of breast cancer diagnosis. Created within the DigiPatICS project, in partnership with 8 hospitals across Catalonia, Spain, this pipeline begins by identifying tissue areas in WSIs and breaking them down into smaller, manageable tiles. Using smart image processing techniques, it filters out tiles that do not contain relevant information and focuses on the essential ones. Advanced deep learning algorithms then work to identify and classify different types of cells within the tissue. Proven effective on key breast cancer markers like HER2, Ki67, ER, and PR, the system precomputes the results overnight, allowing pathologists to simply load the pre-analyzed data for the areas of interest of the WSI that they select during their workday. Each WSI is analyzed in more or less 1110&#¿;s, providing reliable results that are ready when needed. Now, an integral part of routine workflows, this approach is revolutionizing how breast cancer slides are processed, enhancing diagnostic capabilities, and reshaping digital pathology.
El avance de la patología digital y el gran volumen de datos de las Whole Slide Images (WSIs) han creado la necesidad de herramientas que analicen y apoyen de manera efectiva el diagnóstico del cáncer. Para abordar este reto, hemos desarrollado un innovador proceso basado en inteligencia artificial que acelera y mejora la precisión del diagnóstico del cáncer de mama. Desarrollado dentro del proyecto DigiPatICS, en colaboración con ocho hospitales de Cataluña, este proceso identifica áreas de tejido en las WSIs, las divide en partes pequeñas y manejables, y filtra las que no aportan información relevante, centrando el análisis en las esenciales. Algoritmos de aprendizaje profundo avanzados clasifican los diferentes tipos de células, con resultados efectivos en marcadores clave como HER2, Ki67, ER y PR. El sistema precomputa los datos durante la noche, permitiendo a los patólogos trabajar directamente con los resultados preanalizados de las áreas de interés de las WSIs durante su jornada laboral. Cada imagen se analiza en unos 1110 segundos, ofreciendo resultados fiables cuando se necesitan. Integrado ya en la rutina diaria, este enfoque está transformando la forma en que se procesan las muestras de cáncer de mama, mejorando el diagnóstico y redefiniendo la patología digital.
Histopathological analysis plays a key role in cancer diagnosis. With the growing adoption of digital pathology, whole slide imaging (WSI) has emerged as a valuable tool, allowing pathologists to access and analyze high-resolution tissue samples on digital platforms.1–3 However, the manual examination of large WSIs, result of the tissue scanning process, can be time-consuming and labor-intensive. To improve the efficiency of histopathological analysis, there has been considerable interest in artificial intelligence (AI)-based diagnostic systems.4 These AI-based systems have the potential to assist pathologists by automating and streamlining various tasks, such as cell detection, segmentation, and classification.5–7
In this context, this study presents an AI-based generic pipeline for semantic cell segmentation in histopathological images from breast cancer. The primary goal of the pipeline is to accurately detect and classify cells in WSIs, enabling efficient analysis, and interpretation of large histopathological data. In addition, we also present several techniques or methods that have been investigated for research purposes in the project. The approaches described have been developed within the DigiPatICS project,8 which was set up by the Institut Català de la Salut (ICS) in collaboration with the Image Processing Group of the Universitat Politècnica de Catalunya and eight prominent hospitals in Catalunya.
Materials and methodsThe proposed pipeline (see Fig. 1) offers a simplified approach to the complex task of analyzing histopathological images, integrating various image processing and deep learning techniques in 3 principal stages: pre-processing, core processing, and post-processing. The process begins with the detection of tissue using mathematical morphology on a downsampled version of the WSI, which facilitates the identification of important regions for analysis. Subsequently, the identified tissue regions are divided into full-resolution tiles, and a simple convolutional neural network is employed to filter out tiles of no interest. Then, for accurate cell segmentation, different deep learning architectures are applied based on the specific pathology and staining methods used. Finally, the results from the individual tiles are combined to create a comprehensive representation of the entire WSI, facilitating the subsequent thorough analysis of the tissue by the experts.
Proposed WSI processing pipeline. Three main blocks: pre-processing, where the tissue is detected (a) and the tiling process and relevant tiles classification are performed (b); core processing, where the cell segmentation and classification algorithms are applied (c); 3) post-processing, where all the tile-level results are combined to offer WSI-level cell segmentation and classification (d).
This initial stage of the proposed pipeline for analyzing WSIs is designed to facilitate the efficient processing of such large and complex images, with a particular focus on relevant areas for diagnostic evaluation. This section provides an overview of the essential pre-processing steps, including tissue detection, tiling, and the selection of relevant tiles, which are crucial for optimizing computational resources and enhancing the accuracy of the subsequent analysis. By isolating significant tissue regions and managing the vast data contained within WSIs, the pipeline ensures that pathologists can effectively access and interpret critical diagnostic information. The following subsections will provide an explanation of each of these crucial steps in the analytical process.
Tissue detectionThis block in the proposed pipeline is of great importance for the efficient analysis of WSIs by focusing on tissue areas (see Fig. 1a). The objective is to generate a binary mask that accurately identifies tissue regions while excluding artifacts such as image borders, dirt, or markers that are not part of the actual biopsy. By focusing on areas with relevant content, this step significantly enhances the efficiency of subsequent analysis stages.
Initially, we explored the automated segmentation of tissue regions proposed by CLAM.9 However, we encountered challenges with image borders, dirt, and markers, as this method was not specifically designed to handle slides with these artifacts. Consequently, we developed our own method tailored to address the specific characteristics of images from the ICS hospitals, ensuring more accurate tissue detection in the presence of these common artifacts.
Our approach uses a downsampled version of the WSI as input, applying mathematical morphology techniques to this low-resolution image to distinguish tissue from other components. These techniques include several color transformations to enhance tissue visibility, filtering to reduce noise, and thresholding to separate tissue from background and other elements. This combination of methods enables robust tissue detection in a variety of image conditions.
Overall, this step is crucial for isolating tissue regions within the WSI, allowing subsequent analysis to focus on these areas. This focused approach not only increases the overall efficiency of the pipeline but also reduces the potential impact of non-tissue elements on the final results. By effectively separating tissue from artifacts and background, we create a solid foundation for more accurate and reliable analysis in the subsequent stages of the pipeline.
TilingOnce the regions of the slide containing tissue have been identified, it is necessary to process these regions. However, the original size of WSIs, which can reach dimensions of approximately 90.000 × 200.00 pixels at native resolution, makes it impossible to analyze the entire image in detail at once. To address this challenge, the WSI is divided into multiple high-resolution sub-images, or tiles (see Fig. 1b). This tiling approach allows our models to efficiently analyze the tissue regions in subsequent stages of the pipeline, providing a more focused and manageable examination that aids in diagnosis. The processing of these smaller tiles allows for efficient use of computational resources while ensuring that crucial tissue features are not overlooked.
Relevant tiles selectionThis phase of the pipeline focuses on determining the diagnostic relevance of the regions within the WSI. The objective is to identify tissue regions that contain significant information for analysis and to distinguish them from less relevant elements (see Fig. 1b). By implementing this classification step, the pipeline can focus subsequent analysis on only the most relevant tissue regions.
To achieve this goal, a neural network is used to examine each tile extracted in the preceding stage. The network has been trained to recognize and identify elements within the tissue that are relevant to the diagnostic task at hand. Upon receiving a tile as input, the model categorizes it based on its importance for the analysis, effectively separating relevant areas from those that are not diagnostically significant, such as fat tissue.
Core processingAfter the tiling process is complete and the relevant tiles have been identified, each tile undergoes further analysis using specialized algorithms (see Fig. 1c). As discussed earlier, our pipeline is specifically designed to handle immunohistochemical stained images from breast cancer patients. Thus, the core algorithms focus on segmenting and classifying cell nuclei from common diagnostic stains: HER2, Ki67, ER, and PR. These stains are essential for assessing breast cancer pathology, and our algorithms provide detailed segmentation and classification of the nuclei in the relevant tiles.
Membrane staining. The human epidermal growth factor receptor2 (HER2) stain is commonly used to detect the overexpression of HER2, a protein that regulates cell growth and division in breast tissue. Our approach categorizes cells into 4 distinct classes based on the intensity and completeness of the membrane staining, estimating the proportion of cells in each class to assign a diagnostic score. To achieve this, we trained a U-Net model10 for semantic segmentation of cell nuclei, following the approach presented by Mireia Boneta,6 classifying them into HER2 classes. Additional classes were created for non-epithelial cells (e.g., stroma) and background. Since some cell clusters may overlap, we applied the Watershed algorithm to correctly separate them into individual cells, ensuring accurate classification.
Nuclear staining. Similar to the HER2 membrane staining process, the goal of nuclear staining for Ki67, ER, and PR is to identify specific proteins within the cell nuclei. However, the presence of overlapping nuclei poses a significant challenge for segmentation. To overcome this, we proposed a Dual U-Net approach, extending the methodology used for membrane staining. In this case, the first U-Net performs the nuclei segmentation, while a second U-Net predicts the center of each nucleus, effectively addressing overlapping regions by detecting the individual cell centers. This approach, based on the work by Anglada-Rotger et al.,5 utilizes a density map and a contrast filter to accurately pinpoint the nuclei centers. The results from both networks are then combined to ensure the separation of overlapping cells, achieving precise segmentation and classification even in densely populated regions.
GNN approach. While the Dual U-Net approach effectively handles overlapping cells, when spatial relationships between cells are critical for classification, a different strategy can be employed—graph neural networks (GNNs).11 Unlike U-Nets, which primarily focus on pixel-level features, GNNs are designed to take into account the spatial proximity and interactions between cells by representing them as nodes in a graph, with edges encoding the relationships between neighboring cells. This is particularly useful when the classification of a cell depends not only on its individual characteristics but also on the features of nearby cells. For example, neighboring cells may share patterns that influence a cancer diagnosis, and GNNs are well-suited to capture these dependencies. By considering both local cell features and their spatial context, GNNs can provide a more nuanced analysis in cases where interactions between cells influence the overall pathology. Recent research has shown that GNNs can be applied successfully in cell segmentation and classification, offering a powerful alternative when spatial information must be integrated into the classification process.12
Finally, for all stains, it is essential to focus on epithelial tissue only, as the cells present in other tissues (such as stroma) should not be included in the final cell count. To ensure this, we designed a specific U-Net model trained to detect epithelial regions, based on the approach by Montse Pardàs et al.,13 excluding any non-epithelial cells from the analysis. This approach is crucial for accurate quantification of relevant cells and has been effectively used in prior studies on stromal tissue segmentation.
Post-processingThis final stage is a critical component of the WSI analysis pipeline, designed to consolidate and refine the results of the previous steps. This phase ensures that the insights derived from individual tile analyses are coherently integrated, thus providing a comprehensive understanding of the entire slide. In particular, the post-processing procedures address challenges such as the management of overlapping regions between neighboring tiles, the resolution of conflicting classifications, and the creation of a unified representation of the analyzed tissue (see Fig. 1d).
Consequently, for each analyzed tile, a mask of distinct connected components is stored along with assigned labels and predicted cell classes. This process is repeated for each tile, resulting in a collection of files that together represent the analysis of the entire WSI. The integration of these tile-level results allows for the generation of a coherent and accurate overall analysis of the WSI.
Results and discussionIn the previous section, we detailed the distinct components of our pipeline, providing a comprehensive overview of each step involved in analyzing WSIs. These stages, ranging from pre-processing to tissue analysis and post-processing, are designed to accelerate analysis while maintaining high accuracy and efficiency. This section provides a detailed analysis of the implementation of our pipeline and the practical application of the results obtained from WSI processing. We also describe how pathologists are effectively using these results in their diagnostic procedures, facilitating the integration of advanced computational analysis into clinical practice. Furthermore, we provide data on the number of WSIs processed by our pipeline, demonstrating its robustness and its potential to impact pathology practice.
The complete pipeline processes an entire WSI, focusing on relevant tissue regions, in an average time of 1110.1&#¿;±&#¿;619.79&#¿;s, with variations dependent on tissue quantity and area of interest. Pre-processing steps include tissue detection (5.25&#¿;±&#¿;1.85&#¿;s) and relevant tile selection (46.98&#¿;±&#¿;22.17&#¿;s total, or approximately 0.0292&#¿;s per tile). Core processing times differ for membrane and nuclear staining, requiring 0.2787&#¿;±&#¿;00156&#¿;s and 0.5452&#¿;±&#¿;0.0679&#¿;s per tile, respectively, due to distinct algorithmic approaches as detailed in Core processing. Post-processing consumes 0.3413&#¿;±&#¿;0.0429&#¿;s per tile, for a total of approximately 421.74&#¿;s for the complete WSI.
As previously stated, the proposed pipeline is designed to facilitate the efficient processing of large and complex images. This is accomplished primarily during the pre-processing stage, where the focus is on identifying relevant areas for diagnostic evaluation. It is noteworthy that through the process of tissue detection, an average of 86.17% (with a standard deviation of ±4.67%) of tiles from the entire scanned slide area are effectively discarded. Subsequently, the relevant tiles selection block enables the discernment and elimination of approximately 22.80% (±11.41%) of the tissue subimages that are deemed non-relevant. Consequently, following the completion of the pre-processing stage (comprising tissue detection and relevant tiles selection), the number of tiles in the WSI that have been discarded is 89.18% (with a standard deviation of ±4.37%).
Although the processing time may be perceived as slow, it is crucial to highlight that this occurs on servers after the slide has been scanned, typically during the night. This considerably enhances the efficiency of the pathologist's workflow, as the results can be accessed within seconds (typically within the range of 2–10&#¿;s, depending on the size of the selected area). This approach guarantees that pathologists can promptly access results, as the complex calculations have been completed in advance.
The results obtained from the pipeline are presented and analyzed by the pathologists using specialized software, the Clinical Viewer,14 developed by 3DHistech. This program is designed to facilitate the visualization of WSIs while providing robust analytical capabilities through various algorithms. Subsequently, pathologists can select specific regions of interest and display the calculated results for those areas, as seen in Fig. 2.
From the user's perspective, the workflow begins with the opening of a WSI and its subsequent analysis, with the objective of identifying one or more areas of interest. Once these areas have been identified, a dedicated plugin processes the precomputed results to generate the desired visualizations and quantifications for each case. This streamlined approach not only saves time but also allows pathologists to focus on critical areas that require their expertise.
The visualization component presents the segmentation of detected cells, with each cell color-coded according to its predicted class. This intuitive representation facilitates the rapid comprehension of the distribution and classification of cells within the sample by pathologists. Furthermore, a pop-up window displays quantifications or the computed indicators, such as the percentage of cells for each class and/or positivity scores, which vary based on the specific staining utilized. This detailed information is essential for making informed diagnostic decisions.
The implementation of each pipeline stage has enabled the development of an efficient, robust, and reliable approach to WSI analysis. The system has been fully implemented across 8 hospitals within the ICS network and has been successfully integrated into the routine histopathological workflows of these institutions. Moreover, it has evidenced consistent growth in utilization, as evidenced by the analysis of 5.639 WSIs across the 8 centers as of September 2023, and an increase to 8.471 by February 2024.
This approach has been demonstrated to be of particular value in the analysis of immunohistochemically stained images of breast cancer patients. The system provides objective, quantitative analysis that enhances diagnostic accuracy while reducing variability in interpretation. Quantifications for regions of interest are presented to pathologists during slide visualization within seconds, facilitating efficient diagnosis. Additionally, pathologists can combine results from several regions of interest, increasing the number of cells analyzed compared to manual methods and thus enhancing the statistical robustness of the derived indicators. Furthermore, the AI-driven approach offers consistent, unbiased assessments that complement the pathologist's expertise.
An essential aspect to consider is the clinical validation of our approach. The algorithms were initially trained on a database that was validated by medical professionals, ensuring a solid foundation for clinical relevance. Furthermore, the assistive use of these algorithms as a diagnostic support tool by physicians has corroborated their clinical validity in real-world settings. Finally, as part of the algorithm accreditation process, a more structured clinical validation is being conducted. This ongoing validation aims to provide additional evidence of the algorithms' effectiveness and reliability in clinical practice.
ConclusionsThis study presents an AI-based pipeline for the semantic segmentation and classification of cells in histopathological images, developed within the DigiPatICS project. The pipeline has been demonstrated to successfully automate the detection and classification of cells stained with HER2, Ki67, ER, and PR in breast cancer biopsies. By integrating image pre-processing, segmentation, and classification, the system produces precise results, enhancing the efficiency of the diagnostic process.
The pipeline's incorporation into the routine workflow of hospitals represents a substantial advancement in the application of engineering principles to histopathological analysis. By combining efficient processing with rapid result delivery and the ability to analyze larger cell populations, we have developed a tool that enhances the capabilities of pathologists in breast cancer diagnosis and assessment. The increasing adoption of the system across numerous hospitals serves to reinforce its reliability and clinical value, marking a pivotal advancement in the confluence of medical imaging and AI.
The project is currently in progress, with the development team and pathologists engaged in continuous collaboration. The pathologists provide invaluable feedback on both the diagnostic results and the overall usability of the pipeline, which is subsequently utilized for optimizing the pipeline's performance and enhancing its analysis accuracy. Moreover, there are ongoing research initiatives aimed at extending its application to lung and stomach (gastric) cancer tissues, thereby enhancing the project's scope and impact.
