The use of mixed reality in the preoperative planning of colorectal surgery: Preliminary experience with a narrative review

Abstract

New advanced technologies have recently been developed and preliminarily applied to surgery, including virtual reality (VR), augmented reality (AR) and mixed reality (MR). We retrospectively review all colorectal cases in which we used holographic 3D reconstruction from February 2020 to December 2022. This innovative approach was used to identify vascular anomalies, pinpoint tumor locations, evaluate infiltration into neighboring organs and devise surgical plans for both training and educating trainee assistants. We have also provided a state-of-the-art analysis, briefly highlighting what has been stated by the scientific literature to date. VR facilitates training and anatomical assessments, while AR enhances training and laparoscopic performance evaluations. MR, powered by HoloLens, enriches anatomic recognition, navigation, and visualization. Successful implementation was observed in 10 colorectal cancer cases, showcasing the effectiveness of MR in improving preoperative planning and its intraoperative application. This technology holds significant promise for advancing colorectal surgery by elevating safety and reliability standards.

Resumen

Recientemente, se desarrollaron nuevas tecnologías avanzadas que se aplicaron preliminarmente a la cirugía, como la realidad virtual (RV), la realidad aumentada (RA) y la realidad mixta (RM). Revisamos retrospectivamente todos los casos de cirugía colorrectal en los que utilizamos, desdefebrero de 2020 hasta diciembre de 2022, una reconstrucción holográfica en 3D. Este enfoqueinnovador tuvo como objetivo identificar anomalías vasculares, señalar ubicaciones de tumores, evaluar la infiltración en órganos vecinos y diseñar planes quirúrgicos tanto para el entrenamientocomo para la formación de asistentes en entrenamiento. También proporcionamos un análisis de última generación hasta ahora, resaltando brevemente lo que ha sido mencionado por la literaturacientífica hasta el momento. La RV facilita el entrenamiento y las evaluaciones anatómicas, mientras que la RA mejora el entrenamiento y las evaluaciones del rendimiento laparoscópico. La RM, impulsada por HoloLens, enriquece el reconocimiento anatómico, la navegación y la visualización. Se observó una implementación exitosa en 10 casos de cáncer colorrectal, demostrando la efectividad de la RM en la mejora de la planificación preoperatoria y su aplicaciónintraoperatoria. Esta tecnología tiene un gran potencial para avanzar en la cirugía colorrectal al elevar los estándares de seguridad y confiabilidad.

Introduction

Elective gastrointestinal surgery is usually performed using a minimally invasive approach, by experienced surgeons. Several advantages have been attributed to laparoscopy in terms of postoperative morbidity and length of hospital stay.¹ In malignant disease, a preoperative plan based on clinical, radiological, and endoscopic assessments is needed to better understand the vascular anatomy, tumor location or node involvement and to facilitate the surgical procedure. As laparoscopy has some inherent drawbacks compared to open surgery, such as limited tactile feedback and the inability to perform transillumination tests, a comprehensive evaluation of the patient’s condition can decrease intraoperative complications and improve surgical planning.²

For instance, the vascular anatomy of the right colon varies considerably among patients.^3,4 Similarly, there is a non-negligible discrepancy between preoperative diagnostic procedures and intraoperative findings.^2,5 Computed tomography (CT) and magnetic resonance imaging (MRI) are considered the gold standard for preoperative planning. However, they provide 2D visualization of patient anatomy, causing possible inconsistencies between what is preoperatively observed and what is found intraoperatively.⁵ Furthermore, these images can be difficult to interpret, especially for surgical trainees.

Recently, advances made in image analysis technology have led to the development of various surgical simulations and navigation systems based on 3-dimensional (3D) image construction. Patient-specific virtual models are generated from cross-sectional imaging studies, such as CT and MRI scans.⁶

These models can be visualized using virtual reality (VR), augmented reality (AR), and mixed reality (MR). This technology has found extensive use in medical disciplines, such as orthopedics, urology, and hepato-pancreatic surgery.^7–11 However, its utilization in colorectal surgery still remains limited.

The visualization of 3D models could provide more information regarding vascular anatomy, tumor location and infiltration, aiding young and inexperienced surgeons to plan their interventions.^12,13 Additionally, 3D reconstructions can be used for intraoperative navigation, providing real-time assessment.

The aim of the present study is to describe our preliminary experience using preoperative holographic mixed reality, describing its potential applications and advantages in the setting of gastrointestinal surgery. We also provide a state-of-the-art analysis of the scientific literature to date (Table 1).

Table 1.

Details of the included studies.

Author	Year	Type of reality	Participants	Radiological modality	Tumor localization	Surgical procedure	Application	Main findings
Andersen et al.	2021	VR	32	CT	Colorectal cancer	Right/left colectomy	MCA bifurcation anatomy	The wide range of possible MCA bifurcation positions makes 3D models valuable for surgeons, enabling precise identification of vascular structures.
Maki et al.	2016	VR	331	CT	Colon cancer	Transverse colectomy	Preoperative planning	3DCTA is useful in evaluating the anatomic variants of the MCV in preoperative planning for laparoscopic surgery.
Leblanc et al.	2010	AR	29	CT/MRI	Colorectal cancer	Laparoscopic sigmoid colectomy	Using AR to compare HAL approach with SL	The improved scores with the hand-assisted approach (HAL) suggest that, with this simulator. A hand-assisted model may be technically easier to perform.
Guerriero et al.	2018	VR	3	CT	Diverticular disease and colon cancer	Left and right hemicolectomy	Assessment of vascular anatomy	Virtual reality can provide an enhanced understanding of crucial anatomical details, both preoperatively and intraoperatively, which could contribute to improve safety in colorectal surgery.
Soriero et al.	2022	3D reconstruction/VR	14	CT/MRI/colonoscopy	Colorectal cancer	Colon resection	Colorectal lesion localization	The 3D models could present good potential to improve lesion localization and help surgical planning, especially in difficult cases.
Leblanc et al.	2010	AR	38	CT/MRI	Colorectal cancer	Laparoscopic sigmoid colectomies	Using AR to compare HAL approach with SL	A HAL approach for sigmoid colectomy accelerated colonic mobilization and anastomosis.
Ryu et al.	2022	MR	13	CT/holograms	Colorectal cancer	Laparoscopic surgery	Preoperative anatomical study	The use of mixed reality, compared to preoperative 3D reconstruction alone, in complex colorectal surgery has resulted in improved surgical performance, enhanced anatomical clarity, reduced cognitive demand, maintained sterility and 3D visualization.
Luzon et al.	2019	VR	24	CT	Colon cancer	Right colectomy	Preoperative anatomical study	The vast majority of SAM did not contain all SMAV branches relevant to the surgeon performing right colectomy with extended D3 mesenterectomy.
Nepal et al.	2021	VR	167	CT	Colorectal cancer	Left colectomy	Preoperative anatomical study	3D-CT angiography is useful for the preoperative assessment of vascular anatomical variations of the IMV and IMA for safe intraoperative navigation and prevention of complications during laparoscopy-assisted left-sided colorectal surgery.
Matsuhashi et al.	2022	VR	2	CT	Rectal cancer	Lateral lymph node dissection	Images of pelvic anatomy (especially vascular anatomy)	SYNAPSE VINCENT aids in the identification of vascular anomalies and aids in lymph node dissection.
Horie et al.	2018	VR	10	CT	Rectal cancer	Laparoscopic lateral pelvic lymph node dissection (LPLD)	Anatomic reconstruction for preoperative planning	Preoperative 3D simulation may be useful to safely perform laparoscopic LPLD, especially for surgeons with limited prior experience.
Raghupathi et al.	2004	VR	None	Computer modelling	Colorectal cancer	None	Training	The novel anatomical model greatly simplifies the problem of simulating a complex deformable organ in real time.
Luzon et al.	2020	MR	6	CT/holograms	Colorectal cancer	Open right colectomy with CME	Vascular anatomy and extended lymphadenectomy	The effective on precision can be evaluated based on the surgeon’s sightline, which improves when it is perpendicular in all directions (smaller TED).
Fletcher et al.	2021	3D reconstruction/VR	15	CT	Colorectal cancer	Right colectomy and CME	Vascular anatomy and extended lymphadenectomy	Higher scores of anatomical and vascular understanding for young surgeons performing right hemicolectomy and complete mesocolic excision (CME).
Huber et al.	2019	MR	1	CT	Colorectal cancer	Robotic TaTME	Training and preoperative planning

Methods

We have retrospectively reviewed all colorectal cases in which we used holographic 3D reconstruction for preoperative planning of patients with colorectal tumors from February 2020 to December 2022. We used Virtual Reality Exploration, a technology that has been previously used by others for gastrointestinal tract surgery.¹⁴ The aim of our study was to identify any vascular anomalies, locate the tumor, assess its infiltration to surrounding organs, and evaluate surgical plans for training and educational purposes for trainee assistants. We have also provided a state-of-the-art analysis that briefly highlights what has been published in the scientific literature to date.

We used ARTICOR® software, which was developed to generate pre- and intra-operative simulations of patient-specific 3D models with mixed reality interaction technologies. In our specific case, the images were acquired from CT scans and processed through an experimental semi-automatic procedure, enabling 3D reconstruction of the patient’s anatomy and resulting in a three-dimensional holographic model.^15,16

ARTICOR® also allows for multi-user sessions of holographic models to be shared for the discussion of clinical cases, utilizing a simulation tool — in our case, the HoloLens 2¹⁷ — which enables interaction with holograms through instinctive gestures, like manipulating real-world objects. All of this is made possible using a Time of Flight (ToF) depth sensor, coupled with integrated artificial intelligence and semantic understanding. The presence of Windows Hello authentication recognition further facilitates sharing the viewer among multiple operators. This type of viewer allows for 2 types of interaction: augmented reality (AR) and mixed reality (MR), thanks to the powerful Qualcomm Snapdragon 850 processor it is equipped with. The optics feature transparent holographic lenses with a resolution of 2K, allowing for a holographic density of 47 pixels per degree of vision, which doubles the field of view compared to its previous version. It is equipped with head and eye tracking, an accelerometer, gyroscope, magnetometer, an 8 MP camera, as well as Wi-Fi 802.11 ac, Bluetooth 5.0, and USB-C connectivity.

Results

Definition and description of VR, AR and MR with a narrative review

3D virtual reality

VR is an all-enveloping artificial and fully immersive experience, which completely integrates the user into a computer-generated environment.⁷ A fundamental role of VR is its training function, and 2 studies delve into this aspect.^18,19 VR is increasingly playing a crucial role in medical schools and specialized training as a means of education and training. Vertemati et al.¹⁸ has stated that VR training was better than none. The comparison between VR and video trainers was statistically in favor of VR; likewise, Raghupathi et al.¹⁹ have also stated that VR simulator platforms can significantly help young surgeons and medical residents acquire the necessary surgical skills in a low-cost manner. The training function of VR is further emphasized as it is useful both in anatomical assessment, particularly for identifying vascular abnormalities, and in surgical planning. Other authors^12,20,21 have reported the importance and advantages of VR in the preoperative identification of vascular anomalies concerning the ascending and transverse colon. Andersen et al.²⁰ have discussed the utility of VR in preoperatively identifying anatomical variations involving the middle colic artery. As highlighted in previous articles,^20,21 the positive contribution of 3D reconstruction is emphasized when used for anatomical understanding (particularly vascular) in patients with ascending colon carcinoma undergoing right hemicolectomy and complete mesocolic excision (CME). The same is reported for left colon anatomy, in which Nepal and Guerriero et al.^12,14 emphasize that 3D-CT angiography is useful for the preoperative assessment of vascular anatomical variations of the IMV and IMA. They concluded that this technology seems to be useful for safe intraoperative navigation and prevention of complications during laparoscopy in left colorectal surgery.^11,13 Other studies reported the utility of VR in preoperative planning and tailoring lymphadenectomy based on the specific anatomy of each patient.^22–24 Also, Soriero et al.²⁵ reported that the use of 3D models could present good potential to improve tumor localization and help surgical planning, especially in difficult cases.

Augmented reality

AR refers to a concept of superimposing visual information onto the user’s field of vision using, for instance, head-mounted displays (HMD) or glasses.⁷

Leblanc et al. have employed augmented reality (AR) in 2 distinct studies.^26,27 In both studies, the trainees were asked to perform 2 laparoscopic sigmoid colectomies on an AR simulator (ProMIS 2.5, Haptica, Dublin, Ireland), randomly starting with either the straight laparoscopic (SL) or hand-assisted laparoscopic (HAL) approach. In the first study, the utilization of AR was able to assess differences between the 2 techniques in terms of errors and operative time, demonstrating how the HAL approach can expedite the mobilization and anastomosis of the colon. In the second study, they highlighted the need for a period of adaptation to the simulator to enhance simulated laparoscopic performance. However, only 2 articles were identified regarding the use of AR in colorectal surgery. This is likely attributed to the widespread utilization of AR for training purposes among trainees, who primarily use augmented reality simulators for performing simple exercises.

Mixed reality

MR takes the AR concept one step further compared to VR and AR by spatially correlating the real environment with digital objects.⁷ In the case of HoloLens, it overlays images onto a scene or object behind the display, which are referred to as “holograms”.¹⁶ HoloLens is operated with hand gestures and voice commands, enabling surgeons to view 3D holographic images of a patient’s anatomy created from X-rays or other scans, for example CT. Surgeons can move those virtual images around to see them from different angles.

To date, MR with the introduction of HoloLens has gained more attention in various surgical fields as it is commercially available and capable of adding new layers of information regarding surgical anatomy. At the same time, some studies are interested in evaluating its accuracy and precision. Luzon et al.²² have assessed this aspect, evaluating the use of mixed reality navigator for open surgery (MRNOS) by 6 surgeons during right colectomy with extended lymphadenectomy, introducing an important perceptual variable for evaluating surgical accuracy and precision: the surgeon’s sightline. The surgeon’s line of sight is an imaginary line between the surgeon’s eye and the target/hologram. The efficacy and performance were assessed in terms of target error distance (TED), measured in millimeters. A TED value between 0 and 0.5 mm was considered to have hit the bullseye. In the results, if the surgeon was able to maintain a perpendicular position relative to the operating table in both steps and along all axes, the TED was lower, indicating higher accuracy and precision, while also indicating the perpendicular position as the optimal position. Ryu et al.²⁸ investigated the feasibility and potential of holograms for intraoperative navigation in colorectal cancer, comparing this with 3D reconstruction alone. The study included 13 patients who underwent laparoscopic colorectal surgery. The surgical outcomes were found to be comparable in both groups. In all cases, laparoscopic surgery was performed safely while observing the hologram, which the surgeon could manipulate and still maintain sterility. This approach provided better visualization, particularly in patients undergoing right hemicolectomy, where there was greater anatomical variability. Senior surgeons exhibited significantly lower scores compared to junior surgeons in terms of workload, mental demands, time required, and effort. However, there were no significant differences between the 2 groups in terms of physical exertion, performance, and frustration scores. Similarly, Huber et al.²⁹ reported how the use of MR facilitates the surgical field in a patient undergoing robotic transanal total mesorectal excision. This applies to both experienced surgeons and surgeons-in-training, improving surgical precision, spatial awareness, intraoperative navigation, personalized surgical planning, and enhancing interaction among the surgical team. Additionally, the perceived workload (NASA TLX) was reported to be very low for all participants.

Our preliminary experience

We used holographic 3D reconstruction and MR visualization in 10 complex cases of colorectal cancer, utilizing the HoloLens MD System (Table 2).

Table 2.

Our experience, patient characteristics.

Year	Patient	Age	Pathology	Type of surgery	Use	Year
2021	F	53	Colorectal cancer	Anterior resection	Vascular anomalies	2021
2022	M	81	Colorectal cancer	Extended left hemicolectomy	Vascular anomalies	2022
2021	M	59	Colorectal cancer	Extended left hemicolectomy	Vascular anomalies	2020
2021	F	71	Colorectal cancer	Rectum anterior resection	Vascular anomalies	2021
2022	M	70	Colorectal cancer	Left hemicolectomy	Vascular anomalies	2022
2021	F	39	Colorectal cancer	Right hemicolectomy	Tumor localization	2020
2021	M	48	Colorectal cancer	Right hemicolectomy	Tumor localization	2021
2022	F	64	Colorectal cancer	Left hemicolectomy	Tumor localization	2022
2022	M	46	Colorectal cancer	Right hemicolectomy	Locoregional invasion	2022
2021	F	57	Colorectal cancer	Anterior resection	Locoregional invasion	2020

In 3 of these cases, it was used to identify vascular anomalies, including one of the first colorectal cases in which we used mixed reality. The holograms were very useful to identify the lack of left colic artery and very high splenic flexure, which we have observed in another 2 cases. This information changed our surgical approach, preserving the inferior mesenteric artery during left colectomy for diverticular disease. In 3 other cases, it was used for the accurate localization of a tumor. In one patient, the initial tumor location indicated by CT was in the descending colon, but 3D reconstruction later confirmed the location in the splenic flexure, which resulted in a change of surgical approach from left hemicolectomy to splenic flexure resection. In 2 cases, we used the technique to evaluate locoregional invasion of rectal cancer, and in a young male patient, it was used to evaluate the extent of mesorectal invasion, which was not clearly visible on MRI (Fig. 1).

Fig. 1.

3D reconstruction of rectal adenocarcinoma.

In another case, a patient with right colon adenocarcinoma, on CT scan appeared to be invading the liver; with the assistance of mixed reality, subtracting the surrounding organs, it was then observed and demonstrated that it was not involved (Fig. 2).

Fig. 2.

3D reconstruction of right adenocarcinoma.

Discussion

The analysis of images and pre/intraoperative surgical navigation are continuously evolving, and their application fields have expanded beyond medical disciplines to include surgical branches such as urology, orthopedics, and maxillofacial surgery. However, for digestive surgery, this technology is still under development. It has gained increasing recognition in the surgery of the parenchymal organs, such as the liver, but its role in colorectal surgery, specifically its potential benefits in oncologic surgery of the colon and rectum, has still not been defined. We have identified the current fields of application for these new technologies, and this can be helpful to all surgeons. The use of 3D reconstruction and 3D printers has been well documented in the literature.^5,7,28 However, these methods have specific limitations, including the cost of materials and the inability of surgeons to interact with the reconstructions. This limitation is easily overcome by MR, particularly compared to the 2 other possible visualizations, VR and AR. MR allows the surgeon to interact with the 3D model while maintaining sterility and without leaving the surgical field. Using the HoloLens MD system, the surgeon can modify the hologram, viewing only what they want to see, subtracting, moving, and changing their perspective (Fig. 3).

Fig. 3.

Hologram superimposed on the patient in the operation room.

The application of minimally invasive surgery for colorectal cancer has expanded to complex and challenging cases. These new technologies are easy to use and do not increase the workload.^28,29 They have been found to play an increasingly important role in preoperative planning, better and accurate tumor localization, assessment of actual invasiveness, and identification of defined surgical dissection planes (Fig. 4). They enhance training and education for younger and inexperienced surgeons, providing a deeper understanding of anatomy, particularly vascular anatomy, in regions where there is a higher incidence of vascular anomalies, such as the ascending colon.^11,13,28 Furthermore, the hologram and the modifications made by the surgeon can be shared wirelessly with multiple individuals, making them useful in conferences among surgical teams and in the training of young doctors, providing in-depth understanding prior to the procedure, tailored to each patient. In our experience, we have used MR visualization in complex cases and before certain live surgeries to identify vascular anomalies, obtain more accurate localization of a tumor, and assess tumor extension. We can affirm that all the listed applications have proven to be effective, making surgery safer and more reliable. In addition, preoperative identification of vascular anomalies, which are quite common, especially in the right colon, facilitates safer vascular dissection and reduces the risk of bleeding.

Fig. 4.

Hologram visualization for preoperative planning.

Moreover, preoperative information about splenic flexure configuration and descending colon length could change the surgical strategy, for example, in surgeons with different strategies during anterior resection of the rectum.

In conclusion, the use of 3D models and holograms enable better preoperative planning and could potentially be used during the intraoperative phase. In our opinion, the application of these technologies in colorectal surgery could be promising.

Conflicts of interest

None.

Use generative artificial intelligence (AI) and AI-assisted technologies in the writing process

None.

Financial support

None.