Augmented reality is a technology that opens new possibilities in surgery. We present our experience in a hepatobiliary-pancreatic surgery unit in terms of preoperative planning, intraoperative support and teaching.
For surgical planning, we have used 3D CT and MRI reconstructions to evaluate complex cases, which has made the interpretation of the anatomy more precise and the planning of the technique simpler.
At an intraoperative level, it provides for remote holographic connection between specialists, the substitution of physical elements for virtual elements, and the use of virtual consultation models and surgical guides.
In teaching, new lessons include sharing live video of surgery with the support of virtual elements for a better student understanding.
As the experience has been satisfactory, augmented reality could be applied in the future to improve the results of hepatobiliary-pancreatic surgery.
La realidad aumentada es una tecnología que abre nuevas posibilidades en cirugía. Se presenta su implementación en una unidad de cirugía hepato-bilio-pancreática en relación con la planificación preoperatoria, el soporte intraoperatorio y la docencia.
Para la planificación quirúrgica se han utilizado reconstrucciones 3D de TC y RMN para hacer una evaluación de casos complejos, siendo la interpretación de la anatomía más precisa y la planificación de la técnica más simple.
A nivel intraoperatorio ha permitido la conexión remota holográfica entre especialistas, la substitución de elementos físicos por elementos virtuales y el uso de modelos virtuales de consulta y guía quirúrgica.
En docencia se han impartido clases que incluyen la retransmisión de una cirugía con el soporte de elementos virtuales para una mejor comprensión por parte de los estudiantes.
Siendo la experiencia satisfactoria, la realidad aumentada podría aplicarse en el futuro de la cirugía hepato-bilio-pancreática para mejorar sus resultados.
Hepatobiliary-pancreatic surgery (HBP) is a discipline in constant evolution. Surgical techniques used today have radically changed in recent years. This evolution has been due to technological developments, such as the implementation of minimally invasive surgery or robotic surgery,1–3 which provide similar oncological results but have reduced morbidity, mortality and hospital stay,3–5 or telemedicine.6
However, the evolution in patient management has generally been based on improving surgical procedures or perioperative protocols.7 Other factors, such as perioperative planning or the participation of experts in remote surgery, have not undergone relevant changes in recent years.
Augmented reality (AR) is a tool that complements the surgical setting with virtual 3-dimensional elements or holograms through the use of specific devices, such as augmented reality glasses.
The implementation of AR in surgical practice has been described in pilot studies in areas such as communication with the patient, surgical planning or intraoperative support.8–10 Consequently, the use of AR could mean a change in the way in which patients undergoing HBP surgery are studied, planned, operated on and followed up, facilitating and optimizing the work of the surgeon.
Recent studies have also described the benefits of using AR for teaching purposes.11,12 The conventional teaching method used at the university level, in professional courses and medical conferences has changed little in recent decades, as they have been excluded from the great technological revolution that the surgical sector has experienced. Although the COVID-19 pandemic has forced us to change paradigms, such as in-person attendance in classrooms, conferences, or operating rooms, there is still room for improvement.13 Explaining HBP surgical procedures involves anatomical and technical know-how that is difficult to teach and learn with conventional teaching resources.
To evaluate the utility and possible implementation of AR in a Hepatobiliary and Pancreatic Surgery Unit, we designed a cross-sectional study. For 12 months, this technology has supported three main areas: surgical planning, intraoperative support and education.
The study was carried out at the Hospital Germans Trias i Pujol, a tertiary university hospital located outside of Barcelona (Catalonia, Spain) and a referral center for HBP surgery in an area of more than 800 000 inhabitants.
Regarding the surgical training of resident physicians, our hospital works with the Universitat Autónoma de Barcelona (UAB), one of the largest universities in Spain, with 200 medical students in our teaching unit.
AR was introduced in January 2021 with the purchase of 2 augmented reality glasses (Hololens 2® - Microsoft®) that work with Holomedicine® (Apoqlar®) medical software. This software provides various AR-based functionalities, such as displaying holographic files, CT scans, MRIs or 3D models and being able to share one’s operating room environment, complemented by these virtual resources.
In our unit, 6 hepatobiliary and pancreatic surgeons were trained to use these devices in their routine practice. Regarding teaching, we proposed a program of 3 pilot classes using AR as an additional resource to the 200 medical students of the hospital campus.
Surgical planningConventional surgical planning in HBP surgery is performed with CT and MRI images. Some groups also use 3D-printed models when faced with complex cases. These models can facilitate surgical planning as they provide a greater anatomical understanding of the case to be treated. However, since the acquisition of AR technology, our group stopped using 3D-printed models and we have decided to only use virtual 3D models to evaluate our complex cases.
AR was used to prepare 11 complex oncological surgery cases from February 2021 to February 2022. These included 5 liver resections, 4 extreme liver surgeries requiring vena cava resection and replacement, and 2 pancreatic procedures with vascular compromise.
In all cases, the available studies (CT, MRI and 3D models) were evaluated using virtual holograms, and discussion of the anatomy and procedures were based on these.
All participating surgeons (n = 6) subjectively expressed that preparing for surgery with these virtual holograms was much easier than with the conventional 2D display, even when 3D volumetric models were included (Fig. 1).
Intraoperative support
AR has provided us with 3 intraoperative advantages (Fig. 2). First, it allowed us to visualize and interact in a holographic way with the patient's complementary tests, such as CT, MRI, 3D models or relevant information from patient medical records. And all this happens under sterile conditions. Secondly, this technology allows for mixed-reality collaboration by being able to share the surgical procedure in real time with a consultant surgeon in another location or hospital. In addition, this second surgeon can see our operating room virtually and give advice electronically as if physically there. Finally, in minimally invasive surgeries, where many devices are needed and space in the operating room is limited, by using AR glasses we have been able to replace physical monitors with virtual ones.
We conducted a subjective qualitative assessment of AR holographic resources during surgery. Surgeons highly value the ability to consult patient tests and 3D models during surgery without having to access external devices (computers or physical monitors) and being able to view them under sterile conditions. Specifically, the ability to superimpose the holograms onto the patient's actual anatomy, a concept known as superimposition, is the most significant improvement over conventional resources, as it has the potential to provide intraoperative guidance (Fig. 3).
On 2 occasions, a real-time connection was made between 2 surgeons so that a junior surgeon could seek advice from a consultant surgeon located in another hospital. In this case, it was much easier for them to explain the problem and to understand the response of the consultant surgeon than if it had been done by phone or even by video call (Fig. 4).
Lastly, in several surgeries the physical monitors were replaced by virtual monitors using the AR glasses. This resulted in more free space in the operating room, and surgeons were able to work in a more comfortable and ergonomic position. This is advantageous in robotic surgeries, where the instruments, monitors and surgeons must sometimes be placed in uncomfortable positions. In cases of robotic cholecystectomy with intraoperative choledochoscopy, this was visualized virtually by the surgeon using the AR glasses, making it possible to dispense with the corresponding monitor (Fig. 5).
Surgical training
We decided to use AR technology to stream classes from the operating room, combining a theoretical introduction with real-time transmission of the surgery through the AR device, while also being able to use complementary virtual teaching content to explain complex procedures and discuss cases with students.
During the second semester (September 2021 to January 2022), 3 pilot classes were held. Lessons included open pancreatoduodenectomy, robotic hepatic segmentectomy, and laparoscopic cholecystectomy; 72 UAB medical school students were able to connect simultaneously. A surgeon not directly involved in the surgery broadcasted the procedures and answered the students' questions, supported by images, videos and 3D holographic models (Fig. 6).
Students were given a satisfaction survey that consisted of 12 questions, 4 of which were related to the technical quality of the images and 8 about certain teaching elements. In total, 42 responses (58.3%) were received. Video and AR images scored well. Regarding the academic evaluation, 35 students (83.3%) considered that the virtual content facilitates the understanding of the surgery, and 27 (64.3%) responded that, with this technology, it was easier to see and understand the surgeries than when physically in the operating room. In addition, 40 students (95.2%) would like this new type of teaching to be included in official course programs.
Discussion of the experienceAR is a relatively new technology that combines real environments with virtual holograms that can be interacted with as if they were real, which can improve outcomes and the perceived experience in HBP surgery.
With regards to surgical planning, the anatomy of patients and the relationships between organs and neoplastic lesions are evaluated with radiological images (CT or MRI). This requires a great deal of experience, since some images can be difficult to understand, and the mental models that we each create can differ from the actual findings during surgery.14 Recently, a new paradigm has appeared with the introduction of 3D models and 3D printing. Obtaining 3D virtual models from CT/MRI was the first step towards an easier, faster and more thorough understanding of a patient's anatomy, which facilitated the planning of surgical procedures.15–17 However, the 3D models had to be viewed on 2D monitors and 3D printed models are solid and static, not manipulable, which limits their usefulness. With AR, these 3D models can be viewed as holograms that can be manipulated as if they were real. The composition of these models can also be modified in real time, adding or subtracting anatomical components. Therefore, the anatomical evaluation is immediate, which reduces the necessary previous experience and eliminates, in a certain way, the differences in understanding among surgeons. This is especially useful in volumetric?/bulky organs, such as in liver surgery, where metastases within the parenchyma can be difficult to locate or interpret. In addition, compared to printed models, it considerably reduces the waste generated.
While AR is already a reality, the most interesting factor is the projection of this line of work. There are already prototypes of virtual surgical planning programs to establish lines of transection, surgical margins or automated volumetric calculations. All this has two clear implications: first, it allows us to conduct surgical procedures with much greater preparation, which reduces the risk of adverse events or unforeseen situations; and second, it allows us to advance in the concept of personalized medicine. Another characteristic of interest is the intraoperative support that otherwise is currently non-existent.
The next logical evolutionary step in intraoperative navigation is the superimposition of 3D virtual models on the patient's anatomy. As Saito et al. have stated, the intraoperative identification of relevant structures and/or the location of neoplastic lesions in volumetric organs like the liver are benefits that provide faster and more reliable intraoperative evaluations.18
However, our experience with superimposition, while promising, has revealed several technological limitations. Current models do not provide automatic placement on the patient's anatomy, which means that it must be done manually, and any movement of the target organ causes synchrony with the model to be lost. Similarly, any deformation of the target anatomy renders the virtual model unreliable. The definition of 3D holograms is also expected to improve since, to be useful as an intraoperative guide, it is necessary to be able to visualize more details than they currently offer.
In contrast, the virtual support resources provided (3D models, 3D CT or MRI images, medical records, demonstrative videos or even virtual monitors) for minimally invasive surgeries in a sterile field are advancements that facilitate the surgical procedure. Likewise, at a time when the advent of robotic surgery limits space in the operating room and increases costs, the ability to replace physical monitors with virtual monitors is especially advantageous. Although existing latencies and definitions must also be improved, current solutions are good enough to be put to use and move towards operating rooms that are free of physical monitors.
Furthermore, at a time when telemedicine has made necessary progress due to the pandemic, the concept of “virtual mentoring” offered by AR is interesting.9 What is exciting about this new experience is the ability to have the virtual presence of the senior surgeon practically in the operating room with the junior surgeons (unlike conventional video calls), thereby maximizing the ability to communicate and understand the case.
Although this requires good communication channels and expert teams to achieve an adequate connection, we believe that AR is the future line of collaboration between institutions, especially in healthcare systems based on primary or secondary hospitals that are associated with high-complexity HBP tertiary referral centers.
Finally, one of the most interesting aspects of this technology applied to complex procedures like HBP surgery is its use in education, both at the university level and in professional courses or medical conferences.
Our experience with AR in 3 HBP procedures, specifically one that was low complexity (cholecystectomy) and 2 highly complex procedures (PD and major hepatectomy), was of great interest and acceptance. This technology allowed a large number of students to attend surgery simultaneously, with a good field of vision and without the need to commute. In our opinion, in-person visualization of the surgical field during open surgery is generally limited due to a greater number of people in the operating room. In the case of minimally invasive surgeries, this limitation is due to the lack of physical space to situate monitors as well as students. AR resolves this issue by being able to stream the surgical procedure on external devices, while still maintaining the ability to communicate.
This fact, by itself, would not add much value compared to conventionally sharing a surgery by video call. However, what does make a difference are the virtual resources that complement the online streaming of the surgery. As described in areas such as simulation or teaching anatomy,19,20 being able to view virtual images, diagrams, explanatory videos and 3D models while viewing the procedure led most of the participants to have a better understanding of the surgery than when in person in the operating room.21
For the reasons described, and with the experience acquired to date, we believe that AR technology will enable us to make a qualitative leap in HBP surgical procedures. However, further studies are required to provide specific scientific evidence in each of the lines of work mentioned.
FundingThis study has received no specific funding from public, commercial or non-profit organizations.
Conflicts of interestsManel Cremades Pérez collaborates as a consultant for Apoqlar GmbH but receives no remuneration.
Esteban Cugat Andorrà colaborates as a consultant for Cella Medical Solutions SL but receives no remuneration.
