The surgical metaverse

Abstract

Recent developments have seen acceleration in the development of virtual reality (VR) and augmented reality (AR) technology. Through these developments, the metaverse has emerged. Within the metaverse, users create an avatar to experience an immersive, interactive extended reality. Current front-runners in its implementation are the financial, communication and entertainment sectors. This technology, however, is receiving greater recognition in the medical world, with national and international surgical bodies acknowledging the benefits that VR and the metaverse will have on surgical training and patient care. Its versatility means that we are likely to see VR related technology become intimately woven into the fabric of surgery in the next two decade. In this article, the impact of the metaverse on the surgical field will be discussed.

Resumen

El día moderno ha visto una gran aceleración en el desarrollo de la tecnología de la realidad virtual (RV) y de la realidad aumentada (RA). Con este desarrollo, ha nacido el ‘metaverso’. En el metaverso, los usuarios pueden crear un avatar para experimentar una realidad ampliada que es inmersiva e interactiva. En este momento, las industrias que están marcando el camino son de la comunicación y el entretenimiento. Sin embargo, en el campo de la medicina, la RV es más apreciada. Las organizaciones internacionales quirúrgicas han reconocido los beneficios que la RV y el metaverso tendrán con respecto al aprendizaje quirúrgico y el tratamiento del paciente. Es probable que se vea la RV y las tecnologías similares integradas en la práctica quirúrgica en la próxima década por su versatilidad y utilidad. En este artículo, trataremos el impacto del metaverso en la cirugía.

Introduction

For patients undergoing surgery, placing their lives in the hands of the surgical team is a tremendous act of trust and an understandably daunting experience. Therefore, we as surgeons are constantly looking for ways to improve the patient experience and outcomes. For example, the era of ‘minimally invasive’ surgery recently changed the landscape of abdominal, joint and vascular surgery. Now, we look towards virtual reality (VR) related technology to leave the latest footprint on the surgical world. Recent rapid developments have already started to change the way surgery is practised. The Royal College of Surgeons’ Future of Surgery Report has reinforced this prediction and outlines some recommendations.¹ Decreased exposure to operating for trainees, increased patient involvement, and surgical assessment are areas for which virtual and augmented reality (AR) have been highlighted as possible solutions.

Understanding the subtleties of the nomenclature is necessary to grasp how and why these technologies are changing surgical practice. In a general sense, VR is a completely immersive experience in which one is placed into a virtual environment using controllers and sensors to interact within it. AR is designed to aid what is seen in the real world; virtual images are superimposed and overlayed onto the real environment rather than being wholly virtual, as is the case in VR. Traditionally, the user interacts with the internet through a screen or an interface. The metaverse allows one to immerse themselves within the internet. Users create an avatar and are able to interact and collaborate in real-time with others from anywhere around the world. A metaverse may combine VR and AR or may only use one of these. The characteristic feature of the metaverse is the ability to interact with other users present in the metaverse at that moment. Normally, one would interact with the internet using an interface — a 2D screen and physical inputs. However, with the metaverse, you can effectively enter, and become, the internet.

Allowing surgeons, allied healthcare professionals and indeed patients to be immersed in a virtual world has the potential to enhance all aspects of surgery: surgical training, pre-operative planning, intra-operative decision making and patient experience. We will explore how the metaverse will affect each of these.

Surgical training

Surgical trainees are faced with several challenges. The European Working Time Directive 1998 has meant that trainees spend less time in theatre than times gone by; additionally, when in theatre, they are less likely to be the primary surgeon. The traditional ‘see one, do one’ approach has changed to ‘see many, do some of one, then do one’. When ‘seeing one’, however, the surgical trainee is often assisting from the other side of the operating table. The challenge for the trainee when performing a case for the first time is that they have never seen the first surgeon’s view until it is their opportunity to be one. This is akin to a sports player only watching his teammates play, then being thrown into a match where he/she is expected to perform. VR and the metaverse tackle these issues.

A group in South Korea developed a training course in the metaverse in which surgical residents were provided with head-mounted virtual reality devices (HMD) which gave 360º live video coverage of the course. Within this course, participants were given access to three-dimensional first surgeon views of live operating within Seol National University Bundang Hospital’s operating room (OR) thereby providing a unique training experience² (Fig. 1). An 8 K, 360º camera and three-dimensional (3D) XR immersive sound technology allowed over 200 participants from the United Kingdom, Singapore as well as South Korea to benefit from the comfort of their own countries.

Fig. 1.

Avatars in the metaverse watching the operating scene in Seoul National University Bundag Hospital, South Korea.²

Robotic surgery is now becoming commonplace. It has significant benefits over laparoscopic surgery, such as 3D visualisation, improved dexterity and the possibility of telesurgery.³ Novel technology will improve training for robotic surgical trainees. Telestration provides live annotations projected onto the screen that the robotic surgeon sees, enabling the surgeon to perform a task with direct live guidance.⁴ This software uses a 3D proctoring tool, which overlays annotations in the surgeon’s field of view, and has been studied on participants performing common surgical tasks, such as suturing, on a porcine model using the da Vinci Xi surgical system.⁵ The metaverse can enable remote proctoring, and even allow for remote case observation for educational purposes. One may be able to observe a live case being performed by a world leading subspecialist surgeon in another part of the world, from the comfort of one’s own home. In addition, VR headsets allow for the playback of 3D video. An often-understated aspect of robotic platforms are 3D views affording a greater detail of anatomy and surgical planes. Watching this video in 3D on a VR headset would give a view akin to the surgical console and aid in the transition to robotic surgery.

As the surgical metaverse expands, it becomes more resource intensive; there will be a critical point where there will not be sufficient trainers to train trainees. This is where artificial intelligence (AI) will play a significant role within the metaverse. With increasing data input from trainers and trainees, non-player characters (NPC) can be generated. NPC are avatars with whom users will be able to interact but are powered by AI rather than people. Once sufficiently trained, NPC could act as trainers within the metaverse.⁶ This will aid in reducing the burden for such manpower in order for robotic training in the metaverse to be effective.

Surgical trainees will not be the only beneficiaries of the metaverse. Universities have extended this experience onto their medical students in 2022 by providing lectures in the metaverse.⁷ Importantly, using this technology for medical students will help inspire the surgeons of tomorrow. Medical students frequently experience anxiety prior to attending the OR, and this has been proven to be a significant barrier in pursuing surgery.⁸ Providing 360º live video coverage of the OR may alleviate these fears and encourage the next generations of surgeons. The University College London offers a cutting-edge Master’s degree in minimally invasive surgery which almost exclusively utilises online learning. VR allows for distance learning and is the only institution to use telestration. This allows for students to interact with surgeons across the globe and observe live procedures. There are numerous benefits not only to learn the latest surgical developments, but also for overseas students. The natural development of this distance learning degree is the metaverse, where students will be able to interact with trainers and other students, during lectures or procedures, as avatars. The concept of meta-universities may be born.

Although not strictly using the metaverse, it is important to note the significant benefits that VR-assisted training has on the surgeon and the wider surgical team. A randomised controlled trial in 2018 showed that trainees who received a 12-week VR training course on executing a total hip arthroplasty performed better on cadaveric models than their colleagues who received conventional training in this procedure.⁹ VR-based training platforms can also be used to train the multidisciplinary team, including OR staff, on how to set up new equipment. This has significant time and financial implications for the healthcare sector.

Surgical training is likely to be the area that the metaverse will be most impactful. It overcomes common challenges that surgical trainees face, provides a safe environment for trainees to hone their skills, and improves access to training and mentorship owing to its inherent geographical-independent nature.

Pre-operative planning

Decision making prior to making the first incision can arguably be more important than the surgery itself. The decision to operate, the type of procedure and anticipation of technical challenges are necessary to be addressed in the pre-operative setting to ensure the best possible outcome for the patient.

For some time, cancer surgery and complex cases have required input from multiple specialties. Surgeons, radiologists, pathologists, specialist nurses, physiotherapists, together known as the multi-disciplinary team (MDT), must meet to review relevant clinical data and make the most appropriate management plan for the patient. These meetings are referred to as ‘Tumour Boards’ or ‘MDT Meetings’. It is for these meetings that entering the metaverse is likely to be of great benefit. Traditionally in these meetings, each facet of a patient’s clinical data is projected onto a screen one-by-one. In the metaverse, all clinical information including patient history, blood work, imaging and anatomy would be available in one’s vision and can be viewed by a user independently at their leisure, by simply turning one’s head from side to side. In addition, this would provide the opportunity for healthcare professionals from anywhere in the world to join the meeting if a tertiary or quaternary opinion is required.

All surgeons have a meticulous pre-operative plan. However, the subtle and sometimes not so subtle variability in human anatomy may only be encountered intra-operatively. This can pose a significant unexpected challenge. Virtual reality digital surgical planning (VRDSP) is a novel technique designed to produce patient-specific virtual anatomical models of the region of the body to be operated. Surgeons can wear an HMD and view the anatomy first-hand as was reported when planning jaw reconstruction.¹⁰ The next step is for this to be incorporated into the metaverse, whereby multiple surgeons can view the virtual model simultaneously and give real-time input.

With non-clinical work commitments occupying an increasing proportion of a surgeon’s time, the metaverse will improve efficiency in pre-operative planning. And not only efficiency, but the clinical decisions made will be more patient-specific and ultimately of better quality, owing to VR-related technology.

Intra-operative decision making

Cases can often be challenging and require a second opinion. The metaverse would allow a surgeon who may even be at home to simply wear their HMD, log in with their avatar and provide real-time assistance to the operating surgeon. This is possible for laparoscopic, robotic and even open surgery. In fact, this was first reported in the United Kingdom in 2017, with the use of Microsoft Hololens. During a colorectal case, local surgeons were able to interact with surgeons from India and the United States to aid intraoperative management (Fig. 2). The operating surgeon could see two avatars alongside him in the operating surgeon in this extended reality (XR) using Microsoft Hololens, with each having access to recent imaging and intraoperative views.¹¹

Fig. 2.

Illustration of intra-operative decision making with the aid of avatars from Mumbai during Professor Shafi Ahmed’s case. [Available from https://medium.com/@professorshafiahmed/the-surgical-metaverse-bd9d1b623158].

Remote proctoring platforms, such as Proximie, allow someone located outside of the hospital to dial in and communicate with the surgeon with the help of annotations to aid the primary surgeon and provide live expert opinion. This is limited, however, by having a 2D screen setup away from the operating field. The metaverse would allow for the use of an HMD with AR features to overlay instructional information directly over the surgeon’s gaze.

Patient experience

When meeting patients on the day of surgery, their anxiety is often palpable, especially if it is their first procedure. The ‘unknowns’ of the day impart a significant burden for patients in the days and weeks leading up to surgery. The metaverse can help allay fears patients may have by familiarising patients with the hospital or operating room lay-out. An RCT assessed whether the use of VR would help reduce patient anxiety on the day of surgery. They provided 84 patients with an Oculus headset which played a 3D pre-recorded walkthrough of the day of surgery.¹² Although the results did not demonstrate a reduction in patient anxiety, it is the first step of a new avenue that can be exploited in the metaverse. Providing patients with their own avatar that can interact with other healthcare professionals during this ‘walk-through’ to whom they can ask questions would provide a more immersive experience and is likely to provide some familiarity to their day of surgery.

It is easy to forget that the pre-operative work-up and post-operative follow-up are just as vital to the surgery itself. At times, however, attending outpatient appointments can pose a significant challenge for patients; arranging transport, finding sufficient time away from a busy lifestyle and of course the expense of travelling to a clinic are just some of the drawbacks of face-to-face appointments. Since COVID-19, teleclinics have become more commonplace. Although extremely useful for certain circumstances, the human factor is somewhat lacking over the phone. Meeting with a doctor in the metaverse will be the next step for ‘teleclinics’ with some doctors from the United States already providing this service. As well as providing the human touch, more clinical information can be provided to patients, such as scans and investigations, which is not otherwise possible via telephone.

Although there are companies aiming to develop completely virtual hospitals in the metaverse, we are unlikely to see these hospitals replacing surgical departments. Rather, we are more likely to see the metaverse augment our real-world surgical departments to improve our communication with patients. However, meta-hospitals may be able to offer patients a similar experience to walking around a shopping centre, with the ability to explore and visit different healthcare providers in the metaverse.

Conclusions

Although in its early stages, the metaverse will undoubtedly play a significant role in surgical training in the immediate future. It is also likely to aid peri-operative decision making to therefore improve patient outcomes and experience. The benefits are clear. There are, however, significant obstacles that may delay us seeing the metaverse fully incorporated into our surgical world. These include the issues of patient confidentiality, cyber security, and the financial burden of its implementation. The next two decades will prove to be an exciting period for the use of technology in surgery.

Funding

None.

Conflict of interest

None.