New advances in video processing, 3-dimensional designs, and augmented/virtual reality are exciting and evolving fields. These new tools can facilitate the learning phase of basic or advanced endoscopic procedures. Herein, we explain our initial experience, creating an immersive virtual reality (IVR) by using 360-degree recording videos from an interventional endoscopy room.
Some common terms used around this technology, such as Augmented reality (AR), Virtual Reality (VR), Three hundred sixty videos, and Mixed Reality (MR), are discussed below.
Three examples of VR 360 endoscopic room videos are included in this article.
Los nuevos avances en el procesamiento de vídeos, diseños en 3D y realidad virtual y aumentada son áreas de mucho interés y en pleno auge. Estas nuevas tecnologías pueden facilitar la fase de aprendizaje en el campo de la endoscopia, tanto en procedimientos básicos como en avanzados. En este artículo, detallamos nuestra experiencia inicial en la creación de situaciones inmersivas en realidad virtual (IVR) utilizando grabaciones en 360° de una sala de endoscopia.
Asimismo, abordamos algunos términos de uso frecuente en relación con esta tecnología, como son la realidad aumentada (RA), realidad virtual (RV), vídeos en 360° y la realidad mixta (RM).
Se incluyen en este artículo 3 ejemplos de vídeos de RV en 360° en sala de endoscopia.
Historically, teaching gastrointestinal endoscopy's skills has been based on the master-apprentice model. Although this model provides certain advantages, like direct supervision and immediate real-time evaluation, teaching fellows endoscopy could increase procedure time, cost, and patient discomfort.1,2
Trying to come up with solutions, trainers are incorporating ex vivo and in vivo models, mannequin-based or virtual reality (VR) simulators. A recent Cochrane review and meta-analysis,3 found that VR simulation training is advantageous over no training and can supplement conventional endoscopy training, but the cost of VR simulators is high, and not all endoscopy centers could afford it.
Video processing is an exciting and evolving field.
360-degree videos, also known as immersive or spherical videos, are video recordings where a view in every direction is recorded at the same time by an omnidirectional specialized camera or a collection of regular cameras. The footage is merged into one spherical video by the camera itself or using specialized software.
Immersive virtual reality (IVR) using 360° videos could enhance the experience of Live courses and allow endoscopists all around the world to observe advanced procedures in detail without traveling, saving time and costs.
Herein, we share our initial experience applying these recently developed technologies in different endoscopic scenarios. The close-to-real-life perspective that this new technology brings could potentially facilitate and speed-up gastrointestinal endoscopy training. It could help trainees and endoscopists who want to learn new procedures that are not being performed in their units.
To a better understanding, we will explain the following terminologies2–4:
Augmented reality (AR): It is when the real world is augmented by the addition of computer-generated 3D objects that overlay in the physical world.
Virtual Reality (VR): It is a simulated experience that can be similar to or completely different from the real world. The user is teleported into an artificial world by wearing a VR headset. The 3D world perspectives can be created by a computer or by 360-degree video recording (Fig. 1).
Mixed reality (MR): includes a mix of virtual and real components. It differs from AR in that the superimposed objects are solid and can be touched and manipulated.
Previous AR/VR 360 experiences in a surgical environmentAR and VR technology have been used by surgical specialties to provide simulation at a low cost. Some initial studies have shown that trainees performed better if they were taught by using a 3D/360° video instead of a traditional 2D video. A survey of Yoganathan and colleagues indicates that 360° videos were useful as an independent teaching aid and even when it was used as an adjunct to traditional face to face teaching.5
There are some studies showing how superimposing virtual 3D models onto the surgeon's field of view could help in live surgery, by way of AR-based navigation.6
In a recent study using VR glasses for neurosurgery training, Choque-Velasquez and colleagues showed the utility VR/AR to perform complex microsurgical exercises, improving eye-hand coordination and dexterity in microsurgery.7 Another neurosurgical group used VR and haptic force feedback as a successful way to train cerebral aneurysm clipping.8
Although these are promising surgical experiences, there are a few or none experience in GI endoscopy training using 360 IVR videos.1
How to film a 360-Degree VideoInitial experience inside an endoscopic room setting:
We performed a series of pilot videos using a 360° camera (Insta360, Tustin, CA.) that can be watched using the following links on YouTube (YouTube app if a smartphone is used, cardboard mode with VR glasses are recommended). Patients gave informed consent before all procedures.
As everything in the line of sight of the camera is recorded, we have taken special care not to record sensitive material inadvertently (patient identifiable data).
We recorded an EUS-FNA, an ERCP, and a POEM procedure to show the advantages and limitations of this technology:
360 videosCase 1: Pancreatic cyst fine-needle aspiration guided by endoscopic ultrasound. (see QR code 1)
https://www.youtube.com/watch?v=b-VB_kGAJEA
Case 2: Choledocholithiasis extracted by ERCP. (see QR code 2)
https://www.youtube.com/watch?v=m0PY-HjKUXA&t=187s
Case 3: POEM with short myotomy for sigmoid esophagus with type 1 achalasia. (see QR code 3)
https://www.youtube.com/watch?v=pYCCqep_OYA&t=317s
For our first experiences, we used a small 360° consumer level camera that has two spherical lenses. Its small size allows it to be placed almost anywhere in the interventional room. Different settings were tested to find the optimal way to film the experience. This ensured that the scene felt real and also enabled the user to have critical views of each procedure. For ERCP and EUS procedures, the 360 camera was located close to the patient's head, allowing a pivotal perspective view of fluoroscopy and endoscopy screens. For POEM procedure, the best angle was achieved by locating the camera in the middle of the patients’ abdominal surface.
Stitching the videoThe Insta360-one camera has an auto stitch function enabling the export of data (merged video) directly from the camera to the editing software.
How to watch an IVR videoThese videos can be watched in 3 different ways: wearing VR goggles for an immersive experience (Fig. 2), onto a computer moving the perspective with the mouse, or employing a mobile phone or tablet and moving it around the whole area of scanning (Fig. 3).
VR/AR show many advantages for endoscopic training (Table 1), and there are many ways to use them in the near future:
- •
Real-time 360° IVR conferences: Immersive virtual reality using 360° videos could enhance the experience of Live courses without traveling
- •
Augmented reality for telementoring: Complex endoscopic procedures could be facilitated by telecommunication technology and augmented reality, in which an expert provides guidance to a less experienced endoscopist from a remote location. There are some pioneer experiences in Spain using mobile networking (Rosón et al.), via high-speed connections (5G).9
- •
Gamification: Close cooperation between software engineers and experienced physicians would be necessary to incorporate software improvements, leading to realistic virtual simulations suitable for endoscopy.10
- •
Evaluation of technical skills of trainees or endoscopists before starting a new procedure in clinical practice.3,4
- •
IVR plus AR experience: we are developing ways to superimpose objects (hologram-like) to IVR videos, which could allow us to have, for example, previous reports, endoscopy pictures or 3D-CT reconstruction data in real-time by using AR goggles.
Advantages of VR and AR.a
| Hardware |
| Portable |
| VR headset |
| AR glasses |
| Handset |
| Software |
| Flexibility |
| Customizable simulations (polypectomy, bleed, and so forth) |
| No hardware changes required |
| Procedural competency |
| Complex procedures |
| Content can be adapted or changed |
| Potentially greater degree of realism |
| Customizable scenarios for greater breadth and depth of clinical content |
| Greater anatomic knowledge: AR overlay of internal anatomy can help with spatial orientation |
| Broader content capability: 3D reconstruction of pathology |
| Intraprocedural recall of relevant clinical information |
| User feedback |
| Technical AND cognitive skills feedback |
| Intraprocedural guidance from expert mentors |
New technologies in video processing and virtual reality could improve the way we learn and train in endoscopy. This report is a proof-of-concept for the technical feasibility of custom immersive VR setup in our midst. Immersive virtual reality obtained by 360 video recordings is cheap and easy to apply, being a promising training tool.
Conflict of interestThe authors declare that they have no conflict of interest.
