Augmented Reality Based Remote Surgical Education

Augmented reality (AR), as defined by the Oxford dictionary, “is a technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view”.

There is growing literature on multiple applications for AR-based education in both surgical and non-surgical medical specialities.^1-5 In laparoscopic surgery, remote guidance and tele-mentoring has been applied to a variety of abdominal surgical procedures.^6-7 Remote guidance has also been used in robotic surgery education.⁸ The role of head mounted device-based AR teaching has been evaluated in Urology and Plastic Surgery.^9-10 Accelerated by personal contact concerns during the COVID pandemic, the use of AR expanded for educating trainees unable to participate in traditional clinical activities, and remote clinical training seems to be a viable tool for medical education moving forward.^11-12

AR has already been used in orthopedic education in several ways. There is a growing body of literature evaluating the use of AR-based remote education during hip and knee arthroplasty.^13-15 In one of the earliest papers using remote AR assistance during surgery, Ponce et al. demonstrated usefulness of this technology during shoulder arthroplasty with a remote surgeon guiding the operating surgeon from a separate institution.¹⁶ Our spine colleagues have also demonstrated the utility of remote AR based assistance during surgery.^17-18 Finally in arthroscopic surgery, trans-Atlantic telementoring has been demonstrated by Stetson et al.¹⁹

AR in surgical education allows remote interaction between two or more people in a more sophisticated way than simple videoconferencing. The remote participant can engage through both video and audio with the local participant, including engagement with the local environment. In terms of surgery, picture a remote provider with the ability to place their hand or instruments in a video of the surgical field, such that the local person can view this interaction. The view that the remote person engages with can be the first-person point of view: i.e. that of the operating surgeon or proxy. This is accomplished typically with head mounted AR devices such as the Microsoft HoloLens, Vuzix smart glasses, Google Glass and others, where the remote participant can view and interact with the created video and audio feeds. The remote person can also engage with other views captured by any video or even static device. For example, multiple video feeds can be achieved through an array of intraoperative sources such as the arthroscope, in-light cameras, floating cameras, ultrasound machines, etc., or static imaging such as fluoroscopy or O-arm.^TM The local provider has access to at least one monitor on the AR generating platform. The two participants are then able to engage in real time in the combined real and augmented space.

There are innumerable ways that this technology can be useful in orthopedic surgical education. AR allows trainees to engage in the real-time clinical care setting even when physical presence is limited because of illness, distance, quarantine, etc. (something very relevant after we have all experienced emergency pandemic conditions). In an in-person setting, the teaching surgeon can use AR tools to allow more hands-on autonomy and help maximize repetitions while still giving thorough instruction and maintaining safety and quality for the patient.

Such learning can be expanded beyond surgical trainees to other learners involved in orthopedic surgical care, such as medical students, physician assistants, nurses and technicians. AR platforms can bring learners into the orthopedic surgical or clinical care environment earlier in their education as clinical sessions can be broadcast into the classroom setting. Since cases completed on these platforms can be saved and viewed later, the technology can be used for pre-operative learning to let learners preview exactly how a specific case is conducted, in order to amplify learning opportunities and efficiency during hands-on surgery.

Beyond the needs of medical learners, AR technology has obvious applications to enhance peer-to-peer interaction and education. One exciting possibility is the use of AR on an international scale. Operative and non-operative training for a surgeon requires complex skill acquisition. For our colleagues worldwide, especially in resource poor environments, the ability to acquire further complex skills is impacted by time, distance, and cost, among others.

The application of augmented reality technology can change the paradigm for the transfer of skills and knowledge regardless of background, training level or geographic location. Creating access to real-time clinical care across the globe will broaden access to surgical education for learners, including those in resource-poor areas who may not have the same access to patients or surgical mentors. With today’s technology, education can move beyond the physical walls of a specific institution. Applying AR can democratize and improve access to surgical education.

Resources

1. Use of Extended Reality in Medical Education: An Integrative Review. Curran VR, Xu X, Aydin MY, Meruvia-Pastor O. Med Sci Educ. 2022 Dec 19;33(1):275-286.
2. How Augmenting Reality Changes the Reality of Simulation: Ethnographic Analysis.
   Loeb D, Shoemaker J, Parsons A, Schumacher D, Zackoff M. JMIR Med Educ. 2023 Jun 30;9:e45538.
3. A Systematic Review of the Use of Google Glass in Graduate Medical Education. Carrera JF, Wang CC, Clark W, Southerland AM. J Grad Med Educ. 2019 Dec;11(6):637-648.
4. Extended Reality Use in Paediatric Intensive Care: A Scoping Review. Goldsworthy A, Chawla J, Baumann O, Birt J, Gough S. J Intensive Care Med. 2023 Jul 12:8850666231185721.
5. Using Google Glass in Nonsurgical Medical Settings: Systematic Review. Dougherty B, Badawy SM. JMIR Mhealth Uhealth. 2017 Oct 19;5(10):e159.
6. Remote mentoring in laparotomic and laparoscopic cancer surgery during Covid-19 pandemic: an experimental setup based on mixed reality. Simone M, Galati R, Barile G, Grasso E, De Luca R, Cartanese C, Lomonaco R, Ruggieri E, Albano A, Rucci A, Grassi G. Med Educ Online. 2021 Dec;26(1):1996923.
7. A comprehensive review of telementoring applications in laparoscopic general surgery. Antoniou SA, Antoniou GA, Franzen J, Bollmann S, Koch OO, Pointner R, Granderath FA. Surg Endosc. 2012 Aug;26(8):2111-6.
8. Evaluating the ability of students to learn and utilize a novel telepresence platform, Proximie. Patel E, Mascarenhas A, Ahmed S, Stirt D, Brady I, Perera R, Noël J. J Robot Surg. 2022 Aug;16(4):973-979.
9. Looking at plastic surgery through Google Glass: part 1. Systematic review of Google Glass evidence and the first plastic surgical procedures. Davis CR, Rosenfield LK. Plast Reconstr Surg. 2015 Mar;135(3):918-928.
10. Is the use of augmented reality-assisted surgery beneficial in urological education? A systematic review. Alrishan Alzouebi I, Saad S, Farmer T, Green S. Curr Urol. 2021 Sep;15(3):148-152.
11. Smart glasses and video conferencing provide valuable medical student clinical exposure during COVID-19. Baker J, Schultz M, Huecker M, Shreffler J, Mallory MN. AEM Educ Train. 2021 Feb 19;5(3):e10571.
12. A remote access mixed reality teaching ward round. Bala L, Kinross J, Martin G, Koizia LJ, Kooner AS, Shimshon GJ, Hurkxkens TJ, Pratt PJ, Sam AH. Clin Teach. 2021 Aug;18(4):386-390.
13. Exposure to Extended Reality and Artificial Intelligence-Based Manifestations: A Primer on the Future of Hip and Knee Arthroplasty. Shaikh HJF, Hasan SS, Woo JJ, Lavoie-Gagne O, Long WJ, Ramkumar PN. J Arthroplasty. 2023 May 15:S0883-5403(23)00481-3.
14. Surgery Training and Simulation Using Virtual and Augmented Reality for Knee Arthroplasty. Mandal P, Ambade R. Cureus. 2022 Sep 6;14(9):e28823.
15. The effectiveness of virtual reality, augmented reality, and mixed reality training in total hip arthroplasty: a systematic review and meta-analysis. Su S, Wang R, Zhou R, Chen Z, Zhou F. J Orthop Surg Res. 2023 Feb 19;18(1):121.
16. Emerging technology in surgical education: combining real-time augmented reality and wearable computing devices. Ponce BA, Menendez ME, Oladeji LO, Fryberger CT, Dantuluri PK. Orthopedics. 2014 Nov;37(11):751-7.
17. Augmented Reality in Medical Practice: From Spine Surgery to Remote Assistance. Cofano F, Di Perna G, Bozzaro M, Longo A, Marengo N, Zenga F, Zullo N, Cavalieri M, Damiani L, Boges DJ, Agus M, Garbossa D, Calì C. Front Surg. 2021 Mar 30;8:657901.
18. Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation. Stefan P, Pfandler M, Kullmann A, Eck U, Koch A, Mehren C, von der Heide A, Weidert S, Fürmetz J, Euler E, Lazarovici M, Navab N, Weigl M. BMJ Surg Interv Health Technol. 2023 Jan 19;5(1):e000135.
19. The Use of Telesurgery Mentoring and Augmented Reality to Teach Arthroscopy. Stetson WB, Polinsky S, Dilbeck S, Chung BC. Arthrosc Tech. 2022 Jan 20;11(2):e203-e207.