The oxford dictionary defines Augmented Reality (AR) as a “technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view.”
Augmented reality technology has become widely available in multiple domains as rapid advances in computational power have allowed for real-time three-dimensional (3D) rendering. Augmented reality is a natural progression of Virtual Reality (VR). Through advanced computational rendering, tracking, and the use of 3D, wearable, near-eye displays (i.e. Meta Quest, Samsung Gear, Apple Vision Pro), VR technology allows for the user to experience an immersive experience. While VR technology has been shown to be an effective training modality in orthopaedics in the setting of arthroscopy and arthroplasty simulators, it has limited use in the surgical setting as its functions remain isolated to the virtual world. AR differs from VR in that AR technology allows for the overlay of three-dimensional mapping in addition to the user’s direct visual perspective.
There are three general types of AR: projection-based, video-see-through (VST), and optical-see-through (OST). (ref 1) Projection-based AR involves an image projected onto the site of interest such as classroom whiteboard which allows student interaction. Video-see through involves a video screen or camera that allows live video feed of real-world with interaction such as many mobile apps and games with AR (i.e. Pokemon Go). Finally, OST requires an optical lens or glasses that then project the digital image while viewing the real world (Google Glass, Microsoft Hololense). This is the focus of many surgical applications and this article as it allows visualization of a pre-operative plan overlayed with the real-time intraoperative field. The wearable solution, in particular, allows for the AR based information to be visible to the surgeon without the need to remove their visual focus from the surgical field.
From a surgeon's perspective, AR technology allows for the identification of structures which are outside the line of sight. AR technology has the potential to facilitate the more precise placement of surgical implants and can facilitate directly visualized navigation in the surgical field. Thus, two primary uses of AR technology can be considered in the OR: anatomic identification of structures outside the direct field of view, and guidance of instrumentation in real time (navigation), based on previously acquired imaging. This requires two types of modelling: patient anatomy and surgeon instrumentation.
AR technology has been shown to be a useful adjunct in the surgical setting with its ability to overlay proxy anatomy onto the surgeons' field of view. High resolution MRIs and CT scans can be used to create three dimensional models of patient-specific anatomy (i.e. a large sarcoma, complicated joint deformity, etc.). These models can then be manipulated in real time and with multiple degrees of freedom to facilitate better understanding of the anatomy while in the operating room. This technology has been used in cases of complex limb reconstruction to identify and protect important neurovascular structures and could similarly be utilized in complex sports medicine cases.
Surgical navigation uses computational modeling to overlay the surgical instrumentation and implants with relevant anatomy in order to guide the placement of the instrumentation and implants. Navigation can include the linkage of patient anatomy to surgical instrumentation using artificial intelligence (AI). Linkage in real-time is accomplished through registration of patient anatomy. In most cases, a preoperative fiduciary or marker is obtained to create a 3-D model of anatomy (CT scan), and then with use of fixed optical tracker, an intraoperative registration is performed to allow the AR system to identify the anatomy in real-time and space. Due to the natural plasticity of soft-tissues, AR systems cannot currently offer surgical guidance for soft tissue procedures, but can be effective in bony anatomy because of its natural rigidity and ability to attach optical trackers. Surgical instruments can then be utilized with similar optical tracking devices which allows the AR system to link the relative positions of the patient anatomy to the instrumentation in real-time. AR systems have been utilized successfully to accurately place glenoid components in total shoulder arthroplasty (Stryker, Exactech, Medacta). (ref 2) It should be noted, however, the optical tracking systems and registration can be cumbersome to those unaccustomed to its use.
Augmented reality utilization for the purpose of improved understanding of non-directly visualized anatomy as well as for navigation has great promise to assist with surgical precision and management of complex anatomy.
References
- Matthews JH, Shields JS. The Clinical Application of Augmented Reality in Orthopaedics: Where Do We Stand? Curr Rev Musculoskelet Med. 2021 Oct;14(5):316-319. doi: 10.1007/s12178-021-09713-8. Epub 2021 Sep 28. PMID: 34581989; PMCID: PMC8497656.
- Rojas JT, Jost B, Zipeto C, Budassi P, Zumstein MA. Glenoid component placement in reverse shoulder arthroplasty assisted with augmented reality through a head-mounted display leads to low deviation between planned and postoperative parameters. J Shoulder Elbow Surg. 2023 Dec;32(12):e587-e596. doi: 10.1016/j.jse.2023.05.002. Epub 2023 Jun 3. PMID: 37276917.