How Augmented Reality Will Make Surgery Safer
Published 3/20/18 in the Harvard Business Review
By Sarah Murthi, MD, Amitabh Varshney
Some of the biggest medical advances of the last few decades have been in diagnostic imaging—ultrasonogaphy, mammography, computerized tomography (CT), magnetic resonance imaging (MRI) and so on. The same forces that have propelled technology developments elsewhere—tiny cameras, smaller and faster processors, and real-time data streaming—have revolutionized how doctors use imaging in performing procedures. Almost every surgery involves some sort of a scan prior to incision. Even in emergencies, surgeons have ultrasound or CT to help guide the procedure. Imaging can now be performed in real time at the point-of-care during procedures, both big and small.
Yet, while imaging has radically evolved, how images are displayed is basically the same as it was in 1950. Visual data are always shown on a 2D flat screen, on displays that force health care providers to look away from the patient, and even away from their own hands while operating. Further, the images are not displayed from the perspective of the viewer, but rather from that of the imaging device: doctors have to use skill and imagination to understand and mentally project the images into the patient while they are doing procedures. Finally, different types of visual data are displayed separately, so doctors have to direct additional attention to mentally fusing multiple image types, such as angiography and CT, into a coherent representation of the patient. Acquiring this skill takes years of training.
Augmented reality (AR), a set of technologies that superimpose digital information on the physical world, has the potential to change all of this. In our research at the Maryland Blended Reality Center’s “Augmentarium,” we are prototyping AR applications in medicine, as are teams at Stanford, Duke and Johns Hopkins. In envisioned application, a surgeon using an AR headset such as Microsoft’s HoloLens would be able to see digital images and other data directly overlaid on her field of view. In such a scenario, the headset might display a hovering echocardiogram with vital signs and data on the characteristics of the patient’s aneurysm directly above the surgical field. The surgeon needn’t look away from the patient to multiple different displays to gather and interpret this information.
AR’s potential ability to concurrently display imaging data and other patient information could save lives and decrease medical errors. This is especially true for procedures done outside an operating room. The OR may be the safest place in the hospital, where one patient has an entire team of 4 to 8 dedicated doctors and nurses. Because everyone has pre-operative imaging, the procedures are generally well-planned. Anesthesiologists monitor the patient’s physiology and administer pain-controlling and life-saving medications. Surgical nurses make sure all of the necessary equipment is immediately available. Surgeons can be completely immersed in the operative task. But time in the room is extremely costly, and ORs are solidly booked with elective cases. Elective operations are an essential source of revenue for all hospitals, so there is incredible pressure to keep ORs full and flowing. Small, emergent procedures do not easily fit into this reality. As a result, many of these procedures are done outside the OR in intensive care units and emergency departments. It’s during these “bedside procedures” that patients may be most at risk and where AR could provide some of the greatest benefit.
Unlike operations in the OR, bedside procedures have minimal support. There is usually one doctor and one nurse, both of whom have other responsibilities. Often the patients are physiologically unstable. If it is late at night, the physician performing the operation may be a junior trainee, who has both the anesthesiologist’s and surgeon’s roles. The rooms are not specifically designed for procedures, especially those that involve cumbersome carts for imaging. The situation is made worse by a tangle of patient monitors. In tracking multiple image and data displays, it is easy to miss vital cues regarding the patient’s status. A single AR display that integrates all imaging and patient data and allows doctors to keep their eyes on the patient has the potential to improve quality, safety and reduce cost by decreasing procedure-related complications.
In addition to cost reduction from safer procedures, there is also the potential to reduce costs by eliminating the need for redundant screens. Currently ultrasound, endoscopy, and bronchoscopy all require hospitals to buy entire systems, each with its own display. The systems can cost upwards of hundreds of thousands of dollars each. It is not that different modes of imaging require special displays, but that the current economic model is to sell whole systems with incompatible imaging devices. The electronic medical record has its own display. Hemodynamic monitors and ventilators each have their own separate screens, and that data is not merged. AR can provide a shared display, eventually reducing the need for a dedicated monitor for each aspect of a patient’s data, while providing a place where physiological data from multiple sources can merge in real time.
AR technology still needs to evolve for this all to be realized, and doctors need to buy in to the concept. Hardware needs to fit comfortably and securely on the practitioner’s head. For some applications the images will need to be as opaque as possible, while for others they will need to be more translucent. If the projected images are being used for operative guidance, they need to be positioned with extreme accuracy. There are a variety of technical challenges, but none of them are insurmountable.
At our Maryland Blended Reality Center, we are committed to investing in the future of AR as a primary tool to help health care providers save lives while also reducing medical costs. Teams at Stanford, Duke and Johns Hopkins are also working to merge and project visual data, while simultaneously creating AR displays ideal for patient care.
It will take gifted computer scientists and visionary physicians to make augmented reality an actual reality in medicine. But we are excited for the future, where the use of AR in health care will be just as commonplace as use of a stethoscope.
Sarah Murthi, MD, is an Associate Professor of Surgery at the University of Maryland School of Medicine, R Adams Cowley Shock Trauma Center.
Amitabh Varshney is Dean of the College of Computer, Mathematical, and Natural Sciences at the University of Maryland, College Park.
