News

Featured in Research Roundup | September 8, 2023

Extended reality has reached a critical point in biomedicine. The technology is accelerating and excitement about potential health care benefits is mounting but bridging the gap between envisioning what could be and actually making it happen is still a work in progress. Part of the challenge is coordinating all the major players to ease the development pipeline so virtual, augmented, and mixed reality can reach their potential in everything from diagnostics and surgeries to clinician training and telemedicine.

A new university, industry, and U.S. government collaboration is now in place to address that challenge. Called the Center for Medical Innovations in Extended Reality (MIXR), it will bring together the researchers pursuing virtual, augmented, and mixed reality technologies that clinicians need and want; the companies seeking to develop and make products that have both strong market potential and can earn the necessary approvals from the U.S. Food and Drug Administration (FDA); and FDA officials who are working through the agency’s regulatory responsibilities and procedures as medical extended-reality projects surge.

Funded with a $5 million grant from the National Science Foundation (NSF) , , , MIXR is a joint center between the University of Maryland, College Park; the University of Maryland, Baltimore; and the University of Michigan. The FDA and several technology companies, including industry giants Sony and Microsoft, are also partners.

Its time has come

The timing is right for MIXR, said MIXR lead-site principal investigator (PI) Amitabh Varshney, Ph.D., professor of computer science at the University of Maryland and IEEE Fellow (Figure 1). “Just over the last few years, virtual and augmented reality devices have become commoditized so you can go to electronics stores and buy very high-powered headsets, which was not possible before. There have been amazing advances in artificial intelligence and machine learning, and we now have 5G and soon 6G wireless capabilities,” he said. “This confluence makes this a perfect time to invest the effort and resources to advance the field in a direction that will benefit society at large.”

The tipping point between dream and reality is here, agreed Rishi Reddy, M.D., M.B.A., a MIXR partner-site PI and the director of the Center for Surgical Innovation at the University of Michigan (Figure 2). “I don’t think we would have been able to make much headway just five years ago, but with 5G, we have arrived at an opportune time to really plan out thoughtfully what we can accomplish with extended reality in health care,” he said.

Extended reality cannot come soon enough, remarked partner-site PI Sarah Murthi, M.D., of the University of Maryland School of Medicine (Figure 3). While surgeons are already able to access real-time computed tomography (CT), ultrasound, and other images during procedures, the images are 2D and appear on separate computer screens. That means surgeons must continually look back and forth from patient to various screens as they carry out often-complex operations, she said. “You can’t see the images without turning away, and you can’t blend one image with another, so it’s still much like it was in the 1950s. Extended reality, on the other hand, can disrupt how data is displayed, and allow providers to see different data streams, fuse data, and overlay real-time images right on the patient” (Figure 4).

Reddy views student training as having great potential, too, by allowing them to see what the surgeon is seeing, including the 3D overlays on the patient, so they can ask for clarifications as the real-time operation progresses. He believes extended reality will also be very helpful for preoperative patient counseling, and to allow patients to connect more meaningfully with physicians during a telehealth visit.

“Overall, these technologies provide a lot of powerful opportunities to positively impact health care and the way that we currently function,” Reddy said. “Our goal with MIXR is to do that in a consistent way, which includes validating that these virtual, augmented, and mixed reality systems do make surgeries safer, improve the patient experience, and improve training and education.”

Smoothing the way

MIXR will evaluate the entire extended-reality pipeline (Figure 5). That evaluation begins with acquisition of the virtual scene, Varshney said. In training medical students on a surgical procedure, that would mean acquiring the entire technique so the students can observe in close range every step the surgeon takes. “To make it truly scalable, you would need an array of cameras to capture the procedure, so the students can place themselves in the shoes of the surgeon and see it as it is being performed.”

Displays in extended-reality headsets will require evaluation to ensure they perform well. This includes such technical aspects as resolution and field of view, but also issues of user fatigue, Varshney said. “Right now, people can only wear headsets for a short period before they get headaches or nausea, so if a surgeon is starting a 6-hour operation but can only wear the headset for 20 minutes, that is not good. We are looking at what the industry and university roadmap should be so that we design and evolve headsets that are truly effective, ergonomic, and comfortable for people to wear for hours on end so they can be productive at their task.”

Other focus areas for MIXR include best practices for using artificial intelligence and machine learning in the context of visualization and visual immersion for medical diagnosis and medical procedures, and an examination of how next-generation wireless technologies may impact extended reality. As an example of the latter, Varshney proposed that 6G might support a technology to allow a doctor to appear via hologram in a rural or remote location, such as a battlefield, and provide medical direction to on-site personnel who can provide immediate care for a patient.

MIXR will look at the full range of health-related applications. One area that Reddy sees the technology playing an important role is in helping clinicians perform at a high level. “This could be a great boon to procedural training, because there are certain procedures that we surgeons only do once or twice a year, so augmented reality or virtual simulation is a way to practice and maintain skill sets for rare procedures,” he remarked.

Creating a new path

One more critical MIXR emphasis lies in facilitating regulatory approval for extended reality, because it falls outside the normal realm of the FDA, noted Murthi. “If it is a drug, or if it is something that is completely automated, such as a robot doing a tumor resection, that is clearly something for the FDA. A physician decision, such as my decision to choose one antibiotic over another or whether to read certain procedural information before surgery, is not something the FDA oversees,” she said. “Here is the question: What happens with something like augmented reality, where a physician is making a decision but that physician is heavily influenced by this technology? That falls somewhere in between and makes things much murkier for the FDA.”

Within these muddy waters, the FDA hopes to restructure the approval process for extended-reality devices in surgery and medicine, according to Varshney. Rather than going through a lengthy process to approve one-by-one requests to use a certain company’s headset for a specific procedure, the FDA is proposing that universities and companies create teams to define more inclusive guidelines. For example, a guideline might state that any extended-reality headset meeting a certain set of specifications for brightness, resolution, field of view, and weight is appropriate for use with X, Y, and Z surgical procedures, he said. “This way, the whole approval process would be much more streamlined.”

This type of streamlining would be a game-changer both in the United States and abroad, commented Reddy. “Similar groups around the world are working on accelerating the extended-reality field, but MIXR is in the unique position of having the FDA as a partner. Because the U.S. is such a big market for all these extended-reality use cases, if the FDA approval process becomes more efficient, that really opens up a lot of opportunities for companies wherever they’re based.”

MIXR will also have an impact on the research end, Varshney said. “We are already beginning to fund specific research projects that have been collaboratively designed and defined by the FDA, industry, and MIXR,” Varshney said. Although the center is only funding research projects at the three partner universities, he added, “the outcomes of these research projects will help accelerate the evolution of the virtual/augmented reality field for medicine.”

The NSF-funded MIXR to jumpstart the entire extended-reality field, not to push through specific research or products, Murthi noted. “This is specifically pre-commercial, so the point is not for the University of Maryland or the University of Michigan to develop a company to turn a profit, or for a certain tech company to make and sell a product. The idea is to address fundamental questions and issues that are inhibiting or could accelerate the whole field.”

Put another way, the center will advance regulatory standards and lead to foundational insights in a precommercial way, such that all companies in the extended-reality space will benefit, Varshney said. He added. “It’s a win-win-win across university, government, and industry, so it is really an amazing center in that regard.”