Doctors Train in Virtual Reality

Healthcare is just one of the many fields finding new uses for extended reality (XR), comprising both augmented and virtual reality.

Virtual reality (VR) in particular is being used for everything from hosting remote support groups for teen cancer patients to giving medical students a taste of the grisly reality of trauma medicine.

Those projects typically are one-off purpose-built implementations of VR, though. The technology has continued to advance and its availability to increase, enabling the creation of more general-purpose applications.

The potential is not lost on VR developers and investors. According to the June 2022 Global AR-VR in Healthcare Market report from Research and Markets, the sector was valued at $2,748 million in 2021, and is expected to reach $9,796 million by 2027.

Two applications identified by Research and Markets as drivers of that growth are medical training and surgery planning.

The U.K.’s Royal College of Surgeons has set up an commission to examine where surgery is headed, and one of its commissioners, plastic surgeon and medical entrepreneur Dr. Nadine Hachach-Haram, told MobiHealthNews, “We can see the effective use of virtual reality in simulation, in terms of preparation and rehearsal. The challenge here is the cost of scaling, as there are tens of thousands of different procedures with nuances in each of them.”

Several companies are tackling the scaling problem.

How surgeons now train

A main value of VR simulation, as exemplified by its use to expose medical students to trauma situations, is to give surgeons an idea of what they can expect to encounter in actual surgery.

Surgery is traditionally taught as a mentorship program, explains Dr. Michael Ast, an orthopedic surgeon and Chief Medical Innovation Officer at New York’s Hospital for Special Surgery. “We do didactic teaching in a modified classroom setting, and then we guide our trainees through the operations. Eventually, they start to participate in the procedure.”

The problem with that approach is the limited number of those “tens of thousands of different procedures with nuances in each of them” in which a trainee might actually get to participate. Also, those ‘nuances’ are not about just the mechanics of the surgery, says Dr. Ast; “The farther along into training, the more our trainees come to realize that our job is not just to fix the problem, but also to preserve the rest of the anatomy. The questions tend to change from ‘how does this drill work’ to ‘how do I make sure that when I’m using this drill, I’m not causing damage to other things?'”

Training on generic patient models

Dr. Ast is on the advisory board of the surgical training and assessment platform Osso VR, which makes simulations that provide trainees the chance to practice specific surgical techniques before using them in the Operating Room. Osso’s simulations are individual applications for Oculus Quest headsets. Osso vice president of product Anu Kirk explained,
 “You select the procedure you want to do, and you end up in a simulation that looks like a real operating room with a real patient and real equipment.”

The simulations are developed using the Unity game engine, and they rely on the “technique guides” that manufacturers provide along with their surgical devices. “We read the technique guide and write a script,” sahs Kirk “and then a production team takes the script and says, ‘We’re gonna need to make models for these devices, and we’re gonna need this kind of anatomy, and we’re gonna need to build these types of interactions.'” It takes months to prepare the simulation to be “as realistic as we can make it,” Kirk continues. “YouTube has sometimes banned our videos for being too gory.”

In addition to the relevant tools and a patient, the simulations provide highlights and tool tips or floating diagrams to guide the trainee through performing the surgery. (The platform can also be used in test mode, with the aids turned off.) Once the procedure is finished, the system can provide feedback. “If there’s something like where you needed to drill a screw in, we can measure the accuracy of that screw placement,” Kirk says.

“We also do team training, where a surgeon sitting in New York can be in a virtual environment with a surgeon in California, and they can go through a surgical technique and simulate the case before the live procedure,” says Dr. Ast. “I think that’s one of the most valuable parts of virtual reality surgical training for in-practice surgeons adopting new techniques.”

Training on patient models

Osso has considered enabling its platform to import a patient’s data so the simulation would reflect their specific anatomy, but right now, says Kirk, it is not something that can be done at scale. Other companies, on the other hand, are committing to that approach. Surgical Theater and 8chill, for example, each are focusing on turning individual patients’ imaging into VR representations of their anatomies.

It is important for surgeons to understand why a surgery is being performed, said 8chill CEO Aravind Upadhyaya. “They need to look at the diagnostic medical images, and they’re better understood in 3D (three dimensions). CT (computed tomography) scans and MRIs (Magnetic Resonance Imaging scans) are 3D images, so we can reconstruct the anatomy.”

To create such a reconstruction, a patient’s medical images and scans are uploaded to the 8chill platform, which ingests and converts them into a model suiting the surgeon’s needs. “Facial and orthopedic surgeons are more interested in the bone than the tissue; they want to see an SDL or an OBJ file of the 3D model,” says Upadhyaya. “For a vascular surgeon or a neurosurgeon, you’re really interested in the soft tissue. For that 3D reconstruction, we use industry-standard ray marching cubes.”

The resulting model is hosted in the cloud and delivered to the surgeon’s laptop, where he or she can view it with a VR headset. The platform is agnostic when it comes to which headset to use, says Upadhyaya, “but we prefer the Oculus (now Meta) Quest, because they’ve made it very easy to push into their app store and for us to manage the devices.”

In addition to models built from uploaded images, 8chill’s platform also let surgeons generate their own models. “We’ve created a rig with a camera sitting on the surgeon’s head. We’re capturing in VR what a surgeon is going to see.” Another surgeon can then watch the video in a headset and imitate the motions of the surgeon who captured that video.

A further advantage to modeling an actual patient’s anatomy, according to Surgical Theater and 8chill, is that patients who care to could also may don a headset to better understand their conditions and get a preview of what the surgery will be.

Future directions

8chill is using its platform to develop a library of surgical content. “That’s where we are going today,” says Upadhyaya. “We have a library of dental procedures, and now we are building some nursing procedures. There’s a lot of areas to cover; I think it’ll take a couple of years before we really have a significant amount of content that anyone could purchase on something like Udemy.”

 “I think long term, we can not only use VR as a training tool, but also as a assessment tool,” says Dr. Ast. “Right now, we have no way to assess surgical education over time. This technology might eventually let us objectively measure a surgeon’s preparedness for a particular procedure and say, ‘yes, you’re ready to do this on your own,’ or ‘no, let’s have you practice a little bit more.'”

Dr. Ast also sees a role for VR in the live surgical theater. “I think what you’ll see is a surgeon saying during an operation, ‘Okay, I’ve got these next three steps plus a predictive algorithm of the 10 things that could happen. Let’s switch my glasses to viewing virtual reality and go through them quickly to see what could happen.”

 

Jake Widman is a San Francisco, CA-based freelance writer focusing on connected devices, Smart Homes and Cities, Extended Reality, and other emerging technologies.