When companies release new augmented reality glasses, the key details most people focus on are the size and resolution of the AR displays within users’ field of view. The larger and more detailed a headset’s augmented “window,” the better. But Swiss medical researchers are now spotlighting 3D tracking performance as an underappreciated specification when enabling AR use in critical procedures, such as surgery.
According to a group of researchers from the University of Zurich, ETH Zurich, and Balgrist University Hospital, AR headsets such as Microsoft’s HoloLens could be used to improve spinal fusion surgery, allowing surgeons to place digitally generated 3D anatomical maps on a patient’s surgical site rather than using invasive radiation to identify a location. The key problems, however, are that the first HoloLens wasn’t built for high-fidelity 3D tracking, and it’s subject to hologram drifting and application instability even if the wearer attempts to minimize head movement.
Seamlessly combining the real and digital worlds has been the end goal of augmented reality hardware developers for years, but actually achieving the feat has proved tricky, and most systems merely overlay digital objects on top of real ones, rather than “occluding” digital objects with real items in the foreground. As demanding as this might be when adding a virtual pillow to a real sofa, getting the millimeter-accurate positioning right so a doctor can insert a spinal fusion screw is another level of challenge.
The Swiss researchers tested their system by collecting 3D mapping point clouds of bone surface regions from realistic but “phantom” mockups of spines, rather than using human subjects. Notably, the phantoms were rigidly attached to a table to keep them from moving during the tests.
To use HoloLens for the simulated procedure, the researchers needed to activate Research Mode to access the headset’s two front environmental tracking cameras, which operate at only 640 x 480 resolution, and then use custom pointing devices to mark and digitally “register” surgical locations on vertebrae. Surgeons then used the marked locations and virtual imagery to properly bend and insert rods for spinal fusion. In trials with the phantom spines, the researchers found both timing and accuracy of the AR system to be comparable to state-of-the-art non-AR alternatives.
That said, the researchers are hoping for improved software and hardware that will eliminate hologram drift and provide a more stable application environment during high-precision 6DoF tracking — features that might give future developers of enterprise-class AR solutions some guidance as to how to differentiate their glasses from far less expensive consumer models that are coming to market this year. In the future, the Swiss researchers plan to use true depth-sensing cameras, such as Microsoft’s Azure Kinect, to improve accuracy and automation of the process for surgical needs.
