The other reality of biopsies: HoloLens for medicine

Augmented reality-based medical assistant

A project that was born from the conversation of a group of doctors, who sketched it on a napkin in the bar where they were chatting; it was promoted by the Spanish Foundation for Health Innovation and selected by the Massachusetts Institute of Technology (MIT) and changed completely its initial approaches less than a month after being presented to the jury. It was awarded and today is a reality of the Integrated Health Organization Araba (Basque Osakidetza), whose technological development has been carried out by the company Plain Concepts from Leon, which works with Microsoft and has used its Hololens to shape the project. It is currently working on improving it with Hololens 2, while the idea is already a technological reality ready to be used in real patients.

This is an augmented reality-based medical assistant, “designed to guide the doctor and provide him with a complete patient history so that he has all the information to help him during the performance of the biopsies,” explains Pablo Pelaez, CEO of Plain Concepts and regional director of Microsoft. “The use of Hololens, the augmented reality glasses, guides the doctor during the biopsy while providing him with all the information regarding the patient”.

OSI Araba Medical Team Project Selected for Massachusetts Institute of Technology Innovation Accelerator

The application allows the physician to scan a patient’s QR code, connect to the hospital’s PACS, and receive all the necessary information, such as CT images that would be reproduced in 3D by HoloLens. prior to performing the biopsy. The innovative application has been developed by the Plain Concepts research team using the Evergine 3D development engine.

To put it simply, Pelaez points out that six metal ‘cores’ that appear on the computerized tomography (CT, or CAT) scan are placed in the patient’s body, where a tumor is supposed to be and the biopsy is going to be performed. The images appear in 3D in front of the physician when he places the augmented reality glasses, and on them, with his hands-free or with voice instructions, he can make “cuts” and indicate exactly where he wants to perform the puncture.

Less than a month after presenting the project to the jury, the team reconsidered it to use augmented reality technology

On this image that point is fixed, and also the path that the needle must take so as not to damage other organs, which also appear on the image. “When the physician decides on the best path to take to reach the biopsy point, the image is placed on the patient through these ‘witnesses,’ and a line is marked with the path that the needle must follow, from its inclination to the depth it must reach. On this path, the physician performs the biopsy in a more informed and safe manner.

3d Application with HoloLens 2

Plain Concepts has developed this application on HoloLens, Microsoft’s holographic glasses. “The glasses project a 3D image in front of you, a three-dimensional optical effect that in the case of Araba’s medical service “allows a graphic visualization of a patient’s scan, on which the doctor can decide where to take the sample in the biopsy“.

The system transforms a medical CT test into a functional hologram, and with voice commands for each of the actions it allows moving, rotating, scaling, cutting planes, marking targets, or handling markers. It uses artificial vision with the ability to identify the markers on the patient in the real world and position them on his body.

Pablo Peláez points out that “a generalized problem in medicine is that it does not have tools that make it possible to see the patient’s images in a real way, in three dimensions. Nor do they have systems that allow them to act in a guided manner in medical procedures.

With the system developed, “the 3D medical test is projected by means of holograms on the patient’s body, which allows it to act on it in a clearer and more intuitive way; and in the way it has to proceed in order to take the sample in the appropriate way. In addition, it allows the physician to have his hands free throughout the process and to interact with the application by voice commands. This technology “will make it possible to save costs in medicine, as well as to make faster and more accurate diagnoses”.

Healthcare Innovation

This whole innovative system is the result of the idea and work of four specialists from OSI Araba, three in medical physics, and one in nuclear medicine. A group that, as explained by Marian García Fidalgo, head of the Medical Physics and Radiological Protection Service of the health institution, at TEDxVitoriaGasteiz, believes “in innovation, understood as the introduction of changes that improve the quality of life of our patients. A team that fights against cancer and knows the importance of early diagnosis. “How soon a tumor is diagnosed is fundamental. In order to find out, a biopsy is performed, in which we use imaging systems that show the inside of the patient, his structures, and the needle with which we reach the area where the puncture is performed”.

The idea came from a conversation in a cafeteria, “and we put it on a napkin. Then we built it at home and showed so much enthusiasm by showing it to the hospital’s innovation committee that they supported us from the very beginning”.

The next step was to manufacture it in 3D, so we could test the accuracy of mannequins. “We would place it on the patient, more or less where we wanted to make the puncture, and through a calculation program developed for this system, we would trace the inclination that the needle should have, where the tumor was that we wanted to reach and what the depth of the puncture should be.

It was the Spanish Foundation for Innovation in Health that looked at the project and encouraged the team to apply for an acceleration program for innovation projects, which they passed. “We were one of the 15 projects selected by MIT to develop it”.

Garcia Fidalgo acknowledges that his idea “liked it, but it didn’t excite. We worked for months with the experts to perfect aspects that physicists like us are quite alien to, even improving the technological part, by robotizing it. We thought that would add value to the project. When we were already in the final stretch, one of the specialists from MIT explained to us that this would not really succeed in the market”.

The head of the Medical Physics Service explains how they consulted on where the development of the augmented reality glasses technology was at that time. “And we set out, less than three weeks from the end, to turn our project over to an augmented reality-based biopsy assistant. The idea was that the professional was going to do more or less the usual thing, only that he had glasses that would allow him to have his hands free and sterile. And with simple hand gestures or voice instructions, he would be able to handle real-time images of the hologram, of all the patient’s cuts”.

Discover more information in the case study, Augmented Reality guided biopsies.

Original news