AI-CHD: An AI-Based Framework for Cost-Effective Surgical Telementoring of Congenital Heart Disease

Congenital heart disease (CHD), the most common congenital birth defect, has long been known as one of the main causes of infant death during the first year of life.¹ More than one million of the world’s approximately 135 million newborns are born each year with CHD.²¹ Over the last century, cardiac surgery has been an effective approach to tackling CHD; its remarkable advance has decreased the mortality rate of newborns with CHD.¹⁰

Back to Top

However, that lower mortality rate is mostly observed in developed countries rather than developing ones. Surgical treatment of CHD requires highly skilled surgeons along with complex infrastructures and equipment. While developed countries have perfected their treatment of CHD for more than 50 years, developing countries are still in the early stages. It is estimated that the number of congenital cardiac surgeons needs to increase by 1,250 times to satisfy only the basic needs of CHD treatment worldwide,¹⁶ and most of those surgeons reside in developed countries. As a result, the mortality rate in developing countries is currently at 20%, strikingly higher than the 3% to 7% in developed countries,¹⁶ not to mention the fact that mortality rates in developing countries are likely underreported due to the lack of proper diagnosis.

Back to Top

Remote Surgery

Remote surgery has been an active field for decades, enabling experienced surgeons to remotely instruct robots (telerobotics) or guide less-experienced surgeons (surgical telementoring).⁸ It enables high-quality surgical expertise to be passed from surgeons in developed countries to those in developing ones, or from high-end urban medical centers to rural hospitals inside developing nations.

Telerobotics enables surgeons to remotely control robots in a master-slave relationship. Stable camera systems are implemented in both sites. At the robot site, multiple cameras construct a virtual image of the operative field, which is provided to the surgeon site. At the surgeon site, multiple cameras and 3D imaging systems capture the surgeon’s hand movements, which are sent to the robot site and emulated by the robot operating on the patient. Images and movements are required to be transmitted in real time, but large latency may affect the surgeon’s performance or even lead to surgical failure. Telerobotics can also let surgeons sit comfortably while performing some delicate operations requiring fine movements. However, due to the highly demanding nature and potential risks, telerobotics has very limited clinical application. Only a few systems, including da Vinci¹⁴ and Zeus,¹³ are approved for use. At present, telerobotics is still in its early stages.

Surgical telementoring, on the other hand, consists of an expert surgeon remotely guiding a less-experienced surgeon. Such guidance is achieved with real-time audio and video transmission. Thus, the two surgeons can discuss the procedure in real time, and the expert surgeon can deliver precise guidance based on the real-time video streaming of the surgery process. Surgical telementoring can be performed in rural areas and austere environments, not only for difficult surgeries but also for surgery education. Like telerobotics, surgical telementoring requires real-time data transmission. Fortunately, 4G and 5G communication technologies have made this possible across great distances. Surgical telementoring has been widely adopted and explored in clinical use thanks to improved transmission quality, and the technology carries less potential risk compared with telerobotics.¹¹

Figure 1. Surgical telementoring: Technical assistance and guidance via real-time video and audio streams.

Though also in its early stage, surgical telementoring is still more mature than telerobotics, with its relatively lower cost and lower technical complexity. However, surgical telementoring for CHD still faces challenges. First, cardiac surgery for the treatment of CHD is rather complex, generally regarded as the jewel in the crown of surgery. As such, CHD diagnosis and surgical planning are usually time-consuming and costly; delivering this expertise to developing countries or rural areas can often be time- and cost-prohibitive. For example, examining the medical image of a CHD patient for diagnosis takes even a very experienced radiologist several hours, whereas that time is usually on the order of minutes for common heart diseases. Second, the machine quality and operator skill in developing countries or rural areas may be limited, leading to issues such as low imaging quality under non-ideal settings.

Back to Top

AI-CHD

One potential solution to reduce costs is to use artificial intelligence (AI) to automatically construct accurate 3D models of the heart from medical images, a critical yet otherwise time-consuming process in CHD surgical telementoring. Before surgery, this model can help during remote surgical planning and discussions via 3D printing or virtual reality (VR) technologies. During surgery, viewing the model via a 2D screen can enhance operational efficiency by fostering communication between the surgeon and the novice.

Our novel solution, an AI-based framework called AI-CHD, aims to construct accurate and efficient heart models for surgical telementoring of CHD based on 3D computed tomography (CT) images. Considering that the artifact type and pattern in CT images acquired with medium- and low-end machines or by users with limited skills may be different from those in standard training sets, the framework first exploits a weakly supervised way to remove artifacts in a CT image, which does not require a prior training set. Further, considering that hearts in CHD exhibit large variations in structure and/or great vessel connections without local tissue changes, the framework then deploys hybrid deep-learning and graph analysis to tackle the model construction. We evaluate each step with collected datasets and the overall system with a case study.

Figure 2. Overall flow of AI-CHD.

Medical image artifact reduction. Medical images exhibit various types of artifacts, with different patterns and mixtures that depend on many factors, including scan setting, machine condition, patient size and age, surrounding environment, etc. This problem is even worse on middle-of-the-road and low-end CT imaging machines—often operated by less-skilled technologists—which are common in rural areas in developing countries. On the other hand, existing deep learning-based artifact-reduction methods for medical images are restricted by the specific training data that contains predetermined artifact types and patterns, which can hardly capture all possibilities exclusively. Accordingly, they can only work well under the scenarios defined by the training data. In this step, we exploit the power of deep learning but without using pre-trained networks for medical artifact reduction. Specifically, at test time we train a lightweight, image-specific, artifact-reduction network using data synthesized from the input image. Without requiring any prior training data, our method can work with almost any medical images that contain varying or unknown artifacts.

The main flow of artifact reduction contains two modules: artifact synthesis and artifact removal.³ In the first module, radiologists need to annotate a total of 10–20 regions of interest (RoI) from a 3D-input CT image. These RoI are further used to train a lightweight, five-layer synthesis network, which synthesizes a large number of paired patches. Note that as different medical images have different ranges of pixel values, we normalize pixel values so that each has a value between 0 and 1. With synthesized paired patches, theoretically any existing CNN-based artifact-reduction networks can be trained. However, a key issue here is that we perform the task on each input image. Deep and complex networks may need many data pairs and long training times. Smaller networks, on the other hand, may not attain desired performance. Thus, in the second module, we resort to an artifact-removal network—a compact, attentive generative network architecture that can pay special attention to artifacts and train them adversarially for faster convergence. It is formed via a two-step attentive-recurrent network followed by a 10-layer contextual autoencoder to reduce artifacts and restore the information obstructed by them. Once trained, the artifact-removal network is applied to all slices of the 3D volume for artifact reduction.

Fundamental to our method is the fact that artifacts in most medical images exhibit localized patterns—that is, they do not cover the entire image uniformly. It is almost always possible to identify “clean” regions (with few artifacts) and “dirty” regions (with significant artifacts) within an image. This makes it possibile to synthesize paired, dirty-clean training patches from an image with artifacts. In addition, as visual entropy inside a single image is much smaller than in a general external collection of images,³⁰ the synthesized training data does not need to be big, and the associated artifact-reduction network can be compact and converge quickly.

---

AI-CHD is an accurate, AI-based framework for surgical telementoring of CHD developed through deep collaborations between computer scientists, radiologists, and surgeons.

---

We evaluate the performance of this step with CT images containing different levels of Poisson noise collected by our wide-detector, 256-slice MDCT scanner with 8 cm of coverage, using the following protocol: collimation, (96–128)X0.625 mm; rotation time, 270 ms, which corresponds to a 135ms standard temporal resolution; slice thickness, 0.9 mm; and reconstruction interval, 0.45 mm. Adaptive-axial z-collimation was used to optimize the cranio-caudal length. Data was obtained at 40%-50% of the RR interval, using a 5% phase tolerance around the 45% phase. All CT images are qualitatively evaluated by our radiologists on structure-preservation and artifact levels. For quantitative evaluation, due to the lack of ground truth, we follow most existing works^24,28 and select the most homogeneous area in regions of interest chosen by radiologists. Standard deviation (artifact level) of the pixels in the area should be as low as possible.

Our method trains and tests on each individual patient’s image, and no pre-training is involved. We compare our method with state-of-the-art, deep learning-based, medical-image artifact-reduction methods on a cycle-consistent adversarial denoising network (CCADN),⁹ which is trained following the exact same reported settings. The CT training data set for the CCADN contains 100,000 image patches. We consider both the ideal situation, where test images only contain Poisson noise levels such as those in the training set, and non-ideal situations, where test images have higher noise levels. Figure 3(a) shows the results. Qualitatively, our method and CCADN preserve structures well for both ideal and non-ideal situations. Our method outperforms CCADN even at noise levels that the CCADN is trained to reduce—that is, the regions in Figure 3(a)(1). Quantitatively, our method beats CCADN in both ideal and non-ideal situations, achieving up to 29.2% lower standard deviation and 18.6% on average.

Figure 3. Step-by-step performance of AI-CHD.

Though our method is trained and tested on each test image, it has almost the same execution time compared with CCADN, due to the significantly reduced network complexity and faster convergence brought by the internal visual entropy.

Medical image segmentation. CHD usually comes with significant variations in heart structures and great-vessel connections, which renders general whole-heart and great-vessel segmentation methods^18,22 in normal anatomy ineffective. Most existing segmentation methods are only dedicated to CHD target blood pool and myocardium.^23,29 Recently, semi-automated segmentation in CHD has also been explored,¹⁷ requiring users to locate an initial seed. However, fully automated segmentation of whole-heart and great-vessel segmentation in CHD remains a missing piece in the literature. Inspired by the success of graph matching in several applications with large variations,¹² we propose to combine deep learning²⁵ and graph matching for fully automated whole-heart and great-vessel segmentation in CHD.²⁶ Particularly, we leverage deep learning to segment the four chambers and myocardium followed by blood pool, where variations are usually small and accuracy can be high. We then extract the vessel connection information and apply graph matching to determine the categories of all the vessels.

The overall flow for whole-heart and great-vessel segmentation—left ventricle (LV), right ventricle (RV), left atrium (LA), right atrium (RA), myocardium (Myo), aorta (Ao), and pulmonary artery (PA)—contains two modules: whole-heart segmentation and great-vessel segmentation. In whole-heart segmentation, RoI cropping is first presented to extract the area that includes the heart and its surrounding vessels. We resize the input image to a low resolution of 64X64X64 and then adopt the same segmentation-based extraction as in Payer et al.¹⁸ to get the RoI. The RoI are then resized to 64X64X64 and fed into a 3D U-net⁵ for segmentation.

In great-vessel segmentation, blood-pool segmentation is conducted on each 2D slice of the input using a 2D U-net¹⁹ with an input size of 512X512. Note that to detect the blood-pool boundary for easy graph extraction in graph matching later, we add another class: blood-pool boundary in the segmentation. With the high-resolution blood segmentation, whole-heart segmentation achieves chamber and myocardium refinement by refining chamber and myocardium boundaries. By removing the blood pool corresponding to the low-resolution, whole-heart segmentation, great-vessel segmentation obtains the blood pool corresponding to the great vessels and adopts graph matching to identify Ao, PA, and anomalous vessels.

For evaluation, we collected a dataset composed of 68 3D CT images captured by a Siemens Biograph 64 machine. The ages of the associated patients range from one month to 21 years, with the majority between one month and two years. The size of the images is 512 X 512X(130–340), and the typical voxel size is 0.25X0.25X0.5 mm³. The dataset covers 14 types of CHD, including six common types—atrial septal defect (ASD), atrio-ventricular septal defect (AVSD), patent ductus arteriosus (PDA), pulmonary stenosis (PS), ventricular septal defect (VSD), and co-arctation (CA)—plus eight less-common ones—Tetrology of Fallot (ToF), transposition of the great arteries (TGA), pulmonary artery sling (PAS), anomalous drainage (AD), common arterial trunk (CAT), aortic arch anomalies (AAA), single ventricle (SV), and pulmonary atresia (PuA).

Figure. Mean and standard deviation (SD) of Dice score of the state-of-the-art method, Seg-CNN,¹⁸ and our method (in %) for seven substructures of whole-heart and great-vessel segmentation.

All labeling was performed by experienced radiologists, and the labeling time per image was two to three hours. The labels include seven substructures: LV, RV, LA, RA, Myo, Ao, and PA. For easy processing, venae cavae (VC) and pulmonary vein (PV) are also labeled as part of RA and LA respectively, as they are connected, and their boundaries are relatively hard to define. Anomalous vessels are also labeled as one of the above seven substructures based on their connections. The comparison with Seg-CNN¹⁸ is shown in the Table. Our method can achieve a mean Dice score between 5.8% and 19.2% higher across the seven substructures (12% higher on average) with almost the same standard deviation. The highest improvement is achieved in Ao, due to its simple graph connection with successful graph matching. The smallest improvement is obtained in myocardium because it is not well considered in the high-resolution blood-pool segmentation. Figure 3(b) shows visualization of CAT segmentation using our method and Seg-CNN. Our method can clearly segment Ao and PA with some slight mis-segmentation between PA and LA. However, Seg-CNN segments the main part of Ao as PA, since pixel-level segmentation by U-net is based only on the surrounding pixels, and the connection information is not well exploited.

Back to Top

Real-Time Video and Audio Transmission

Real-time video and audio transmission is also a critical part of surgical telementoring. Such transmission needs high data rates and low latency so that important decisions can be made in real time to avoid potential complications. In addition, wireless transmission is always preferred over wired transmission in an operating room. The most common wireless transmission method so far, 4G, offers about 50-ms latency and a 10-Mbps average data rate. Such moderate transmission quality and speeds can support less-complex telemonitoring procedures, such as addiction management²⁰ and training.²

5G wireless communication is emerging to offer about 10-ms latency (1 ms in special cases) and higher data rates of 50+ Mbps on average. Such high transmission quality has enabled many complex procedures, including intestinal tumor procedure,¹⁵ liver removal of a laboratory test animal,⁶ laparoscopic cholecystectomy,²⁷ and gall bladder surgery.⁴ However, surgical telementoring has not been reported in cardiac surgery applications for CHD treatment, possibly due to its extremely high complexity and high risk.

Back to Top

AI-CHD Case Study

On April 3, 2019, we used AI-CHD to perform China’s first 5G-based heart telementoring procedure, with collaboration between Guangdong Provincial People’s Hospital (GPPH) and the People’s Hospital of Gaozhou (PHG). Guangdong Mobile Communications Group and Huawei provided 5G capabilities. AI-CHD produced an accurate heart model of the patient, which was used for pre-surgical planning and training, and for real-time guidance during the surgery.

The 41-year-old patient, Ms. Green (alias), from Gaozhou, Guangdong Province, was experiencing shortness of breath, chest pain, difficulty walking, and insomnia. She was diagnosed with atrial septal defect (ASD) with tricuspid insufficiency, severe pulmonary hypertension, and cardiac failure at PHG. ASD is a moderate type of CHD that can be treated through a relatively easy, low-risk operation at a young age; however, her condition had not been detected due to lack of screening availability in the rural area where she lived. And, the untreated ASD brought about other conditions, such as severe pulmonary hypertension and heart failure. Thus, her condition changed from a relatively simple CHD to a complex one. Only surgery could save her heart.

Being in a less-developed region, PHG surgeons only have experience in conventional open-heart surgery, a higher-risk procedure, especially for a long-suffering patient such as Ms. Green. A minimally invasive, lower-risk surgery was preferred. However, Ms. Green would have had to travel 250 miles to GPPH, one of China’s largest cardiac medical centers, and the nearest hospital with surgeons capable of performing such a complex procedure. Considering Ms. Green’s weakened condition, the journey was not feasible. Therefore, telementoring was the most suitable approach. With comprehensive discussion and analysis, Dr. Huiming Guo, GPPH’s chief physician of cardiac surgery, agreed to serve as telementor in this surgery. Dr. Guo brought much expertise with minimally invasive surgeries to the table and is well known internationally in the surgical treatment of CHD.

Before Ms. Green’s surgery, we first collected her 3D cardiac CT images from PHG. AI-CHD then produced her 3D heart model, see Figure 4(a), with a clinically acceptable accuracy of 0.81 (Dice score) for the surgery. Runtime was less than two minutes, much faster than manual segmentation—which could take two to three hours or even longer—thus significantly reducing the cost. With the 3D heart model, surgical planning and training were then performed. As shown in Figure 4(b), the heart model was first printed out with a 3D printer (thin vessels of pulmonary artery and pulmonary vein were removed, as they were not related to the surgery). The printed model provided a straightforward view of the heart’s appearance and structure and showed where the problem was. Dr. Guo used the printed model to discuss the surgical plan with other members of his team and the remote team at PHG. Once the plan was set, virtual surgical training was carried out via VR, shown in Figure 4(c).

Figure 4. AI-CHD case study.

As seen in Figure 4(d), VR enables doctors to enlarge and shrink specific heart structures, including its inner structures, and to perform virtual operations such as infusion and suture as a practice. In this way, Dr. Guo could get a comprehensive understanding of the surgery to be telemonitored and establish the best operations/parameters for effective guidance.

After Dr. Guo confirmed the surgical plan and the detailed process with the help of AI-CHD, telementoring of Ms. Green’s surgery began at 9:35 AM on April 3, 2019. Figure 4(d) shows pictures of the guidance team at GPPH and the actual surgery at PHG. The procedure featured four real-time video streams: the view of the surgeon, the corresponding VR view of the heart, the telementor (Dr. Guo), and the operating room. Since this was China’s first CHD telementoring surgery, doctors from the cardiac surgery, cardiac ultrasound, and cardiac imaging departments all showed up to observe the event, as seen in the telementor view.

Based on the real-time view of the surgeon and the corresponding VR scene, Dr. Guo could easily recognize the current view of the heart and offer immediate guidance via a real-time audio stream. For example, when determining the opening point at the pericardium, the surgeon asked, “Should the pericardium be opened here?” Dr. Guo answered instantaneously, “Move up three centimeters.” For the suture in the operation, which is also a key part of the surgery, Dr. Guo reminded the surgeon, “Do not be too close to the Koch’s triangle when stitching; otherwise, it is easy to cause myocardial injury and block the heart-beating rhythm conduction.” Koch’s triangle was drawn in VR video and shown to surgeons in the operating room 250 miles away in real time.

Throughout the telemonitoring, the transmission rate for video streams is stabilized at around 25 Mbps with a latency of 30 ms. The surgery went smoothly and finished at 1:00 PM. The heart was sutured, Ms. Green’s heart resumed beating, and after a week in recovery, she was discharged. The post-operative review showed that pulmonary artery pressure and mitral regurgitation were within the normal range. The patient remains in good health as of the writing of this article (Dec. 1, 2019). Our case study has received extensive media coverage in some of the biggest and most influential news organizations in China, including Xinhua²⁸ and Global Times.⁷

Back to Top

Looking Forward

This is the third year of collaboration between computer scientists, cardiac surgeons, and radiologists in our team. The original work of artifact reduction and segmentation of cardiac CT images has gradually evolved into a holistic system of 3D heart-model construction. In the future, in addition to further optimizing AI-CHD, we plan to explore the following four promising areas for automatic and cost-efficient treatment of CHD:

Artifact-aware segmentation. The current approach involves two steps: artifact reduction and segmentation. However, it may be possible to improve efficiency by performing segmentation directly on noisy images in just one step, as shown in Figure 5(a). We believe this is promising for two reasons: First, artifacts typically display some patterns, making it possible to capture and remove them and segment targets jointly in one neural network. Second, while it may be a problem to obtain the training dataset, we can use our artifact reduction method or other existing methods to get clean images and then perform manual labeling.

Figure 5. Future work of AI-CHD on medical image-artifact reduction and segmentation: Artifact-aware segmentation (a) that directly performs segmentation on noisy images rather than artifact reduction first and then segmentation, and graph-aware segmentation (b) that takes graph information among thin vessels into consideration for accurate segmentation.

Graph-aware segmentation. Our method and other existing methods are still challenged to correctly segment thin vessels, especially thin PA vessels. The main reason is that the rich connection information among these vessels is not well exploited. As shown in Figure 5(b), we may extract the graphs of these thin vessels from blood-pool segmentation results and whole-heart and great-vessel segmentation results to represent connection information. Then, graph-aware analysis that takes both segmentation and connection information into consideration can be performed to obtain more accurate segmentation results.

Automatic diagnosis. Accurate diagnosis of CHD is more significant compared with artifact reduction and segmentation. The lack of CHD diagnosis experience in developing regions means many cases are not diagnosed correctly and miss timely treatment.¹⁶ To gain the expertise to be able to make such a diagnosis, radiologists require more than 10 years of training, which can be time-consuming and costly. Even experienced radiologists may need up to a half-hour to diagnose a patient with CHD. Thus, automatic CHD diagnosis is preferred, for its ability to provide large-scale, high-quality, cost-efficient medical care. To be clinically acceptable, automatic CHD diagnosis also needs to report the features or reasons for the diagnosis with an accompanying confidence score. Radiologists could more easily verify the results; low confidence scores would denote cases in need of manual diagnosis.

Automatic surgery planning. Due to the large structure variations in CHD, dozens of surgical procedures exist, each containing parameters such as opening point, incision size, direction. Currently, surgeons plan based on their experience, which may or may not be the optimal choice in terms of prognosis. We will further extend AI-CHD to enable accurate automatic surgical planning for optimal treatment.

Back to Top

Conclusion

AI-CHD is an accurate, AI-based framework for surgical telementoring of CHD. It is developed through deep collaborations between computer scientists, radiologists, and surgeons. The technology enables cost-effective and timely model construction of hearts in CHD, which assists radiologists and surgeons with performing efficient surgical planning and training in CHD surgery, as demonstrated by the case study. AI-CHD can reduce costs while improving the quality of CHD surgery telementoring in developing countries and regions.

Back to Top

Acknowledgments

This work was approved by the Research Ethics Committee of Guangdong General Hospital, Guangdong Academy of Medical Science under Protocol No. 20140316. It was supported by the National Key Research and Development Program of China (2018YFC1002600), the Science and Technology Planning Project of Guangdong Province, China (No.2017B090904034, 2017B030314109, 2018B090944002, 2019B020230003), Guangdong Peak Project (DFJH201802), and the National Natural Science Foundation of China (No.62006050).