作者投稿
专家审稿
编辑办公
邮件订阅
RSS
Research Progress on Generative Adversarial Network in Cross-modal Medical Image Reconstruction
-
摘要: 单一模态的医学影像所包含的疾病特征信息有限，临床医生可通过综合分析多种模态的医学影像信息以明确诊断，但由于医疗资源及诊疗时间受限，医生一般无法在短时间内获得所需的多模态影像信息。跨模态医学影像重建技术能够生成临床所需的多种模态医学影像，有望辅助临床医生对疾病进行精准诊疗。目前，传统跨模态重建技术已实现部分临床场景的应用，但重建影像的生成质量有待进一步提高，生成对抗网络可重建出临床所需的高质量多模态医学影像，最大程度地节约医疗资源并缩短患者就诊时间。本文就生成对抗网络在X线、CT、MRI、PET等多模态影像之间的跨模态重建应用研究作一综述，以期为开发更先进的跨模态重建技术提供借鉴。Abstract: Single-modal medical images contain limited disease-specific information. To analyze and diagnose patients, clinicians often need to integrate multiple modal images. However, due to limited medical resources and treatment time, it may be difficult to obtain multi-modal images. Cross-modal image reconstruction can generate medical images for clinical needs, thus assisting clinicians in accurately diagnosing and treating diseases. Traditional cross-modal reconstruction techniques have been applied in some clinical scenarios, but the quality of the reconstructed images needs further improvement. Generative adversarial network (GAN) can recover high-quality and complete image data from low-quality or incomplete medical image data, maximally savings medical equipment resources and accelerating medical treatment speed. This article summarizes the applications of GAN technology in cross-modal image reconstruction across X-ray imaging, computed tomography imaging, magnetic resonance imaging, and positron emission tomography imaging, to provide reference for the development of more advanced cross-modal reconstruction techniques.作者贡献：李卓远、徐国豪负责论文撰写和修订；王峻晨、王赛硕负责文献检索；翟吉良、王传涛负责论文审校。利益冲突：所有作者均声明不存在利益冲突
-
表 1 X射线与CT之间的跨模态重建模型
重建方向 部位 模型名称 模型改进 重建结果 X线→3D结构[18] 脊柱 X-CTRSNet 将用于重建的SpaDRNet模块、用于分割的MulSISNet模块以及RSC学习模块结合 重建并分割出椎骨3D结构，可应用于手术导航等领域 单平面X线→CT[21] 胸部 X2CT-GAN 仅使用单个前后位X线进行CT重建；使用KNN或PCA预测方法补充缺失影像 涉及肿瘤的影像重建时，重建得到的影像结构不完整 双平面X线→CT[19] 胸部 X2CT-GAN 使用DRR技术合成X线用以训练网络; 使用DenseNet构建生成器；提出一种全新的跳跃连接结构 重建得到的肺部影像表面结构清晰合理，但小解剖结构仍存在伪影 双平面X线→CT[20] 脊柱 2D to 3D GAN 使用基于GPU的层析迭代工具的反投影合成X线；使用ResNet代替DenseNet进行特征提取 重建结果可提供更多植入物的细节信息，更具有时间成本效益 双平面X线→CT[22] 胸部 GA-GAN 生成器中引入空间和通道的注意力模块；使用矢量量化法进行重建 模型内存使用减少的同时可提高重建CT影像质量 X-CTRSNet：用于重建的互耦合SpaDRNet模块、用于分割的MulSISNet模块以及用于任务一致性的RSC学习模块相结合的架构；X2CT-GAN：X线到计算断层扫描生成对抗网络；2D to 3D GAN：二维到三维生成对抗网络；GA-GAN：引导注意力机制生成对抗网络；KNN：K近邻预测方法；PCA：主成分分析预测方法；DRR：数字图像重建技术；DenseNet：密集连接网络；GPU: 图形处理器；ResNet：残差网络 
下载: 导出CSV
表 2 CT与MRI之间的跨模态重建模型
重建方向 部位 模型名称 模型改进 重建结果 CT→MRI[24] 肺部 GAN 引入一种针对肿瘤的特异性感知损失 重建的MRI影像可更好地保留肿瘤细节信息 CT→MRI[25] 脑部 MR-GAN 结合双循环一致损失和体素损失 重建结果明显优于使用未配对数据的结果 MRI→CT[26] 骨盆 CGAN 使用配对好的切片数据训练CGAN 重建的CT影像有助于前列腺癌患者剂量计算 MRI→CT[32] 脑部 Cycle GAN 密集模块构建Cycle GAN生成器 实现较短时间内重建高质量CT影像 MRI→CT[34] 腹部 Cycle GAN 新增密集模块和新型复合损失函数 重建结果相比于其他模型结构一致性提高 MRI→CT[33] 腹部 3D Cycle GAN 定义新的结构一致性损失 重建效果明显优于传统Cycle GAN MRI→CT[27] 咽喉 GAN+U-Net 使用改进的U-Net网络构建生成器 重建结果更接近真实CT影像 CT=MRI[31] 心脏 Cycle GAN 将形状一致性损失融入到Cycle GAN 可解决配对跨模态心脏数据获取困难的问题 CT=MRI[28] 脑部 UagGAN 使用配对数据进行预训练；之后在未配对的数据上重新训练 MRI→CT重建效果良好；CT→MRI重建效果一般 CT=MRI[30] 盆腔 Cycle GAN 使用FCN构建生成器和判别器 重建的盆腔CT影像符合临床剂量精度要求 GAN：同表 1；MR-GAN: 磁共振生成对抗网络；CGAN: 条件生成对抗网络；Cycle GAN：循环生成对抗网络；U-Net：U型卷积神经网络；UagGAN：无监督注意力引导生成对抗网络; FCN：全卷积神经网络
下载: 导出CSV
表 3 MRI与PET之间的跨模态重建模型
重建方向 部位 模型名称 模型改进 重建结果 MRI→PET[38] 脑部 BMGAN 提出3D-Dense U-Net构建生成器 重建的PET影像可辅助MRI影像进行阿尔茨海默症分类 MRI→PET[39] 脑部 Cycle GAN 3D CGAN网络和LM3IL网络分别学习MRI与PET之间的双向映射 重建的PET影像与MRI影像共同进行脑疾病分类效果更好 MRI→PET[42] 脑部 Sketcher-Refiner GAN 先使用Sketcher大体绘制结构信息; 再使用Refiner细致绘制影像纹理信息 重建的PET影像在病变区域可保留更多细节信息 MRI→PET[41] 脑部 3D GAN 提出两个阶段的3D-GAN框架 重建的PET影像可作为医生诊断的客观补充 MRI→PET[40] 脑部 TPA-GAN 将金字塔卷积模块和注意力模块引入到GAN中 重建结果有助于插补缺失影像和脑部疾病诊断 PET→MRI[43] 脑部 CGAN 使用U-Net构建生成器 能够重建出逼真的MRI影像，但存在一定的伪影 Cycle GAN、CGAN、U-Net：同表 2；PET：正电子发射断层显像；BMGAN：双向映射生成对抗网络；Sketcher-Refiner GAN：草图生成-精细生成生成对抗网络；3D GAN：3D生成对抗网络；TPA-GAN：任务导向的金字塔和注意力生成对抗网络; 3D-Dense-U-Net：3D密集连接U型卷积神经网络；3D CGAN: 3D循环生成对抗网络；LM3IL：基于地标的多模态实例学习网络
下载: 导出CSV
表 4 CT与PET之间的跨模态重建模型
重建方向 部位 模型名称 模型改进 重建结果 CT→PET[46] 肝脏 FCN-CGAN 使用全卷积神经网络构建生成器；使用自定义损失函数 无需手动标记病变区域数据 PET→CT[47] 肺部 Cycle GAN 使用自注意力U-Net构建生成器 重建的CT影像与真实CT影像相似且在软组织上显示出良好的对比度 PET→CT[48] 脑部 Med-GAN 将多个全卷积神经网络通过跳跃连接结构连接在一起，构建生成器 重建得到高分辨率的CT影像 PET→CT[49] 脑部 UP-GAN 将几个Med-GAN模型级联 重建得到保真度增加的CT影像 PET: 同表 3；Cycle GAN、U-Net：同表 2；FCN-CGAN：全卷积条件生成对抗网络；Med-GAN：医学影像转换生成对抗网络；UP-GAN：渐进式生成对抗网络
下载: 导出CSV
-
[1] Yu B, Wang Y, Wang L, et al. Medical image synthesis via deep learning[J]. Adv Exp Med Biol, 2020, 1213: 23-44. [2] Luo Y, Nie D, Zhan B, et al. Edge-preserving MRI image synthesis via adversarial network with iterative multi-scale fusion[J]. Neurocomputing, 2021, 452: 63-77. doi: 10.1016/j.neucom.2021.04.060 [3] Avants BB, Epstein CL, Grossman M, et al. Symmetric diffeomorphic image registration with cross-correlation: evaluat-ing automated labeling of elderly and neurodegenerative brain[J]. Med Image Anal, 2008, 12: 26-41. doi: 10.1016/j.media.2007.06.004 [4] Roy S, Carass A, Shiee N, et al. MR contrast synthesis for lesion segmentation[C]. Proc IEEE Int Symp Biomed Imaging, 2010, 2010: 932-935. [5] Ye DH, Zikic D, Glocker B, et al. Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization[C]. Med Image Comput Comput Assist Interv, 2013, 16: 606-613. [6] Huynh T, Gao Y, Kang J, et al. Estimating CT image from MRI data using structured random forest and auto-context model[J]. IEEE Trans Med Imaging, 2016, 35: 174-183. doi: 10.1109/TMI.2015.2461533 [7] Jog A, Carass A, Roy S, et al. Random forest regression for magnetic resonance image synthesis[J]. Med Image Anal, 2017, 35: 475-488. doi: 10.1016/j.media.2016.08.009 [8] Jiang J, Veeraraghavan H. Unified cross-modality feature disentangler for unsupervised multi-domain MRI abdomen organs segmentation[C]. Med Image Comput Comput Assist Interv, 2020, 12262: 347-358. [9] Xiong Z, Stiles MK, Yao Y, et al. Automatic 3D surface reconstruction of the left atrium from clinically mapped point clouds using convolutional neural networks[J]. Front Physiol, 2022, 13: 880260. doi: 10.3389/fphys.2022.880260 [10] Shen L, Zhao W, Xing L. Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning[J]. Nat Biomed Eng, 2019, 3: 880-888. doi: 10.1038/s41551-019-0466-4 [11] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Adv Neural Inf Process Syst, 2014, 27: 2672-2680. [12] Aggarwal A, Mittal M, Battineni G. Generative adversarial network: An overview of theory and applications[J]. Int J Inf Manage, 2021, 1: 100004. [13] Wang K, Gou C, Duan Y, et al. Generative adversarial networks: introduction and outlook[J]. IEEE/CAA J Autom Sin, 2017, 4: 588-598. doi: 10.1109/JAS.2017.7510583 [14] Singh NK, Raza K. Medical image generation using generative adversarial networks: A review[J]. Health Informatics J, 2021: 77-96. [15] Zhou J. Research on Generative Adversarial Networks and Their Applications in Image Generation[C]. IEEE Int Conf Adv Electr Eng Comput Appl, 2022: 1144-1147. [16] Ravi D, Blumberg SB, Ingala S, et al. Degenerative adversarial neuroimage nets for brain scan simulations: Applica-tion in ageing and dementia[J]. Med Image Anal, 2022, 75: 102257. doi: 10.1016/j.media.2021.102257 [17] Jecklin S, Jancik C, Farshad M, et al. X23D-Intraopera-tive 3D Lumbar Spine Shape Reconstruction Based on Sparse Multi-View X-ray Data[J]. J Imaging, 2022, 8: 271. doi: 10.3390/jimaging8100271 [18] Ge R, He Y, Xia C, et al. X-CTRSNet: 3D cervical vertebra CT reconstruction and segmentation directly from 2D X-ray images[J]. Knowl Based Syst, 2022, 236: 107680. doi: 10.1016/j.knosys.2021.107680 [19] Ying X, Guo H, Ma K, et al. X2CT-GAN: reconstructing CT from biplanar X-rays with generative adversarial networks[C]. Proc IEEE Comput Soc Vision Pattern Recognit, 2019: 10619-10628. [20] Yang CJ, Lin CL, Wang CK, et al. Generative Adversarial Network (GAN) for Automatic Reconstruction of the 3D Spine Structure by Using Simulated Bi-Planar X-ray Images[J]. Diagnostics, 2022, 12: 1121. doi: 10.3390/diagnostics12051121 [21] Schock J, Lan YC, Truhn D, et al. Monoplanar CT Reconstruction with GANs[C]. IPTA, 2022: 1-6. [22] Sinha A, Sugawara Y, Hirano Y. GA-GAN: CT reconstruction from biplanar DRRs using GAN with guided attention[J]. CORR, 2019. doi: 10.48550/arXiv.1909.12525. [23] Zhan B, Xiao J, Cao C, et al. Multi-constraint generative adversarial network for dose prediction in radiotherapy[J]. Med Image Anal, 2022, 77: 102339. doi: 10.1016/j.media.2021.102339 [24] Jiang J, Hu YC, Tyagi N, et al. Tumor-aware, adversarial domain adaptation from CT to MRI for lung cancer segmentation[C]. Med Image Comput Comput Assist Interv, 2018: 777-785. [25] Jin CB, Kim H, Liu M, et al. Deep CT to MR synthesis using paired and unpaired data[J]. Sensors, 2019, 19: 2361. doi: 10.3390/s19102361 [26] Maspero M, Savenije MHF, Dinkla AM, et al. Dose evaluation of fast synthetic-CT generation using a generative adversarial network for general pelvis MR-only radiotherapy[J]. Phys Med Biol, 2018, 63: 185001. doi: 10.1088/1361-6560/aada6d [27] 王蕾, 李媛茜. 基于生成对抗网络的CT图像生成[J]. 成都信息工程大学学报, 2021, 36: 286-292. https://www.cnki.com.cn/Article/CJFDTOTAL-CDQX202103008.htm [28] Abu-Srhan A, Almallahi I, Abushariah MAM, et al. Paired-unpaired Unsupervised Attention Guided GAN with transfer learning for bidirectional brain MR-CT synthesis[J]. Comput Biol Med, 2021, 136: 104763. doi: 10.1016/j.compbiomed.2021.104763 [29] Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]. Proc IEEE Comput Soc Vision Pattern Recognit, 2017: 2223-2232. [30] 吴香奕, 曹锋, 曹瑞芬, 等. 基于循环一致性生成对抗网络的盆腔伪CT生成方法[J]. 中国医学物理学杂志, 2021, 38: 21-29. https://www.cnki.com.cn/Article/CJFDTOTAL-YXWZ202101006.htm [31] Zhang Z, Yang L, Zheng Y. Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network[C]. Proc IEEE Comput Soc Vision Pattern Recognit, 2018: 9242-9251. [32] Lei Y, Harms J, Wang T, et al. MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks[J]. Med Phys, 2019, 46: 3565-3581. doi: 10.1002/mp.13617 [33] Yang H, Sun J, Carass A, et al. Unsupervised MR-to-CT synthesis using structure-constrained Cycle GAN[J]. IEEE Trans Med Imaging, 2020, 39: 4249-4261 doi: 10.1109/TMI.2020.3015379 [34] Liu Y, Lei Y, Wang T, et al. Liver synthetic CT generation based on a dense-Cycle GAN for MRI-only treatment planning[C]. Image Processing, 2020, 11313: 659-664. [35] Wang C, Yang G, Papanastasiou G, et al. DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis[J]. Inf Fusion, 2021, 67: 147-160. doi: 10.1016/j.inffus.2020.10.015 [36] Wang R, Bashyam V, Yang Z, et al. Applications of generative adversarial networks in neuroimaging and clinical neuroscience[J]. Neuroimage, 2023, 269: 119898. doi: 10.1016/j.neuroimage.2023.119898 [37] Zhang J, He X, Qing L, et al. BPGAN: Brain PET synthesis from MRI using generative adversarial network for multi-modal Alzheimer's disease diagnosis[J]. Comput Meth Prog Bio, 2022, 217: 106676. doi: 10.1016/j.cmpb.2022.106676 [38] Hu S, Lei B, Wang S, et al. Bidirectional mapping generative adversarial networks for brain MR to PET synthesis[J]. IEEE Trans Med Imaging, 2021, 4: 145-157. [39] Pan Y, Liu M, Lian C, et al. Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for Alzheimer's disease diagnosis[C]. Med Image Comput Comput Assist Interv, 2018: 455-463. [40] Gao X, Shi F, Shen D, et al. Task-induced pyramid and attention GAN for multimodal brain image imputation and classification in alzheimer's disease[J]. IEEE J Biomed Health Inf, 2021, 26: 36-43. [41] Yaakub SN, McGinnity CJ, Clough JR, et al. Pseudo-normal PET synthesis with generative adversarial networks for localising hypometabolism in epilepsies[C]. SASHIMI, 2019: 42-51. [42] Wei W, Poirion E, Bodini B, et al. Learning myelin content in multiple sclerosis from multimodal MRI through adver-sarial training[C]. Med Image Comput Comput Assist Interv, 2018: 514-522. [43] Choi H, Lee DS. Generation of structural MR images from amyloid PET: application to MR-less quantification[J]. J Nucl Med, 2018, 59: 1111-1117. doi: 10.2967/jnumed.117.199414 [44] Zhan B, Li D, Wang Y, et al. LR-CGAN: Latent representation based conditional generative adversarial network for multi-modality MRI synthesis[J]. Biomed Signal Process Control, 2021, 66: 102457. doi: 10.1016/j.bspc.2021.102457 [45] Zhan B, Zhou L, Li Z, et al. D2FE-GAN: Decoupled dual feature extraction based GAN for MRI image synthesis[J]. Knowl Based Sys, 2022, 252: 109362. doi: 10.1016/j.knosys.2022.109362 [46] Ben-Cohen A, Klang E, Raskin SP, et al. Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection[J]. Eng Appl Artif Intell, 2019, 78: 186-194. doi: 10.1016/j.engappai.2018.11.013 [47] Dong X, Wang T, Lei Y, et al. Synthetic CT generation from non-attenuation corrected PET images for whole-body PET imaging[J]. Phys Med Biol, 2019, 64: 215016. doi: 10.1088/1361-6560/ab4eb7 [48] Armanious K, Jiang C, Fischer M, et al. MedGAN: Medical image translation using GANs[J]. Comput Med Imag Grap, 2020, 79: 101684. doi: 10.1016/j.compmedimag.2019.101684 [49] Upadhyay U, Chen Y, Hepp T, et al. Uncertainty-guided progressive GANs for medical image translation[C]. Med Image Comput Comput Assist Interv, 2021: 614-624. [50] Tomar D, Lortkipanidze M, Vray G, et al. Self-attentive spatial adaptive normalization for cross-modality domain adaptation[J]. IEEE Trans Med Imaging, 2021, 40: 2926-2938. doi: 10.1109/TMI.2021.3059265 [51] Chen J, Wei J, Li R. TarGAN: target-aware generative adversarial networks for multi-modality medical image translation[C]. Med Image Comput Comput Assist Interv, 2021: 24-33. [52] Yu B, Zhou L, Wang L, et al. Sample-adaptive GANs: linking global and local mappings for cross-modality MR image synthesis[J]. IEEE Trans Med Imaging, 2020, 39: 2339-2350. doi: 10.1109/TMI.2020.2969630 -
点击查看大图
图(3) / 表(4)
计量
- 文章访问数: 361
- HTML全文浏览量: 86
- PDF下载量: 46
- 被引次数: 0
