Email Alert RSS
Volume 14 Issue 6
Nov.  2023
Turn off MathJax
Article Contents
Abstract
References
Related Articles
LI Zhuoyuan, XU Guohao, WANG Junchen, WANG Saishuo, WANG Chuantao, ZHAI Jiliang. Research Progress on Generative Adversarial Network in Cross-modal Medical Image Reconstruction[J]. Medical Journal of Peking Union Medical College Hospital, 2023, 14(6): 1162-1169. DOI: 10.12290/xhyxzz.2023-0409
Citation: LI Zhuoyuan, XU Guohao, WANG Junchen, WANG Saishuo, WANG Chuantao, ZHAI Jiliang. Research Progress on Generative Adversarial Network in Cross-modal Medical Image Reconstruction[J]. Medical Journal of Peking Union Medical College Hospital, 2023, 14(6): 1162-1169. DOI: 10.12290/xhyxzz.2023-0409
shu

Research Progress on Generative Adversarial Network in Cross-modal Medical Image Reconstruction

  • 1.

    School of Electromechanical and Vehicle Engineering, Beijing University of Civil Engineering and Architecture, Beijing 102616, China

  • 2.

    Department of Orthopaedics, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing 100730, China

Funds: 

CAMS Innovation Fund for Medical Sciences 2022-I2M-C&T-B-035

National High Level Hospital Clinical Research Funding 2022-PUMCH-A-121

More Information
    Graphical Abstract
    • Abstract
  • Single-modal medical images contain limited disease-specific information. To analyze and diagnose patients, clinicians often need to integrate multiple modal images. However, due to limited medical resources and treatment time, it may be difficult to obtain multi-modal images. Cross-modal image reconstruction can generate medical images for clinical needs, thus assisting clinicians in accurately diagnosing and treating diseases. Traditional cross-modal reconstruction techniques have been applied in some clinical scenarios, but the quality of the reconstructed images needs further improvement. Generative adversarial network (GAN) can recover high-quality and complete image data from low-quality or incomplete medical image data, maximally savings medical equipment resources and accelerating medical treatment speed. This article summarizes the applications of GAN technology in cross-modal image reconstruction across X-ray imaging, computed tomography imaging, magnetic resonance imaging, and positron emission tomography imaging, to provide reference for the development of more advanced cross-modal reconstruction techniques.
    • FullText(HTML)
    • References (52)
  • [1]
    Yu B, Wang Y, Wang L, et al. Medical image synthesis via deep learning[J]. Adv Exp Med Biol, 2020, 1213: 23-44.
    [2]
    Luo Y, Nie D, Zhan B, et al. Edge-preserving MRI image synthesis via adversarial network with iterative multi-scale fusion[J]. Neurocomputing, 2021, 452: 63-77. DOI: 10.1016/j.neucom.2021.04.060
    [3]
    Avants BB, Epstein CL, Grossman M, et al. Symmetric diffeomorphic image registration with cross-correlation: evaluat-ing automated labeling of elderly and neurodegenerative brain[J]. Med Image Anal, 2008, 12: 26-41. DOI: 10.1016/j.media.2007.06.004
    [4]
    Roy S, Carass A, Shiee N, et al. MR contrast synthesis for lesion segmentation[C]. Proc IEEE Int Symp Biomed Imaging, 2010, 2010: 932-935.
    [5]
    Ye DH, Zikic D, Glocker B, et al. Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization[C]. Med Image Comput Comput Assist Interv, 2013, 16: 606-613.
    [6]
    Huynh T, Gao Y, Kang J, et al. Estimating CT image from MRI data using structured random forest and auto-context model[J]. IEEE Trans Med Imaging, 2016, 35: 174-183. DOI: 10.1109/TMI.2015.2461533
    [7]
    Jog A, Carass A, Roy S, et al. Random forest regression for magnetic resonance image synthesis[J]. Med Image Anal, 2017, 35: 475-488. DOI: 10.1016/j.media.2016.08.009
    [8]
    Jiang J, Veeraraghavan H. Unified cross-modality feature disentangler for unsupervised multi-domain MRI abdomen organs segmentation[C]. Med Image Comput Comput Assist Interv, 2020, 12262: 347-358.
    [9]
    Xiong Z, Stiles MK, Yao Y, et al. Automatic 3D surface reconstruction of the left atrium from clinically mapped point clouds using convolutional neural networks[J]. Front Physiol, 2022, 13: 880260. DOI: 10.3389/fphys.2022.880260
    [10]
    Shen L, Zhao W, Xing L. Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning[J]. Nat Biomed Eng, 2019, 3: 880-888. DOI: 10.1038/s41551-019-0466-4
    [11]
    Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Adv Neural Inf Process Syst, 2014, 27: 2672-2680.
    [12]
    Aggarwal A, Mittal M, Battineni G. Generative adversarial network: An overview of theory and applications[J]. Int J Inf Manage, 2021, 1: 100004.
    [13]
    Wang K, Gou C, Duan Y, et al. Generative adversarial networks: introduction and outlook[J]. IEEE/CAA J Autom Sin, 2017, 4: 588-598. DOI: 10.1109/JAS.2017.7510583
    [14]
    Singh NK, Raza K. Medical image generation using generative adversarial networks: A review[J]. Health Informatics J, 2021: 77-96.
    [15]
    Zhou J. Research on Generative Adversarial Networks and Their Applications in Image Generation[C]. IEEE Int Conf Adv Electr Eng Comput Appl, 2022: 1144-1147.
    [16]
    Ravi D, Blumberg SB, Ingala S, et al. Degenerative adversarial neuroimage nets for brain scan simulations: Applica-tion in ageing and dementia[J]. Med Image Anal, 2022, 75: 102257. DOI: 10.1016/j.media.2021.102257
    [17]
    Jecklin S, Jancik C, Farshad M, et al. X23D-Intraopera-tive 3D Lumbar Spine Shape Reconstruction Based on Sparse Multi-View X-ray Data[J]. J Imaging, 2022, 8: 271. DOI: 10.3390/jimaging8100271
    [18]
    Ge R, He Y, Xia C, et al. X-CTRSNet: 3D cervical vertebra CT reconstruction and segmentation directly from 2D X-ray images[J]. Knowl Based Syst, 2022, 236: 107680. DOI: 10.1016/j.knosys.2021.107680
    [19]
    Ying X, Guo H, Ma K, et al. X2CT-GAN: reconstructing CT from biplanar X-rays with generative adversarial networks[C]. Proc IEEE Comput Soc Vision Pattern Recognit, 2019: 10619-10628.
    [20]
    Yang CJ, Lin CL, Wang CK, et al. Generative Adversarial Network (GAN) for Automatic Reconstruction of the 3D Spine Structure by Using Simulated Bi-Planar X-ray Images[J]. Diagnostics, 2022, 12: 1121. DOI: 10.3390/diagnostics12051121
    [21]
    Schock J, Lan YC, Truhn D, et al. Monoplanar CT Reconstruction with GANs[C]. IPTA, 2022: 1-6.
    [22]
    Sinha A, Sugawara Y, Hirano Y. GA-GAN: CT reconstruction from biplanar DRRs using GAN with guided attention[J]. CORR, 2019. doi: 10.48550/arXiv.1909.12525.
    [23]
    Zhan B, Xiao J, Cao C, et al. Multi-constraint generative adversarial network for dose prediction in radiotherapy[J]. Med Image Anal, 2022, 77: 102339. DOI: 10.1016/j.media.2021.102339
    [24]
    Jiang J, Hu YC, Tyagi N, et al. Tumor-aware, adversarial domain adaptation from CT to MRI for lung cancer segmentation[C]. Med Image Comput Comput Assist Interv, 2018: 777-785.
    [25]
    Jin CB, Kim H, Liu M, et al. Deep CT to MR synthesis using paired and unpaired data[J]. Sensors, 2019, 19: 2361. DOI: 10.3390/s19102361
    [26]
    Maspero M, Savenije MHF, Dinkla AM, et al. Dose evaluation of fast synthetic-CT generation using a generative adversarial network for general pelvis MR-only radiotherapy[J]. Phys Med Biol, 2018, 63: 185001. DOI: 10.1088/1361-6560/aada6d
    [27]
    王蕾, 李媛茜. 基于生成对抗网络的CT图像生成[J]. 成都信息工程大学学报, 2021, 36: 286-292. https://www.cnki.com.cn/Article/CJFDTOTAL-CDQX202103008.htm
    [28]
    Abu-Srhan A, Almallahi I, Abushariah MAM, et al. Paired-unpaired Unsupervised Attention Guided GAN with transfer learning for bidirectional brain MR-CT synthesis[J]. Comput Biol Med, 2021, 136: 104763. DOI: 10.1016/j.compbiomed.2021.104763
    [29]
    Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]. Proc IEEE Comput Soc Vision Pattern Recognit, 2017: 2223-2232.
    [30]
    吴香奕, 曹锋, 曹瑞芬, 等. 基于循环一致性生成对抗网络的盆腔伪CT生成方法[J]. 中国医学物理学杂志, 2021, 38: 21-29. https://www.cnki.com.cn/Article/CJFDTOTAL-YXWZ202101006.htm
    [31]
    Zhang Z, Yang L, Zheng Y. Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network[C]. Proc IEEE Comput Soc Vision Pattern Recognit, 2018: 9242-9251.
    [32]
    Lei Y, Harms J, Wang T, et al. MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks[J]. Med Phys, 2019, 46: 3565-3581. DOI: 10.1002/mp.13617
    [33]
    Yang H, Sun J, Carass A, et al. Unsupervised MR-to-CT synthesis using structure-constrained Cycle GAN[J]. IEEE Trans Med Imaging, 2020, 39: 4249-4261 DOI: 10.1109/TMI.2020.3015379
    [34]
    Liu Y, Lei Y, Wang T, et al. Liver synthetic CT generation based on a dense-Cycle GAN for MRI-only treatment planning[C]. Image Processing, 2020, 11313: 659-664.
    [35]
    Wang C, Yang G, Papanastasiou G, et al. DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis[J]. Inf Fusion, 2021, 67: 147-160. DOI: 10.1016/j.inffus.2020.10.015
    [36]
    Wang R, Bashyam V, Yang Z, et al. Applications of generative adversarial networks in neuroimaging and clinical neuroscience[J]. Neuroimage, 2023, 269: 119898. DOI: 10.1016/j.neuroimage.2023.119898
    [37]
    Zhang J, He X, Qing L, et al. BPGAN: Brain PET synthesis from MRI using generative adversarial network for multi-modal Alzheimer's disease diagnosis[J]. Comput Meth Prog Bio, 2022, 217: 106676. DOI: 10.1016/j.cmpb.2022.106676
    [38]
    Hu S, Lei B, Wang S, et al. Bidirectional mapping generative adversarial networks for brain MR to PET synthesis[J]. IEEE Trans Med Imaging, 2021, 4: 145-157.
    [39]
    Pan Y, Liu M, Lian C, et al. Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for Alzheimer's disease diagnosis[C]. Med Image Comput Comput Assist Interv, 2018: 455-463.
    [40]
    Gao X, Shi F, Shen D, et al. Task-induced pyramid and attention GAN for multimodal brain image imputation and classification in alzheimer's disease[J]. IEEE J Biomed Health Inf, 2021, 26: 36-43.
    [41]
    Yaakub SN, McGinnity CJ, Clough JR, et al. Pseudo-normal PET synthesis with generative adversarial networks for localising hypometabolism in epilepsies[C]. SASHIMI, 2019: 42-51.
    [42]
    Wei W, Poirion E, Bodini B, et al. Learning myelin content in multiple sclerosis from multimodal MRI through adver-sarial training[C]. Med Image Comput Comput Assist Interv, 2018: 514-522.
    [43]
    Choi H, Lee DS. Generation of structural MR images from amyloid PET: application to MR-less quantification[J]. J Nucl Med, 2018, 59: 1111-1117. DOI: 10.2967/jnumed.117.199414
    [44]
    Zhan B, Li D, Wang Y, et al. LR-CGAN: Latent representation based conditional generative adversarial network for multi-modality MRI synthesis[J]. Biomed Signal Process Control, 2021, 66: 102457. DOI: 10.1016/j.bspc.2021.102457
    [45]
    Zhan B, Zhou L, Li Z, et al. D2FE-GAN: Decoupled dual feature extraction based GAN for MRI image synthesis[J]. Knowl Based Sys, 2022, 252: 109362. DOI: 10.1016/j.knosys.2022.109362
    [46]
    Ben-Cohen A, Klang E, Raskin SP, et al. Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection[J]. Eng Appl Artif Intell, 2019, 78: 186-194. DOI: 10.1016/j.engappai.2018.11.013
    [47]
    Dong X, Wang T, Lei Y, et al. Synthetic CT generation from non-attenuation corrected PET images for whole-body PET imaging[J]. Phys Med Biol, 2019, 64: 215016. DOI: 10.1088/1361-6560/ab4eb7
    [48]
    Armanious K, Jiang C, Fischer M, et al. MedGAN: Medical image translation using GANs[J]. Comput Med Imag Grap, 2020, 79: 101684. DOI: 10.1016/j.compmedimag.2019.101684
    [49]
    Upadhyay U, Chen Y, Hepp T, et al. Uncertainty-guided progressive GANs for medical image translation[C]. Med Image Comput Comput Assist Interv, 2021: 614-624.
    [50]
    Tomar D, Lortkipanidze M, Vray G, et al. Self-attentive spatial adaptive normalization for cross-modality domain adaptation[J]. IEEE Trans Med Imaging, 2021, 40: 2926-2938. DOI: 10.1109/TMI.2021.3059265
    [51]
    Chen J, Wei J, Li R. TarGAN: target-aware generative adversarial networks for multi-modality medical image translation[C]. Med Image Comput Comput Assist Interv, 2021: 24-33.
    [52]
    Yu B, Zhou L, Wang L, et al. Sample-adaptive GANs: linking global and local mappings for cross-modality MR image synthesis[J]. IEEE Trans Med Imaging, 2020, 39: 2339-2350. DOI: 10.1109/TMI.2020.2969630
    • Related Articles

  • Related Articles

    [1]YAN Xinchun, HUO Li. Evaluation of Von Hippel-Lindau Syndrome Through Novel Small Molecular Tracer 68Ga-NY104 PET/CT Imaging[J]. Medical Journal of Peking Union Medical College Hospital, 2024, 15(4): 911-915. DOI: 10.12290/xhyxzz.2024-0216
    [2]ZHANG Xueyuan, XU Hongyan, DONG Yueming, LIU Danfeng, SUN Pengrui, YAN Rui, CUI Hongliang, LEI Hong, REN Fei. Fungal Microscopic Image Classification Based on Multi-scale Attention Mechanism[J]. Medical Journal of Peking Union Medical College Hospital, 2023, 14(1): 139-147. DOI: 10.12290/xhyxzz.2022-0169
    [3]LIU Yuan, ZHAO Lin. Update and Interpretation of 2022 National Comprehensive Cancer Network Clinical Practice Guidelines for Gastric Cancer[J]. Medical Journal of Peking Union Medical College Hospital, 2022, 13(6): 999-1004. DOI: 10.12290/xhyxzz.2022-0271
    [4]WANG Jixian, GUI Kun, CHEN Bingxian, RU Guoqing, ZHAO Di, CHEN Wanyuan, ZHANG Zhiyong. Gastric Cancer Diagnostic Model Based on Convolutional Neural Network[J]. Medical Journal of Peking Union Medical College Hospital, 2022, 13(4): 597-604. DOI: 10.12290/xhyxzz.2022-0021
    [5]Teng ZHU, Hong-zhong JIN. mRNA Expression of Methylase and Methyl-CpG Binding Protein in Peripheral Blood Mononuclear Cells from Patients with Generalized Pustular Psoriasis[J]. Medical Journal of Peking Union Medical College Hospital, 2019, 10(4): 358-363. DOI: 10.3969/j.issn.1674-9081.2019.04.009
    [6]Hao SUN, Gu-mu-yang ZHANG, Li-li XU, Zheng-yu JIN. The Value of Radiologist in the New Era's Multidisciplinary Team[J]. Medical Journal of Peking Union Medical College Hospital, 2019, 10(1): 63-67. DOI: 10.3969/j.issn.1674-9081.2019.01.009
    [7]Zheng-yu JIN, Hao SUN. Contemplation of the Clinical Diagnostic Pathway of Radiological Imaging in the New Era[J]. Medical Journal of Peking Union Medical College Hospital, 2019, 10(1): 2-5. DOI: 10.3969/j.issn.1674-9081.2019.01.001
    [8]Hang-ning ZHOU, Feng-ying XIE, Zhi-guo JIANG, Jie LIU, Hong-zhong JIN, Ru-song MENG, Yong CUI. Classification of Skin Images Based on Deep Learning[J]. Medical Journal of Peking Union Medical College Hospital, 2018, 9(1): 15-18. DOI: 10.3969/j.issn.1674-9081.2018.01.004
    [10]Zhi-lan MENG, Liang GAO, Jian-gang GU, Yu-feng LUO, Tao-ling ZHONG, Chen-yan ZHU. Application of Automatic DNA Image Cytometry in Diagnosis of Pleural Effusion[J]. Medical Journal of Peking Union Medical College Hospital, 2012, 3(1): 36-40. DOI: 10.3969/j.issn.1674-9081.2012.01.009

  • Cited by

    Periodical cited type(0)

    Other cited types(2)

Catalog

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (565) PDF downloads (108) Cited by(2)
    Related
    Links: National Health Commission of the People's Republic of China Chinese Academy of Medical Sciences(CAMS) AMLC Check CNKI Database CNKI Term Online Chinese General Practice
    Address：Room 302,Third Floor,Building 7, No. 1 Shuaifuyuan, Wangfujing, Dongcheng District, Beijing 100730, China
    Tel：010-69154261/4262
    E - mail：medj@pumch.cn mjpumch@126.com
    Copyright of website：Editorial Office of Medical Journal of Peking Union Medical College Hospital

    Supported by: Beijing Renhe Information Technology Co., Ltd.  Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return
    x Close Forever Close