Skip to main content
SpringerLink
Log in

Medical image fusion based on transfer learning techniques and coupled neural P systems

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Medical image fusion is an essential task for clinical diagnosis because it allows physicians to make more accurate diagnoses. Up to now, many medical image synthesis algorithms have been proposed, but the obtained images are often of low quality and do not preserve details from the input images. This paper proposes a new algorithm that allows the creation of composite images with advantages in two aspects: good quality and information preservation. Initially, a method for image decomposition is introduced, which separates the image into five distinct components: two highly detailed components (HDCs), two low-detailed components (LDCs), and a base component (BC). This methodology is founded on the utilization of the Gaussian filter (GF) and the rolling guidance filter (RGF). Then, a modified VGG19 (called MVGG-19) network is built by a transfer learning technique for the VGG-19 network to classify four classifications (MRI, CT, PET, and SPECT). The MVGG-19 is utilized to build fusion rules for HDCs and LDCs. Ultimately, a coupled neural P system (CNPS) is utilized to create fusion rules for BCs. In order to evaluate the effectiveness of the proposed algorithm, seven advanced algorithms and eight metrics were employed. The experiments have shown that the composite image generated by the proposed algorithm exhibits a marked improvement in quality and successfully maintains a substantial amount of information from the input image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Algorithm 2
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Data availibility statement

The datasets analyzed during the current study are available in the public resources: http://www.med.harvard.edu/AANLIB/.

References

  1. Zhang YD, Dong Z, Wang SH, Yu X, Yao X, Zhou Q, Hu H, Li M, Jiménez-Mesa C, Ramirez J, Martinez FJ, Gorriz JM (2020) Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation. Inf Fusion 64:149–187. https://doi.org/10.1016/j.inffus.2020.07.006

    Article  PubMed  PubMed Central  Google Scholar 

  2. Hermessi H, Mourali O, Zagrouba E (2021) Multimodal medical image fusion review: theoretical background and recent advances. Signal Process 183:108036. https://doi.org/10.1016/j.sigpro.2021.108036

    Article  Google Scholar 

  3. Azam MA, Khan KB, Salahuddin S, Rehman E, Khan SA, Khan MA, Kadry S, Gandomi AH (2022) A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics. Comput Biol Med 144:105253. https://doi.org/10.1016/j.compbiomed.2022.105253

    Article  PubMed  Google Scholar 

  4. Du J, Li W, Xiao B, Nawaz Q (2016) Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing 194:326–339. https://doi.org/10.1016/j.neucom.2016.02.047

    Article  Google Scholar 

  5. Wang Z, Cui Z, Zhu Y (2020) Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation. Comput Biol Med 123:103823. https://doi.org/10.1016/j.compbiomed.2020.103823

    Article  PubMed  Google Scholar 

  6. Fu J, Li W, Du J, Xiao B (2020) Multimodal medical image fusion via Laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy. Comput Biol Med 126:104048. https://doi.org/10.1016/j.compbiomed.2020.104048

    Article  PubMed  Google Scholar 

  7. Zhu Z, Zheng M, Qi G, Wang D, Xiang Y (2019) A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain. IEEE Access 7:20811–20824. https://doi.org/10.1109/access.2019.2898111

    Article  Google Scholar 

  8. Wang S, Shen Y (2020) Multi-modal image fusion based on saliency guided in NSCT domain. IET Image Proc. https://doi.org/10.1049/iet-ipr.2019.1319

    Article  Google Scholar 

  9. Wang Z, Li X, Duan H, Su Y, Zhang X, Guan X (2021) Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform. Expert Syst Appl 171:114574. https://doi.org/10.1016/j.eswa.2021.114574

    Article  Google Scholar 

  10. Liu X, Mei W, Du H (2018) Multi-modality medical image fusion based on image decomposition framework and nonsubsampled Shearlet transform. Biomed Signal Process Control 40:343–350. https://doi.org/10.1016/j.bspc.2017.10.001

    Article  Google Scholar 

  11. Ding Z, Zhou D, Nie R, Hou R, Liu Y (2020) Brain medical image fusion based on dual-branch CNNs in NSST domain. Biomed Res Int 2020:1–15. https://doi.org/10.1155/2020/6265708

    Article  Google Scholar 

  12. Nair RR, Singh T (2021) An optimal registration on shearlet domain with novel weighted energy fusion for multi-modal medical images. Optik 225:165742. https://doi.org/10.1016/j.ijleo.2020.165742

    Article  ADS  Google Scholar 

  13. Nair RR, Singh T (2021) MAMIF: multimodal adaptive medical image fusion based on b-spline registration and non-subsampled shearlet transform. Multimed Tools Appl 80(12):19079–19105. https://doi.org/10.1007/s11042-020-10439-x

    Article  Google Scholar 

  14. Wang L, Dou J, Qin P, Lin S, Gao Y, Wang R, Zhang J (2021) Multimodal medical image fusion based on nonsubsampled shearlet transform and convolutional sparse representation. Multimed Tools Appl 80(30):36401–36421. https://doi.org/10.1007/s11042-021-11379-w

    Article  Google Scholar 

  15. Gao Y, Ma S, Liu J, Liu Y, Zhang X (2021) Fusion of medical images based on salient features extraction by PSO optimized fuzzy logic in NSST domain. Biomed Signal Process Control 69:102852. https://doi.org/10.1016/j.bspc.2021.102852

    Article  Google Scholar 

  16. Zhang Y, Jin M, Huang G (2022) Medical image fusion based on improved multi-scale morphology gradient-weighted local energy and visual saliency map. Biomed Signal Process Control 74:103535. https://doi.org/10.1016/j.bspc.2022.103535

    Article  Google Scholar 

  17. Shibu DS, Priyadharsini SS (2021) Multi scale decomposition based medical image fusion using convolutional neural network and sparse representation. Biomed Signal Process Control 69:102789. https://doi.org/10.1016/j.bspc.2021.102789

    Article  Google Scholar 

  18. Yousif AS, Omar Z, Sheikh UU (2022) An improved approach for medical image fusion using sparse representation and Siamese convolutional neural network. Biomed Signal Process Control 72:103357. https://doi.org/10.1016/j.bspc.2021.103357

    Article  Google Scholar 

  19. Li Q, Wang W, Chen G, Zhao D (2021) Medical image fusion using segment graph filter and sparse representation. Comput Biol Med 131:104239. https://doi.org/10.1016/j.compbiomed.2021.104239

    Article  PubMed  Google Scholar 

  20. Hu Q, Hu S, Zhang F (2021) Multi-modality image fusion combining sparse representation with guidance filtering. Soft Comput 25(6):4393–4407. https://doi.org/10.1007/s00500-020-05448-9

    Article  Google Scholar 

  21. Barba-J L, Vargas-Quintero L, Calderón-Agudelo JA (2022) Bone SPECT/CT image fusion based on the discrete Hermite transform and sparse representation. Biomed Signal Process Control 71:103096. https://doi.org/10.1016/j.bspc.2021.103096

    Article  Google Scholar 

  22. Dinh PH (2021) A novel approach based on grasshopper optimization algorithm for medical image fusion. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2021.114576

    Article  Google Scholar 

  23. Das M, Gupta D, Radeva P, Bakde AM (2021) Optimized CT-MR neurological image fusion framework using biologically inspired spiking neural model in hybrid l1–l0 layer decomposition domain. Biomed Signal Process Control 68:102535. https://doi.org/10.1016/j.bspc.2021.102535

    Article  Google Scholar 

  24. Maqsood S, Javed U (2020) Multi-modal medical image fusion based on two-scale image decomposition and sparse representation. Biomed Signal Process Control 57:101810. https://doi.org/10.1016/j.bspc.2019.101810

    Article  Google Scholar 

  25. Li X, Zhou F, Tan H, Zhang W, Zhao C (2021) Multimodal medical image fusion based on joint bilateral filter and local gradient energy. Inf Sci 569:302–325. https://doi.org/10.1016/j.ins.2021.04.052

    Article  MathSciNet  Google Scholar 

  26. Dinh PH (2021) A novel approach based on three-scale image decomposition and marine predators algorithm for multi-modal medical image fusion. Biomed Signal Process Control 67:102536. https://doi.org/10.1016/j.bspc.2021.102536

    Article  Google Scholar 

  27. Dinh PH (2021) An improved medical image synthesis approach based on marine predators algorithm and maximum Gabor energy. Neural Comput Appl 34(6):4367–4385. https://doi.org/10.1007/s00521-021-06577-4

    Article  Google Scholar 

  28. Dinh PH (2021) Multi-modal medical image fusion based on equilibrium optimizer algorithm and local energy functions. Appl Intell 51(11):8416–8431. https://doi.org/10.1007/s10489-021-02282-w

    Article  Google Scholar 

  29. Dinh PH (2021) Combining Gabor energy with equilibrium optimizer algorithm for multi-modality medical image fusion. Biomed Signal Process Control 68:102696. https://doi.org/10.1016/j.bspc.2021.102696

    Article  Google Scholar 

  30. Feng X, Fang C, Qiu G (2023) Multimodal medical image fusion based on visual saliency map and multichannel dynamic threshold neural p systems in sub-window variance filter domain. Biomed Signal Process Control 84:104794. https://doi.org/10.1016/j.bspc.2023.104794

    Article  Google Scholar 

  31. Yang Y, Cao S, Wan W, Huang S (2023) Multi-modal medical image super-resolution fusion based on detail enhancement and weighted local energy deviation. Biomed Signal Process Control 80:104387. https://doi.org/10.1016/j.bspc.2022.104387

    Article  Google Scholar 

  32. Li X, Wan W, Zhou F, Cheng X, Jie Y, Tan H (2023) Medical image fusion based on sparse representation and neighbor energy activity. Biomed Signal Process Control 80:104353. https://doi.org/10.1016/j.bspc.2022.104353

    Article  Google Scholar 

  33. Feng Y, Wu J, Hu X, Zhang W, Wang G, Zhou X, Zhang X (2023) Medical image fusion using bilateral texture filtering. Biomed Signal Process Control 85:105004. https://doi.org/10.1016/j.bspc.2023.105004

    Article  Google Scholar 

  34. Zhang L, Li H, Zhu R, Du P (2022) An infrared and visible image fusion algorithm based on ResNet-152. Multimed Tools Appl 81(7):9277–9287. https://doi.org/10.1007/s11042-021-11549-w

    Article  Google Scholar 

  35. Li H, Xj Wu, Durrani TS (2019) Infrared and visible image fusion with ResNet and zero-phase component analysis. Infrared Phys Technol 102:103039. https://doi.org/10.1016/j.infrared.2019.103039

    Article  Google Scholar 

  36. Ding Z, Li H, Guo Y, Zhou D, Liu Y, Xie S (2023) M4fnet: multimodal medical image fusion network via multi-receptive-field and multi-scale feature integration. Comput Biol Med 159:106923. https://doi.org/10.1016/j.compbiomed.2023.106923

    Article  PubMed  Google Scholar 

  37. Li W, Zhang Y, Wang G, Huang Y, Li R (2023) DFENet: a dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion. Biomed Signal Process Control 80:104402. https://doi.org/10.1016/j.bspc.2022.104402

    Article  Google Scholar 

  38. Li W, Li R, Fu J, Peng X (2022) MSENet: a multi-scale enhanced network based on unique features guidance for medical image fusion. Biomed Signal Process Control 74:103534. https://doi.org/10.1016/j.bspc.2022.103534

    Article  Google Scholar 

  39. Shehanaz S, Daniel E, Guntur SR, Satrasupalli S (2021) Optimum weighted multimodal medical image fusion using particle swarm optimization. Optik 231:166413. https://doi.org/10.1016/j.ijleo.2021.166413

    Article  ADS  Google Scholar 

  40. Tannaz A, Mousa S, Sabalan D, Masoud P (2019) Fusion of multimodal medical images using nonsubsampled shearlet transform and particle swarm optimization. Multidimens Syst Signal Process 31(1):269–287. https://doi.org/10.1007/s11045-019-00662-7

    Article  Google Scholar 

  41. Dinh PH (2023) Combining spectral total variation with dynamic threshold neural p systems for medical image fusion. Biomed Signal Process Control 80:104343. https://doi.org/10.1016/j.bspc.2022.104343

    Article  Google Scholar 

  42. Xu L, Si Y, Jiang S, Sun Y, Ebrahimian H (2020) Medical image fusion using a modified shark smell optimization algorithm and hybrid wavelet-homomorphic filter. Biomed Signal Process Control 59:101885. https://doi.org/10.1016/j.bspc.2020.101885

    Article  Google Scholar 

  43. Daniel E, Anitha J, Kamaleshwaran K, Rani I (2017) Optimum spectrum mask based medical image fusion using gray wolf optimization. Biomed Signal Process Control 34:36–43. https://doi.org/10.1016/j.bspc.2017.01.003

    Article  Google Scholar 

  44. Do OC, Luong CM, Dinh PH, Tran GS (2024) An efficient approach to medical image fusion based on optimization and transfer learning with VGG19. Biomed Signal Process Control 87:105370. https://doi.org/10.1016/j.bspc.2023.105370

    Article  Google Scholar 

  45. Dinh PH (2023) A novel approach using the local energy function and its variations for medical image fusion. Imaging Sci J 71(7):660–676. https://doi.org/10.1080/13682199.2023.2190947

    Article  Google Scholar 

  46. James AP, Dasarathy BV (2014) Medical image fusion: a survey of the state of the art. Inf Fusion 19:4–19. https://doi.org/10.1016/j.inffus.2013.12.002

    Article  Google Scholar 

  47. Liu Y, Wang L, Cheng J, Li C, Chen X (2020) Multi-focus image fusion: a survey of the state of the art. Inf Fusion 64:71–91. https://doi.org/10.1016/j.inffus.2020.06.013

    Article  Google Scholar 

  48. Chao Z, Duan X, Jia S, Guo X, Liu H, Jia F (2022) Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network. Appl Soft Comput 118:108542. https://doi.org/10.1016/j.asoc.2022.108542

    Article  Google Scholar 

  49. Tan W, Tiwari P, Pandey HM, Moreira C, Jaiswal AK (2020) Multimodal medical image fusion algorithm in the era of big data. Neural Comput Appl. https://doi.org/10.1007/s00521-020-05173-2

    Article  Google Scholar 

  50. Tan W, Thitøn W, Xiang P, Zhou H (2021) Multi-modal brain image fusion based on multi-level edge-preserving filtering. Biomed Signal Process Control 64:102280. https://doi.org/10.1016/j.bspc.2020.102280

    Article  Google Scholar 

  51. Zhang Q, Shen X, Xu L, Jia J (2014) Rolling guidance filter. In: Computer Vision—ECCV 2014. Springer International Publishing, pp 815–830. https://doi.org/10.1007/978-3-319-10578-9_53

  52. Yang J, Guo Z, Zhang D, Wu B, Du S (2022) An anisotropic diffusion system with nonlinear time-delay structure tensor for image enhancement and segmentation. Comput Math Appl 107:29–44. https://doi.org/10.1016/j.camwa.2021.12.005

    Article  MathSciNet  Google Scholar 

  53. Liu K, Xu W, Wu H, Yahya AA (2022) Weighted hybrid order total variation model using structure tensor for image denoising. Multimed Tools Appl 82(1):927–943. https://doi.org/10.1007/s11042-022-12393-2

    Article  Google Scholar 

  54. Zhou Z, Li S, Wang B (2014) Multi-scale weighted gradient-based fusion for multi-focus images. Inf Fusion 20:60–72. https://doi.org/10.1016/j.inffus.2013.11.005

    Article  Google Scholar 

  55. Peng H, Wang J (2019) Coupled neural p systems. IEEE Trans Neural Netw Learn Syst 30(6):1672–1682. https://doi.org/10.1109/tnnls.2018.2872999

    Article  MathSciNet  PubMed  Google Scholar 

  56. Li B, Peng H, Luo X, Wang J, Song X, Pérez-Jiménez MJ, Riscos-Núñez A (2020) Medical image fusion method based on coupled neural p systems in nonsubsampled shearlet transform domain. Int J Neural Syst 31(01):2050050. https://doi.org/10.1142/s0129065720500501

    Article  PubMed  Google Scholar 

  57. Wang G, Li W, Gao X, Xiao B, Du J (2022) Multimodal medical image fusion based on multichannel coupled neural p systems and max-cloud models in spectral total variation domain. Neurocomputing. https://doi.org/10.1016/j.neucom.2022.01.059

    Article  Google Scholar 

  58. Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Trans Image Process 22(7):2864–2875. https://doi.org/10.1109/tip.2013.2244222

    Article  ADS  PubMed  Google Scholar 

  59. Liu Y, Chen X, Ward RK, Wang ZJ (2016) Image fusion with convolutional sparse representation. IEEE Signal Process Lett 23(12):1882–1886. https://doi.org/10.1109/lsp.2016.2618776

    Article  ADS  Google Scholar 

  60. Guihong Q, Zhang D, Yan P (2002) Information measure for performance of image fusion. Electron Lett 38:313–315

    Article  ADS  Google Scholar 

  61. Wang Q, Shen Y, Jin J (2008) Performance evaluation of image fusion techniques. In: Image fusion. Elsevier, pp 469–492. https://doi.org/10.1016/b978-0-12-372529-5.00017-2

  62. Yeganeh H, Wang Z (2013) Objective quality assessment of tone-mapped images. IEEE Trans Image Process 22(2):657–667. https://doi.org/10.1109/tip.2012.2221725

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  63. Xydeas C, Petrovic V (2000) Objective image fusion performance measure. Electron Lett 36:308. https://doi.org/10.1049/el:20000267

    Article  ADS  Google Scholar 

  64. Piella G, Heijmans H (2003) A new quality metric for image fusion. In: Proceedings 2003 international conference on image processing (Cat. No. 03CH37429). IEEE. https://doi.org/10.1109/icip.2003.1247209

  65. Li B, Peng H, Wang J (2021) A novel fusion method based on dynamic threshold neural p systems and nonsubsampled contourlet transform for multi-modality medical images. Signal Process 178:107793. https://doi.org/10.1016/j.sigpro.2020.107793

    Article  Google Scholar 

  66. Zhu R, Li X, Huang S, Zhang X (2021) Multimodal medical image fusion using adaptive co-occurrence filter-based decomposition optimization model. Bioinformatics 38(3):818–826. https://doi.org/10.1093/bioinformatics/btab721

    Article  CAS  Google Scholar 

  67. Sufyan A, Imran M, Shah SA, Shahwani H, Wadood AA (2021) A novel multimodality anatomical image fusion method based on contrast and structure extraction. Int J Imaging Syst Technol 32(1):324–342. https://doi.org/10.1002/ima.22649

    Article  Google Scholar 

  68. Zhang Y, Xiang W, Zhang S, Shen J, Wei R, Bai X, Zhang L, Zhang Q (2022) Local extreme map guided multi-modal brain image fusion. Front Neurosci. https://doi.org/10.3389/fnins.2022.1055451

    Article  PubMed  PubMed Central  Google Scholar 

  69. Veshki FG, Ouzir N, Vorobyov SA, Ollila E (2022) Multimodal image fusion via coupled feature learning. Signal Process 200:108637. https://doi.org/10.1016/j.sigpro.2022.108637

    Article  Google Scholar 

  70. Xie Q, Hu J, Wang X, Zhang D, Qin H (2022) Novel and fast EMD-based image fusion via morphological filter. Vis Comput. https://doi.org/10.1007/s00371-022-02588-x

    Article  Google Scholar 

  71. Dinh PH, Giang NL (2022) A new medical image enhancement algorithm using adaptive parameters. Int J Imaging Syst Technol 32(6):2198–2218. https://doi.org/10.1002/ima.22778

    Article  Google Scholar 

  72. Dinh PH (2023) A novel approach based on marine predators algorithm for medical image enhancement. Sens Imaging. https://doi.org/10.1007/s11220-023-00411-y

    Article  Google Scholar 

  73. Dinh PH (2022) A novel approach using structure tensor for medical image fusion. Multidimens Syst Signal Process 33(3):1001–1021. https://doi.org/10.1007/s11045-022-00829-9

    Article  Google Scholar 

Download references

Acknowledgements

This research is funded by Thuyloi University Foundation for Science and Technology under grant number TLU.STF.23-05.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Engineering, Thuyloi University, 175 Tay Son, Dong Da, Hanoi, Vietnam

    Phu-Hung Dinh

  2. Institute of Information Technology, Vietnam Academy of Science and Technology, Hanoi, Vietnam

    Nguyen Long Giang

Authors
  1. Phu-Hung Dinh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nguyen Long Giang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Phu-Hung Dinh.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dinh, PH., Giang, N.L. Medical image fusion based on transfer learning techniques and coupled neural P systems. Neural Comput & Applic 36, 4325–4347 (2024). https://doi.org/10.1007/s00521-023-09294-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09294-2

Keywords

Search

Navigation