سابِق|تال

Review of deep learning in optical and SAR image fusion

  • role： First author第一作者
  • Affiliation：

    Faculty of Land and Resources Engineering, Kunming University of Science and Technology, Kunming 650093, China

  • Email：cff0829@126.com
  • Introduction：成飞飞，研究方向为深度学习和多源遥感影像配准与融合。E-mail：cff0829@126.com
CHENG Feifei1,  
  • role： Corresponding author通信作者
  • Affiliation：

    Faculty of Land and Resources Engineering, Kunming University of Science and Technology, Kunming 650093, China

  • Email：zhitaofu@126.com
  • Introduction：付志涛， 研究方向为异源影像特征描述符与配准。E-mail：zhitaofu@126.com
FU Zhitao1*,  
  • Affiliation：

    Faculty of Land and Resources Engineering, Kunming University of Science and Technology, Kunming 650093, China

    Surveying and Mapping Geo-Informatics Technology Research Center on Plateau Mountains of Yunnan Higher Education, Kunming 650093, China

HUANG Liang12,  
  • Affiliation：

    Faculty of Land and Resources Engineering, Kunming University of Science and Technology, Kunming 650093, China

NIU Baosheng1,  
  • Affiliation：

    Faculty of Land and Resources Engineering, Kunming University of Science and Technology, Kunming 650093, China

CHEN Pengdi1,  
  • Affiliation：

    Research Institute of Big Data and Artificial Intelligence, Southwest Forestry University, Kunming 650024, China

WANG Leiguang3,  
  • Affiliation：

    Faculty of Land and Resources Engineering, Kunming University of Science and Technology, Kunming 650093, China

JI Xinran1

ملخص

Remote sensing image fusion, as the most challenging work in the field of image processing, has been an academic research hotspot. SAR has various features, such as all-weather service, and penetrating clouds. However, the image is difficult to interpret due to the speckle noise problem. In contrast, optical images can reflect the spectral information of ground objects, which are easy to interpret, but interference by clouds and fog can easily occur, resulting in information loss. The fusion of optical and SAR image data can realize the complementary information between different types of sensor imaging, which can facilitate the subsequent image analysis and interpretation.In this paper, the research status and future development trend of optical and SAR remote sensing fusion is reviewed. The introduction part presents the motivation of this paper by explaining the importance of image fusion. The second part outlines the classification of optical and SAR image fusion from traditional methods to deep learning methods. The third part presents the datasets in terms of optical and SAR images and explains each dataset in detail. The fourth part summarizes the difficulties and some challenging problems of optical and SAR image fusion and highlights the future trends in the field of optical and SAR image fusion.The future development trend of fusion has four main aspects, namely, datasets, time series image fusion, fusion evaluation system, and algorithm lightweight. First, having well-targeted and sufficient datasets is an important part of training excellent fusion models, and the production of data sets is the key to improving the applicability of future models. Second, time series optical images are being used as supplementary information to SAR data with the increasing amount of satellite time series data, thus improving the utilization of image fusion. Third, the evaluation of the image fusion performance is an essential topic. No unified evaluation metric is available for objectively and comprehensively evaluating fusion algorithms, and an evaluation metric system in the field of optical and SAR image fusion must be developed. Finally, the lightweight of deep learning algorithms is an important future research direction. This paper provides a reference for researchers in optical and SAR image fusion.

مفهوم

deep learning;remote sensing image;optical image;SAR;image fusion;datasets

References

  1. 1.
    Bermudez J D, Happ P N, Feitosa R Q and Oliveira D A B. 2019. Synthesis of multispectral optical images from SAR/optical multitemporal data using conditional generative adversarial networks. IEEE Geoscience and Remote Sensing Letters, 16(8): 1220-1224
  2. 2.
    Chandrakanth R, Saibaba J, Varadan G and Raj P A. 2011. Feasibility of high resolution SAR and multispectral data fusion//2011 IEEE International Geoscience and Remote Sensing Symposium. Vancouver, BC: IEEE: 356-359
  3. 3.
    Chen Y and Bruzzone L. 2019. Self-supervised sar-optical data fusion of sentinel-1/-2 images. IEEE Transactions on Geoscience and Remote Sensing, 60: 1-11
  4. 4.
    Chen K M, Zhou Z X, Xu G L, Fu K and Zhang D B (R & D Personnel). 2017. A Multi-source Remote Sensing Image Fusion Method Based on Deep Learning//Chinese Patent, Applicant (patent right): Institute of Electronics, Chinese Academy of Sciences. Beijing: CN106295714A, 2017-01-04
  5. 5.
    Chen S H, Zhang R F, Su H B, Tian J and Xia J. 2010. SAR and multispectral image fusion using generalized IHS transform based on à Trous wavelet and EMD decompositions. IEEE Sensors Journal, 10(3): 737-745
  6. 6.
    Cheng C Y, Wu X J, Xu T Y and Chen G Y. 2021. Unifusion: A lightweight unified image fusion network. IEEE Transactions on Instrumentation and Measurement,70: 1-14
  7. 7.
    Chou K C, Golden S A and Willsky A S. 1993. Multiresolution stochastic models, data fusion, and wavelet transforms. Signal Processing, 34(3): 257-282
  8. 8.
    Dupas C A. 2000. SAR and LANDSAT TM image fusion for land cover classification in the Brazilian Atlantic forest domain.International Archives of Photogrammetry and Remote Sensing, 33(1): 96-103.
  9. 9.
    El-Taweel G S and Helmy A K. 2014. Fusion of multispectral and full polarimetric SAR images in NSST domain. International Journal of Image Processing, 8(6): 497-513
  10. 10.
    Enomoto K, Sakurada K, Wang W M, Kawaguchi N, Matsuoka and Nakamura R. 2018. Image translation between SAR and optical imagery with generative adversarial nets//2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia: IEEE: 1752-1755
  11. 11.
    Feng P M, Lin Y T, Guan J, Dong Y, He G J, Xia Z H and Shi H F. 2019a. Embranchment CNN based local climate zone classification using Sar and multispectral remote sensing data//2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE: 6344-6347
  12. 12.
    Feng Q L, Yang J Y, Zhu D H, Liu J T, Guo H, Bayartungalag B and Li B G. 2019b. Integrating multitemporal sentinel-1/2 data for coastal land cover classification using a multibranch convolutional neural network: a case of the Yellow River Delta. Remote Sensing, 11(9): 1-22
  13. 13.
    Fu S L, Xu F and Jin Y Q. 2021. Reciprocal translation between SAR and optical remote sensing images with cascaded-residual adversarial networks. Science China Information Sciences, 64(2): 1-15
  14. 14.
    Fuentes Reyes M, Auer S, Merkle N, Henry C and Schmitt M. 2019. SAR-to-optical image translation based on conditional generative adversarial networks—Optimization, opportunities and limits. Remote Sensing, 11(17): 1-19
  15. 15.
    Gao J H, Zhang H and Yuan Q Q. 2019. Cloud removal with fusion of SAR and optical images by deep learning//Proceedings of the 2019 10th International Workshop on the Analysis of Multitemporal Remote Sensing Images (MultiTemp). Shanghai: IEEE: 1-3
  16. 16.
    Ghamisi P, Rasti B, Yokoya N, Wang Q M, Hofle B, Bruzzone L, Bovolo F, Chi M M, Anders K, Gloaguen R, Atkinson P M and Benediktsson J A. 2019. Multisource and multitemporal data fusion in remote sensing: a comprehensive review of the state of the art. IEEE Geoscience and Remote Sensing Magazine, 7(1): 6-39
  17. 17.
    Ghassemian H. 2016. A review of remote sensing image fusion methods. Information Fusion, 32: 75-89
  18. 18.
    Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: MIT Press: 2672-2680
  19. 19.
    Goshtasby A A and Nikolov S. 2007. Image fusion: advances in the state of the art. Information Fusion, 8(2): 114-118
  20. 20.
    Grohnfeldt C, Schmitt M and Zhu X X. 2018. A conditional generative adversarial network to fuse SAR and multispectral optical data for cloud removal from Sentinel-2 images//2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia: IEEE: 1726-1729
  21. 21.
    He W and Yokoya N. 2018. Multi-temporal Sentinel-1 and -2 data fusion for optical image simulation. ISPRS International Journal of Geo-Information, 7(10): 1-11
  22. 22.
    Hoffmann S, Brust C A, Shadaydeh M and Denzler J. 2019. Registration of high resolution Sar and optical satellite imagery using fully convolutional networks//2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE: 5152-5155
  23. 23.
    Huang M Y, Xu Y, Qian L X, Shi W L, Zhang Y Q, Bao W, Wang N, Liu X J and Xiang X S. 2021. The QXS-SAROPT dataset for deep learning in SAR-optical data fusion.arXiv: Image and Video Processing
  24. 24.
    Huang Y Q, Liao J J, Guo H D and Zhong X L. 2005. The fusion of multispectral and SAR images based wavelet transformation over urban area//Proceedings of the 2005 IEEE International Geoscience and Remote Sensing Symposium. Seoul: IEEE: 3942-3944
  25. 25.
    Ienco D, Interdonato R, Gaetano R and Minh D H T. 2019. Combining Sentinel-1 and Sentinel-2 satellite image time series for land cover mapping via a multi-source deep learning architecture. ISPRS Journal of Photogrammetry and Remote Sensing, 158: 11-22
  26. 26.
    Isola P, Zhu J Y, Zhou T and Efros A A. 2017. Image-to-image translation with conditional adversarial networks//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE: 5967-5976
  27. 27.
    Jia Y H. 1998. Fusion of Landsat TM and SAR images based on principal component analysis. Remote Sensing Technology and Application, 13(1): 46-49
  28. 28.
    Kim S, Song W J and Kim S H. 2018. Double weight-based SAR and infrared sensor fusion for automatic ground target recognition with deep learning. Remote Sensing, 10(1): 1-24
  29. 29.
    Kulkarni S C and Rege P P. 2020. Pixel level fusion techniques for SAR and optical images: a review. Information Fusion, 59: 13-29
  30. 30.
    LeCun Y and Bengio Y. 1995. Convolutional networks for images, speech, and time series//Arbib M A ed. The Handbook of Brain Theory and Neural Networks. Cambridge, MA: MIT Press: 255-258
  31. 31.
    Ley A, D’Hondt O, Valade S, Hänsch R and Hellwich O. 2018. Exploiting GAN-based SAR to optical image transcoding for improved classification via deep learning//Proceedings of the 12th European Conference on Synthetic Aperture Radar. Aachen: IEEE: 396-401
  32. 32.
    Li H and Wu X J. 2019. DenseFuse: a fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5): 2614-2623
  33. 33.
    Li H H, Guo L and Liu K. 2009. SAR and optical image fusion based on curvelet transform. Journal of Optoelectronics·Laser, 20(8): 1110-1113
  34. 34.
    Li S T, Li C Y and Kang X D. 2021. Development status and future prospects of multi-source remote sensing image fusion. National Remote Sensing Bulletin, 25(1): 148-166
  35. 35.
    Li S Y, Zhang W F and Yang S. 2017. Intelligence fusion method research of multisource high-resolution remote sensing images. Journal of Remote Sensing, 21(3): 415-424
  36. 36.
    Li X, Lei L, Sun Y L, Li M and Kuang G Y. 2020. Multimodal bilinear fusion network with second-order attention-based channel selection for land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13: 1011-1026
  37. 37.
    Liu L and Lei B. 2018. Can SAR images and optical images transfer with each other?//2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia: IEEE: 7019-7022
  38. 38.
    Liu Y, Chen X, Cheng J and Peng H. 2017a. A medical image fusion method based on convolutional neural networks//Proceedings of the 2017 20th International Conference on Information Fusion. Xi’an: IEEE: 1-7
  39. 39.
    Liu Y, Chen X, Peng H and Wang Z F. 2017b. Multi-focus image fusion with a deep convolutional neural network. Information Fusion, 36: 191-207
  40. 40.
    Liu Y, Chen X, Wang Z F, Wang Z J, Ward R K and Wang X S. 2018. Deep learning for pixel-level image fusion: recent advances and future prospects. Information Fusion, 42: 158-173
  41. 41.
    Jung H, Kim Y, Jang H, Ha N and Sohn K. 2020. Unsupervised deep image fusion with structure tensor representations. IEEE Transactions on Image Processing, 29(99): 3845-3858
  42. 42.
    Ma J Y, Ma Y and Li C. 2019. Infrared and visible image fusion methods and applications: a survey. Information Fusion, 45: 153-178
  43. 43.
    Manaf S A, Mustapha N, Sulaiman M N, Husin N A and Hamid M R A. 2016. Comparison of classification techniques on fused optical and SAR images for shoreline extraction: a case study at northeast coast of peninsular Malaysia. Journal of Computer Science, 12(8): 399-411
  44. 44.
    Meraner A, Ebel P, Zhu X X and Schmitt M. 2020. Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS Journal of Photogrammetry and Remote Sensing, 166: 333-346
  45. 45.
    Mirza M and Osindero S. 2014. Conditional generative adversarial nets. 2672-2680.
  46. 46.
    Pandit V R and Bhiwani R J. 2015. Image fusion in remote sensing applications: a review. International Journal of Computer Applications, 120(10): 22-32
  47. 47.
    Prabhakar K R, Srikar V S and Babu R V. 2017. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs//2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE: 4724-4732
  48. 48.
    Ranchin T, Aiazzi B, Alparone L, Baronti S and Wald L. 2003. Image fusion - The ARSIS concept and some successful implementation schemes. ISPRS Journal of Photogrammetry and Remote Sensing, 58(1/2): 4-18
  49. 49.
    Scarpa G, Gargiulo M, Mazza A and Gaetano R. 2018. A CNN-based fusion method for feature extraction from sentinel data. Remote Sensing, 10(2): 236
  50. 50.
    Schmitt M, Hughes L H, Körner M and Zhu X X. 2018a. Colorizing sentinel-1 sar images using a variational autoencoder conditioned on sentinel-2 imagery. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2: 1045-1051
  51. 51.
    Schmitt M, Hughes L H, Qiu C and Zhu X X. 2019. SEN12MS—A curated dataset of georeferenced multi-spectral Sentinel-1/2 imagery for deep learning and data fusion. ISPRSAnnals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, IV-2/W7: 153-160
  52. 52.
    Schmitt M, Hughes L H and Zhu X X. 2018b. The SEN1-2 dataset for deep learning in SAR-optical data fusion. ISPRSAnnals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, IV-1: 141-146
  53. 53.
    Schmitt M, Tupin F and Zhu X X. 2017. Fusion of SAR and optical remote sensing data-challenges and recent trends//Proceedings of the 2017 IEEE International Geoscience and Remote Sensing Symposium. Fort Worth, TX: IEEE: 5458-5461
  54. 54.
    Shakya A, Biswas M and Pal M. 2020. CNN-based fusion and classification of SAR and Optical data. International Journal of Remote Sensing, 41(22): 8839-8861
  55. 55.
    Shao Z F, Wu W F and Guo S J. 2020. IHS-GTF: a fusion method for optical and synthetic aperture radar data. Remote Sensing, 12(17): 2796
  56. 56.
    Song X, Wu X J, Li H. 2019. MSDNet for medical image fusion//International Conference on Image and Graphics. Beijing: Springer: 278-288.
  57. 57.
    Stathaki T. 2008. Image Fusion: Algorithms and Applications. New York, USA: Academic Press，1-30
  58. 58.
    Sun C Q, Zhang C and Xiong, N X. 2020. Infrared and visible image fusion techniques based on deep learning: a review. Electronics, 9(12), 1-24
  59. 59.
    Toriya H, Dewan A and Kitahara I. 2019. SAR2OPT: image alignment between multi-modal images using generative adversarial networks//2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE: 923-926
  60. 60.
    Wan J H, Zang J X and Liu S W. 2017. Fusion and classification of SAR and optical image with consideration of polarization characteristics. Acta Optica Sinica, 37(6): 0628001
  61. 61.
    Wang X L and Chen C X. 2016. Image fusion for synthetic aperture radar and multispectral images based on sub-band-modulated non-subsampled contourlet transform and pulse coupled neural network methods. The Imaging Science Journal, 64(2): 87-93
  62. 62.
    Wang Y Y and Zhu X X. 2018. The SARptical dataset for joint analysis of SAR and optical image in dense urban area//2018 International Geoscience and Remote Sensing Symposium. Valencia: IEEE: 6840-6843
  63. 63.
    Xia Y, Zhang H Y, Zhang L P and Fan Z Y. 2019. Cloud removal of optical remote sensing imagery with multitemporal Sar-optical data using X-Mtgan//2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama: IEEE: 3396-3399
  64. 64.
    Xu J C and Cheng X J. 2015. Comparative analysis of different fusion rules for SAR and multi-spectral image fusion based on NSCT and IHS transform//Proceedings of the 2015 International Conference on Computer and Computational Sciences. Greater Noida: IEEE: 271-274
  65. 65.
    Yang B, Zhong J Y, Li Y H and Chen Z Z. 2017. Multi-focus image fusion and super-resolution with convolutional neural network. International Journal of Wavelets, Multiresolution and Information Processing, 15(4): 1750037
  66. 66.
    Yi W, Zeng Y and Yuan Z. 2018. Fusion of GF-3 SAR and optical images based on the nonsubsampled contourlet transform. Acta Optica Sinica, 38(11): 1110002
  67. 67.
    Zhang H, Shen H F and Zhang L P. 2016. Fusion of multispectral and SAR images using sparse representation//2016 IEEE International Geoscience and Remote Sensing Symposium. Beijing: IEEE: 7200-7203
  68. 68.
    Zhang H S, Wan L M, Wang T, Lin Y Y, Lin H and Zheng Z Z. 2019. Impervious surface estimation from optical and polarimetric SAR data using small-patched deep convolutional networks: a comparative study. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7): 2374-2387
  69. 69.
    Zhang J X, Zhou J J and Lu X W. 2020a. Feature-guided SAR-to-optical image translation. IEEE Access, 8: 70925-70937
  70. 70.
    Zhang X C. 2021. Benchmarking and comparing multi-exposure image fusion algorithms. Information Fusion, 74: 111-131
  71. 71.
    Zhang X C, Ye P and Xiao G. 2020b. VIFB: a visible and infrared image fusion benchmark//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle, WA: IEEE: 468-478
  72. 72.
    Zhao W Z, Qu Y, Chen J G and Yuan Z L. 2020. Deeply synergistic optical and SAR time series for crop dynamic monitoring. Remote Sensing of Environment, 247: 111952
  73. 73.
    Zhong J Y, Yang B, Huang G Y, Zhong F and Chen Z Z. 2016. Remote sensing image fusion with convolutional neural network. Sensing and Imaging, 17(1): 10
  74. 74.
    Zhou Y, Luo J C, Feng L, Yang Y P, Chen Y H and Wu W. 2019. Long-short-term-memory-based crop classification using high-resolution optical images and multi-temporal SAR data. GIScience and Remote Sensing, 56(8): 1170-1191
  75. 75.
    Zhu J Y, Park T, Isola P and Efros A A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks//Proceedings of the 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 2242-2251
  76. 76.
    Zhu X X, Montazeri S, Ali M, Hua Y S, Wang Y Y, Mou L C, Shi Y L, Xu F and Bamler R. 2021. Deep learning meets SAR: concepts, models, pitfalls, and perspectives. IEEE Geoscience and Remote Sensing Magazine, 9(4): 143-172

قراءة النص الكامل

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website