Anterior|Siguiente

Progreso de la investigación sobre redes generativas antagónicas en la fusión de imágenes de teledetección

  • role： First author第一作者
  • Affiliation：

    School of Software, Yunnan University, Kunming 650000, China

    Engineering Research Center of Cyberspace, Yunnan University, Kunming 650000, China

  • Email：zhenghuangqimei@stu.ynu.edu.cn
  • Introduction：郑黄齐眉，研究方向为遥感图像融合。E-mail： zhenghuangqimei@stu.ynu.edu.cn
ZHENG Huangqimei,  
  • Affiliation：

    School of Software, Yunnan University, Kunming 650000, China

    Engineering Research Center of Cyberspace, Yunnan University, Kunming 650000, China

PAN Chengyi,  
  • role： Corresponding author通信作者
  • Affiliation：

    School of Software, Yunnan University, Kunming 650000, China

    Engineering Research Center of Cyberspace, Yunnan University, Kunming 650000, China

  • Email：xinjin@ynu.edu.cn
  • Introduction：金鑫，研究方向为人工智能、图像处理。E-mail： xinjin@ynu.edu.cn
JIN Xin*,  
  • Affiliation：

    School of Software, Yunnan University, Kunming 650000, China

    Engineering Research Center of Cyberspace, Yunnan University, Kunming 650000, China

WANG Qianqian,  
  • Affiliation：

    School of Software, Yunnan University, Kunming 650000, China

    Engineering Research Center of Cyberspace, Yunnan University, Kunming 650000, China

MIAO Shengfa,  
  • Affiliation：

    School of Software, Yunnan University, Kunming 650000, China

    Engineering Research Center of Cyberspace, Yunnan University, Kunming 650000, China

JIANG Qian

resumen

La fusión de imágenes de teledetección, como una rama importante de la fusión de datos, tiene un significado importante para el estudio de objetos terrestres. La selección eficaz de un método de fusión adecuado es especialmente importante para mejorar la precisión de las imágenes. Con el desarrollo de la tecnología de teledetección, los métodos tradicionales de fusión de imágenes ya no pueden satisfacer los requisitos de precisión de las imágenes, y continuamente se proponen nuevos métodos de fusión. En el campo del procesamiento de imágenes de teledetección, las redes generativas antagónicas (GAN), debido a su poderosa capacidad generativa y su capacidad para modelar distribuciones complejas, se han convertido rápidamente en una tecnología importante para mejorar la calidad de la fusión de imágenes. Este artículo revisa los métodos tradicionales de fusión de imágenes de teledetección y sus limitaciones, y analiza las ventajas del aprendizaje profundo, especialmente de las GAN, en este campo. A través de una introducción detallada de varias arquitecturas GAN y funciones de pérdida, revela su gran potencial para mejorar la resolución espacial y espectral de las imágenes fusionadas. Además, se detalla la aplicación de diversos métodos de fusión de imágenes de teledetección basados en GAN en los últimos años, y se discute su uso en tareas de afilado pancromático y afilado hiperespectral. Este artículo resume el desarrollo de métodos de fusión de imágenes de teledetección basados en GAN y analiza los problemas actuales y las direcciones futuras desde tres aspectos.

palabra clave

imágenes de teledetección; fusión de datos multisource; aprendizaje profundo; redes generativas antagónicas; afilado pancromático de imágenes de teledetección; imágenes multiespectrales; imágenes hiperespectrales

References

  1. 1.
    Achalakul T and Taylor S. 2003. A distributed spectral-screening PCT algorithm. Journal of Parallel and Distributed Computing, 63(3): 373-384
  2. 2.
    Bai L Y, Xu C M and Wang C. 2015. A review of fusion methods of multi-spectral image. Optik, 126(24): 4804-4807
  3. 3.
    Balakrishnan S, Cacciola M, Udpa L, Rao B P, Jayakumar T and Raj B. 2012. Development of image fusion methodology using discrete wavelet transform for eddy current images. NDT and E International, 51: 51-57
  4. 4.
    Bello J L G, Seo S and Kim M. 2020. Pan-sharpening with color-aware perceptual loss and guided re-colorization//Proceedings of the 2020 IEEE International Conference on Image Processing(ICIP). Abu Dhabi: IEEE: 908-912
  5. 5.
    Benzenati T, Kessentini Y and Kallel A. 2022. Pansharpening approach via two-stream detail injection based on relativistic generative adversarial networks. Expert Systems with Applications, 188: 115996
  6. 6.
    Carper W J, Lillesand T M and Kiefer R W. 1990. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data. Photogrammetric Engineering and Remote Sensing, 56(4): 459-467
  7. 7.
    Cliche G, Bonn F and Teillet P. 1985. Integration of the SPOT panchromatic channel into its multispectral mode for image sharpness enhancement. Photogrammetric Engineering and Remote Sensing, 51(3): 311-316
  8. 8.
    Diao W X, Jin M Y, Zhang K and Xiao L. 2023. Unsupervised single-generator CycleGAN-based pansharpening with spatial-spectral degradation modeling. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 16: 10246-10263
  9. 9.
    Dong C, Loy C C, He K M and Tang X O. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307
  10. 10.
    Dong W Q, Hou S X, Xiao S, Qu J H, Du Q and Li Y S. 2022. Generative dual-adversarial network with spectral fidelity and spatial enhancement for hyperspectral pansharpening. IEEE Transactions on Neural Networks and Learning Systems, 33(12): 7303-7317
  11. 11.
    Garzelli A and Nencini F. 2007. Panchromatic sharpening of remote sensing images using a multiscale Kalman filter. Pattern Recognition, 40(12): 3568-3577
  12. 12.
    Ghosh A, Kulharia V, Namboodiri V P, Torr P H S and Dokania P K. 2018. Multi-agent diverse generative adversarial networks//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE: 8513-8521
  13. 13.
    Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y. 2020. Generative adversarial networks. Communications of the ACM, 63(11): 139-144
  14. 14.
    Gui J, Sun Z N, Wen Y G, Tao D C and Ye J P. 2023. A review on generative adversarial networks: algorithms, theory, and applications. IEEE Transactions on Knowledge and Data Engineering, 35(4): 3313-3332
  15. 15.
    He L J, Ren Z H, Zhang W Y, Li F and Mei S H. 2024. Unsupervised pansharpening based on double-cycle consistency. IEEE Transactions on Geoscience and Remote Sensing, 62: 5613015
  16. 16.
    Hu J W, Du C G and Fan S S. 2020. Two-stage pansharpening based on multi-level detail injection network. IEEE Access, 8: 156442-156455
  17. 17.
    Hu J W, Hu P, Kang X D, Zhang H and Fan S S. 2021. Pan-sharpening via multiscale dynamic convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 59(3): 2231-2244
  18. 18.
    Hu J W, Wang Z P and Hu P. 2023. A review of pansharpening methods based on deep learning. Remote Sensing for Natural Resources, 35(1): 1-14
  19. 19.
    Jin X, Feng Y T, Jiang Q, Miao S F, Chu X, Zheng H Q M and Wang Q Q. 2024. UPGAN: an unsupervised generative adversarial network based on U-shaped structure for pansharpening. ISPRS International Journal of Geo-Information, 13(7): 222
  20. 20.
    Jin X, Huang S S, Jiang Q, Lee S J, Wu L W and Yao S W. 2021. Semisupervised remote sensing image fusion using multiscale conditional generative adversarial network with siamese structure. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 7066-7084
  21. 21.
    Johnson J, Alahi A and Fei-Fei L. 2016. Perceptual losses for real-time style transfer and super-resolution//Proceedings of the 14th European Conference on Computer Vision–ECCV 2016. Amsterdam: Springer: 694-711
  22. 22.
    Khaleghi B, Khamis A, Karray F O and Razavi S N. 2013. Multisensor data fusion: a review of the state-of-the-art. Information Fusion, 14(1): 28-44
  23. 23.
    Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z H and Shi W Z. 2017. Photo-realistic single image super-resolution using a generative adversarial network//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE: 105-114
  24. 24.
    Liu M Y and Tuzel O. 2016. Coupled generative adversarial networks//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona: Curran Associates Inc.: 469-477
  25. 25.
    Liu P, Li J, Wang L Z and He G J. 2022. Remote sensing data fusion with generative adversarial networks: state-of-the-art methods and future research directions. IEEE Geoscience and Remote Sensing Magazine, 10(2): 295-328
  26. 26.
    Liu Q J, Zhou H Y, Xu Q Z, Liu X Y and Wang Y H. 2021. PSGAN: a generative adversarial network for remote sensing image pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, 59(12): 10227-10242
  27. 27.
    Liu X B, Liu X, Dai H R, Kang X D, Plaza A and Zu W J. 2023. Mun-GAN: a multiscale unsupervised network for remote sensing image pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 61: 5404018
  28. 28.
    Liu X Y, Liu Q J and Wang Y H. 2020. Remote sensing image fusion based on two-stream fusion network. Information Fusion, 55: 1-15
  29. 29.
    Luo X, Tong X H and Hu Z W. 2021. Improving satellite image fusion via generative adversarial training. IEEE Transactions on Geoscience and Remote Sensing, 59(8): 6969-6982
  30. 30.
    Ma J Y, Xu H, Jiang J J, Mei X G and Zhang X P. 2020b. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Transactions on Image Processing, 29: 4980-4995
  31. 31.
    Ma J Y, Yu W, Chen C, Liang P W, Guo X J and Jiang J J. 2020a. Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion, 62: 110-120
  32. 32.
    Masi G, Cozzolino D, Verdoliva L and Scarpa G. 2016. Pansharpening by convolutional neural networks. Remote Sensing, 8(7): 594
  33. 33.
    Mehra I and Nishchal N K. 2014. Image fusion using wavelet transform and its application to asymmetric cryptosystem and hiding. Optics Express, 22(5): 5474-5482
  34. 34.
    Mirza M and Osindero S. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784
  35. 35.
    Odena A, Olah C and Shlens J. 2017. Conditional image synthesis with auxiliary classifier GANs//Proceedings of the 34th International Conference on Machine Learning. Sydney: JMLR.org: 2642-2651
  36. 36.
    Ozcelik F, Alganci U, Sertel E and Unal G. 2021. Rethinking CNN-based pansharpening: guided colorization of panchromatic images via GANs. IEEE Transactions on Geoscience and Remote Sensing, 59(4): 3486-3501
  37. 37.
    Ratliff L J, Burden S A and Sastry S S. 2013. Characterization and computation of local Nash equilibria in continuous games//Proceedings of the 2013 51st Annual Allerton Conference on Communication, Control, and Computing(Allerton). Monticello: IEEE: 917-924
  38. 38.
    Romero L S, Marcello J and Vilaplana V. 2020. Super-resolution of sentinel-2 imagery using generative adversarial networks. Remote Sensing, 12(15): 2424
  39. 39.
    Shang Y L, Liu J J, Zhang J Y and Wu Z B. 2024. MFT-GAN: a multiscale feature-guided transformer network for unsupervised hyperspectral pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 62: 5518516
  40. 40.
    Shao Z M, Lu Z X, Ran M S, Fang L Y, Zhou J L and Zhang Y. 2020. Residual encoder–decoder conditional generative adversarial network for pansharpening. IEEE Geoscience and Remote Sensing Letters, 17(9): 1573-1577
  41. 41.
    Su L J, Sui Y X and Yuan Y. 2023. An unmixing-based multi-attention GAN for unsupervised hyperspectral and multispectral image fusion. Remote Sensing, 15(4): 936
  42. 42.
    Sun S L and Deng Z L. 2004. Multi-sensor optimal information fusion Kalman filter. Automatica, 40(6): 1017-1023
  43. 43.
    Tu T M, Cheng W C, Chang C P, Huang P S and Chang J C. 2007. Best tradeoff for high-resolution image fusion to preserve spatial details and minimize color distortion. IEEE Geoscience and Remote Sensing Letters, 4(2): 302-306
  44. 44.
    Wald L, Ranchin T and Mangolini M. 1997. Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6): 691-699
  45. 45.
    Wang X T, Yu K, Wu S X, Gu J J, Liu Y H, Dong C, Qiao Y and Loy C C. 2018. ESRGAN: enhanced super-resolution generative adversarial networks//Proceedings of the European Conference on Computer Vision (ECCV) Workshops. Munich: Springer: 63-79
  46. 46.
    Wang Y J, Xie Y Y, Wu Y Y, Liang K and Qiao J L. 2022. An unsupervised multi-scale generative adversarial network for remote sensing image pan-sharpening//Proceedings of the 28th International Conference on Multimedia Modeling. Phu Quoc: Springer: 356-368
  47. 47.
    Wang Z W, She Q and Ward T E. 2021. Generative adversarial networks in computer vision: a survey and taxonomy. ACM Computing Surveys(CSUR), 54(2): 37
  48. 48.
    Wei Y C, Yuan Q Q, Shen H F and Zhang L P. 2017. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geoscience and Remote Sensing Letters, 14(10): 1795-1799
  49. 49.
    Xiao J J, Li J, Yuan Q Q, Jiang M H and Zhang L P. 2021. Physics-based GAN with iterative refinement unit for hyperspectral and multispectral image fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 6827-6841
  50. 50.
    Xie W Y, Cui Y H, Li Y S, Lei J, Du Q and Li J J. 2021. HPGAN: hyperspectral pansharpening using 3-D generative adversarial networks. IEEE Transactions on Geoscience and Remote Sensing, 59(1): 463-477
  51. 51.
    Yang J F, Fu X Y, Hu Y W, Huang Y, Ding X H and Paisley J. 2017. PanNet: a deep network architecture for pan-sharpening//Proceedings of the 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 1753-1761
  52. 52.
    Yang Z G, Chen Y P, Le Z L and Ma Y. 2021. GANFuse: a novel multi-exposure image fusion method based on generative adversarial networks. Neural Computing and Applications, 33(11): 6133-6145
  53. 53.
    Yuan Q Q, Wei Y C, Meng X C, Shen H F and Zhang L P. 2018. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3): 978-989
  54. 54.
    Zhang H and Ma J Y. 2023. STP-SOM: scale-transfer learning for pansharpening via estimating spectral observation model. International Journal of Computer Vision, 131(12): 3226-3251
  55. 55.
    Zhang H, Xu H, Tian X, Jiang J J and Ma J Y. 2021. Image fusion meets deep learning: a survey and perspective. Information Fusion, 76: 323-336
  56. 56.
    Zhang H, Xu T, Li H S, Zhang S T, Wang X G, Huang X L and Metaxas D. 2017. StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks//Proceedings of the 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 5908-5916
  57. 57.
    Zhong X W, Qian Y R, Liu H, Chen L, Wan Y L, Gao L, Qian J and Liu J. 2021. Attention_FPNet: two-branch remote sensing image pansharpening network based on attention feature fusion. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 11879-11891
  58. 58.
    Zhou H Y, Liu Q J and Wang Y H. 2021. PGMAN: an unsupervised generative multiadversarial network for pansharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 6316-6327
  59. 59.
    Zhou H Y, Liu Q J, Weng D W and Wang Y H. 2022. Unsupervised cycle-consistent generative adversarial networks for pan sharpening. IEEE Transactions on Geoscience and Remote Sensing, 60: 5408814
  60. 60.
    Zhu J Y, Park T, Isola P and Efros A A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks//Proceedings of the 2017 IEEE International Conference on Computer Vision. Venice: IEEE: 2242-2251

Leer el texto completo

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website