سابِق|تال

Pan-sharpening by deep recursive residual network

  • role： First author第一作者
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

    School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100049, China

  • Email：wangfen@radi.ac.cn
  • Introduction：王芬，1994年生，女，硕士研究生，研究方向为遥感图像处理、深度学习。E-mail：wangfen@radi.ac.cn
WANG Fen12,  
  • role： Corresponding author通信作者
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

  • Email：guoqing@aircas.ac.cn
  • Introduction：郭擎，1980年生，女，研究员，研究方向为遥感图像处理、遥感信息提取与应用、深度学习。E-mail：guoqing@aircas.ac.cn
GUO Qing1*,  
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

GE Xiaoqing1

ملخص

Pan-sharpening is a task in the field of remote sensing data fusion, in which multispectral (MS) images with rich spectral information but low spatial resolution and panchromatic (PAN) images with rich spatial details but only grey information are fused to yield images with high spatial and spectral resolution. Traditional Component Substitution (CS) methods replace a particular component of the MS image transformation with a PAN image, and then inversely transforms it to obtain the final fused image. The traditional MultiResolution Analysis (MRA) methods first extract spatial structures from the PAN image by using MRA transforms, and then the extracted spatial structure information is injected into the up-sampled MS images to obtain the fused image. The whole fusing process of the CS and MRA methods can be described as linear functions. However, the performance of such linear models are limited by their linearity, which often has spectral distortion. In recent years, many advanced nonlinear deep learning models have been proposed. However, those existing deep learning fusion models are relatively simple and pose difficultly in learn in-depth features. To overcome the shortcomings of the current models, we propose a deep recursive residual network that is specifically designed for the pan-sharpening task.Considering that the low-resolution input image and the high-resolution output image have high similarity, learning the relationship between input and output is highly redundant and difficult. If the sparse residual features between input and output are learned directly, then the network convergence can be significantly improved. Thus, the residual learning introduces the network structure, in which the introduced residuals include global residuals and local residuals. Such a structure is conducive to learning and not prone to overfitting. Moreover, the residual network can solve the problem of deep network gradient disappearance and gradient explosion well. Recursive network improves accuracy by increasing the number of network layers without increasing weight parameters. Specifically, as we use the residual network globally, recursive learning is introduced into residual learning by constructing recursive blocks structure, whereas multiple local residual units are stacked together in the recursive block. Through such an end-to-end network design, a better image fusion effect is obtained.Given that no ideal fusion result has been used as a label, we made a data set according to Wald’s protocol using the original MS as the ideal fused image, downsampling and then upsampling the MS as the MS of the network input, and the downsampled PAN as the PAN of the network input. To comprehensively analyze our experimental results, we performed a large number of simulation experiments and real experiments on the 4-band GaoFen-1 data and 8-band WorldView-2 data with abundant feature types. We then generalized them to 4-band GeoEye data and 8-band WorldView-3 data. Experimental results are compared with traditional methods and the existing deep learning methods. The subjective visual analysis and objective evaluation indicators show that the proposed method reduces the spectral distortion phenomenon of traditional methods and preserves the spectrum of an image better than the existing deep learning method does.The deep network designed in this paper has learned more in-depth and more luxurious image features and has achieved better fusion effects than existing methods. It uses a residual network to solve profound network gradient disappearance, gradient explosion, and degradation problems. In addition, the weight parameters are reduced by the design of the recurrent recursive block, and the network speed is improved. The generalization experiment shows that our network has a good generalization ability.

مفهوم

remote sensing image fusion;space spectrum fusion;deep learning;convolutional neural network;residual network;recursive network

References

  1. 1.
    Azarang A and Ghassemian H. 2017. A new pansharpening method using multi resolution analysis framework and deep neural networks//Proceedings of the 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA). Shahrekord, Iran: IEEE: 1-6
  2. 2.
    Chavez P S and Kwarteng A Y. 1988. Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis. Photogrammetric Engineering and Remote Sensing, 55(3): 339-348
  3. 3.
    Choi J, Yu K and Kim Y. 2011. A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing, 49(1): 295-309
  4. 4.
    Easley G, Labate D and Lim W Q. 2008. Sparse directional image representations using the discrete shearlet transform. Applied and Computational Harmonic Analysis, 25(1): 25-46
  5. 5.
    Ehlers M. 1991. Multisensor image fusion techniques in remote sensing. ISPRS Journal of Photogrammetry and Remote Sensing, 46(1): 19-30
  6. 6.
    Koutsias N, Karteris M and Chuvieco E. 2000. The use of intensity-hue-saturation transformation of landsat-5 thematic mapper data for burned land mapping. Photogrammetric Engineering and Remote Sensing, 66(7): 829-839
  7. 7.
    Laben C A and Brower B V. 2000. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S., No. 6011875
  8. 8.
    Le Pennec E and Mallat S. 2000. Image compression with geometrical wavelets//Proceedings 2000 International Conference on Image Processing. Vancouver, Canada: IEEE: 661-664
  9. 9.
    Liang M and Hu X L. 2015. Recurrent convolutional neural network for object recognition//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE: 3367-3375
  10. 10.
    Mallat S G. 1989. Multifrequency channel decompositions of images and wavelet models. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(12): 2091-2110
  11. 11.
    Masi G, Cozzolino D, Verdoliva L and Scarpa G. 2016. Pansharpening by convolutional neural networks. Remote Sensing, 8(7): 594
  12. 12.
    Micheal A A and Vani K. 2017. Multi-sensor image fusion of the lunar image data using DT-CWT and curvelet transform//Proceedings of the 2017 4th International Conference on Electronics and Communication Systems (ICECS). Coimbatore, India: IEEE: 49-53
  13. 13.
    Piella G. 2009. Image fusion for enhanced visualization: A variational approach. International Journal of Computer Vision, 83(1): 1-11
  14. 14.
    Rao Y Z, He L and Zhu J W. 2017. A residual convolutional neural network for pan-shaprening//Proceedings of 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP). Shanghai, China: IEEE: 1-4
  15. 15.
    Simonyan K and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv: 1409.1556.
  16. 16.
    Socolinsky D A and Wolff L B. 2002. Multispectral image visualization through first-order fusion. IEEE Transactions on Image Processing, 11(8): 923-931
  17. 17.
    Tai Y, Yang J and Liu X M. 2017. Image super-resolution via deep recursive residual network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE: 2790-2798
  18. 18.
    Tan H, Huang X H, Tan H C and He C T. 2012. A comparative analysis of pixel-level image fusion based on sparse representation//Proceedings of 2012 International Conference on Computational Problem-Solving (ICCP). Leshan, China: IEEE: 332-334
  19. 19.
    Wald L, Ranchin T and Mangolini M. 1997. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6): 691-699
  20. 20.
    Wang F and Cheng Y M. 2017. Visible and infrared image enhanced fusion based on MSSTO and NSCT transform. Control and Decision, 32(2): 269-274
  21. 21.
    Wei Y C, Yuan Q Q, Shen H F and Zhang L P. 2017. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geoscience and Remote Sensing Letters, 14(10): 1795-1799
  22. 22.
    Xu X, Chen Q, Sun H J and Xia D S. 2011. Image visualization improvement based on gradient fusion. Journal of Image and Graphics, 16(2): 278-286
  23. 23.
    Zhong J Y, Yang B, Huang G Y, Zhong F and Chen Z Z. 2016. Remote sensing image fusion with convolutional neural network. Sensing and Imaging, 17(1): 10

قراءة النص الكامل

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website