收稿：2019-07-11，
纸质出版：2021-06-07
移动端阅览
为了充分利用多光谱图像的光谱信息和全色图像的空间信息，本文提出一种基于深度递归残差网络的遥感图像空谱融合方法。方法将残差网络和递归网络相结合，利用残差网络学习低空间分辨率多光谱图像与高空间分辨率多光谱图像之间的残差，同时结合全局残差和局部残差，加快网络的收敛速度，解决深层次网络容易出现的梯度消失和梯度爆炸问题；利用递归神经网络在不增加权重参数、减轻过拟合的情况下通过提高网络层数来提高精度，得到更好的图像融合效果。为了验证本文方法的有效性，应用遥感图像进行模拟实验、真实实验和泛化实验，实验结果与传统方法和现有深度学习方法进行对比分析。主观视觉和客观定量评价表明，本文方法很好地改善了传统方法存在的光谱失真现象，并且较现有深度学习方法学习到更深层次丰富的图像特征，更好地保留了图像的空谱信息，同时泛化实验也说明本文的网络具有较好的泛化能力。
Pan-sharpening is a task in the field of remote sensing data fusion
in which multispectral (MS) images with rich spectral information but low spatial resolution and panchromatic (PAN) images with rich spatial details but only grey information are fused to yield images with high spatial and spectral resolution. Traditional Component Substitution (CS) methods replace a particular component of the MS image transformation with a PAN image
and then inversely transforms it to obtain the final fused image. The traditional MultiResolution Analysis (MRA) methods first extract spatial structures from the PAN image by using MRA transforms
and then the extracted spatial structure information is injected into the up-sampled MS images to obtain the fused image. The whole fusing process of the CS and MRA methods can be described as linear functions. However
the performance of such linear models are limited by their linearity
which often has spectral distortion. In recent years
many advanced nonlinear deep learning models have been proposed. However
those existing deep learning fusion models are relatively simple and pose difficultly in learn in-depth features. To overcome the shortcomings of the current models
we propose a deep recursive residual network that is specifically designed for the pan-sharpening task.
Considering that the low-resolution input image and the high-resolution output image have high similarity
learning the relationship between input and output is highly redundant and difficult. If the sparse residual features between input and output are learned directly
then the network convergence can be significantly improved. Thus
the residual learning introduces the network structure
in which the introduced residuals include global residuals and local residuals. Such a structure is conducive to learning and not prone to overfitting. Moreover
the residual network can solve the problem of deep network gradient disappearance and gradient explosion well. Recursive network improves accuracy by increasing the number of network layers without increasing weight parameters. Specifically
as we use the residual network globally
recursive learning is introduced into residual learning by constructing recursive blocks structure
whereas multiple local residual units are stacked together in the recursive block. Through such an end-to-end network design
a better image fusion effect is obtained.
Given that no ideal fusion result has been used as a label
we made a data set according to Wald’s protocol using the original MS as the ideal fused image
downsampling and then upsampling the MS as the MS of the network input
and the downsampled PAN as the PAN of the network input. To comprehensively analyze our experimental results
we performed a large number of simulation experiments and real experiments on the 4-band GaoFen-1 data and 8-band WorldView-2 data with abundant feature types. We then generalized them to 4-band GeoEye data and 8-band WorldView-3 data. Experimental results are compared with traditional methods and the existing deep learning methods. The subjective visual analysis and objective evaluation indicators show that the proposed method reduces the spectral distortion phenomenon of traditional methods and preserves the spectrum of an image better than the existing deep learning method does.
The deep network designed in this paper has learned more in-depth and more luxurious image features and has achieved better fusion effects than existing methods. It uses a residual network to solve profound network gradient disappearance
gradient explosion
and degradation problems. In addition
the weight parameters are reduced by the design of the recurrent recursive block
and the network speed is improved. The generalization experiment shows that our network has a good generalization ability.
Azarang A and Ghassemian H . 2017 . A new pansharpening method using multi resolution analysis framework and deep neural networks // Proceedings of the 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA) . Shahrekord, Iran : IEEE : 1 - 6 [ DOI: 10.1109/PRIA.2017.7983017 http://dx.doi.org/10.1109/PRIA.2017.7983017 ]
Chavez P S and Kwarteng A Y . 1988 . Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis . Photogrammetric Engineering and Remote Sensing , 55 ( 3 ): 339 - 348
Choi J , Yu K and Kim Y . 2011 . A new adaptive component-substitution-based satellite image fusion by using partial replacement . IEEE Transactions on Geoscience and Remote Sensing , 49 ( 1 ): 295 - 309 [ DOI: 10.1109/TGRS.2010.2051674 http://dx.doi.org/10.1109/TGRS.2010.2051674 ]
Easley G , Labate D and Lim W Q . 2008 . Sparse directional image representations using the discrete shearlet transform . Applied and Computational Harmonic Analysis , 25 ( 1 ): 25 - 46 [ DOI: 10.1016/j.acha.2007.09.003 http://dx.doi.org/10.1016/j.acha.2007.09.003 ]
Ehlers M . 1991 . Multisensor image fusion techniques in remote sensing . ISPRS Journal of Photogrammetry and Remote Sensing , 46 ( 1 ): 19 - 30 [ DOI: 10.1016/0924-2716(91)90003-e http://dx.doi.org/10.1016/0924-2716(91)90003-e ]
Koutsias N , Karteris M and Chuvieco E . 2000 . The use of intensity-hue-saturation transformation of landsat-5 thematic mapper data for burned land mapping . Photogrammetric Engineering and Remote Sensing , 66 ( 7 ): 829 - 839
Laben C A and Brower B V . 2000 . Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S ., No . 6011875
Le Pennec E and Mallat S . 2000 . Image compression with geometrical wavelets // Proceedings 2000 International Conference on Image Processing . Vancouver, Canada : IEEE : 661 - 664 [ DOI: 10.1109/ICIP.2000.901045 http://dx.doi.org/10.1109/ICIP.2000.901045 ]
Liang M and Hu X L . 2015 . Recurrent convolutional neural network for object recognition // Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Boston, MA, USA : IEEE : 3367 - 3375 [ DOI: 10.1109/cvpr.2015.7298958 http://dx.doi.org/10.1109/cvpr.2015.7298958 ]
Mallat S G . 1989 . Multifrequency channel decompositions of images and wavelet models . IEEE Transactions on Acoustics, Speech, and Signal Processing , 37 ( 12 ): 2091 - 2110 [ DOI: 10.1109/29.45554 http://dx.doi.org/10.1109/29.45554 ]
Masi G , Cozzolino D , Verdoliva L and Scarpa G . 2016 . Pansharpening by convolutional neural networks . Remote Sensing , 8 ( 7 ): 594 [ DOI: 10.3390/rs8070594 http://dx.doi.org/10.3390/rs8070594 ]
Micheal A A and Vani K . 2017 . Multi-sensor image fusion of the lunar image data using DT-CWT and curvelet transform // Proceedings of the 2017 4th International Conference on Electronics and Communication Systems (ICECS) . Coimbatore, India : IEEE : 49 - 53 [ DOI: 10.1109/ECS.2017.8067835 http://dx.doi.org/10.1109/ECS.2017.8067835 ]
Piella G . 2009 . Image fusion for enhanced visualization: A variational approach . International Journal of Computer Vision , 83 ( 1 ): 1 - 11 [ DOI: 10.1007/s11263-009-0206-4 http://dx.doi.org/10.1007/s11263-009-0206-4 ]
Rao Y Z , He L and Zhu J W . 2017 . A residual convolutional neural network for pan-shaprening // Proceedings of 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP) . Shanghai, China : IEEE : 1 - 4 [ DOI: 10.1109/RSIP.2017.7958807 http://dx.doi.org/10.1109/RSIP.2017.7958807 ]
Simonyan K and Zisserman A . 2014 . Very deep convolutional networks for large-scale image recognition . arXiv: 1409.1556 .
Socolinsky D A and Wolff L B . 2002 . Multispectral image visualization through first-order fusion . IEEE Transactions on Image Processing , 11 ( 8 ): 923 - 931 [ DOI: 10.1109/tip.2002.801588 http://dx.doi.org/10.1109/tip.2002.801588 ]
Tai Y , Yang J and Liu X M . 2017 . Image super-resolution via deep recursive residual network // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu, HI, USA : IEEE : 2790 - 2798 [ DOI: 10.1109/cvpr.2017.298 http://dx.doi.org/10.1109/cvpr.2017.298 ]
Tan H , Huang X H , Tan H C and He C T . 2012 . A comparative analysis of pixel-level image fusion based on sparse representation // Proceedings of 2012 International Conference on Computational Problem-Solving (ICCP) . Leshan, China : IEEE : 332 - 334 [ DOI: 10.1109/ICCPS.2012.6384257 http://dx.doi.org/10.1109/ICCPS.2012.6384257 ]
Wald L , Ranchin T and Mangolini M . 1997 . Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images . Photogrammetric Engineering and Remote Sensing , 63 ( 6 ): 691 - 699
Wang F and Cheng Y M . 2017 . Visible and infrared image enhanced fusion based on MSSTO and NSCT transform . Control and Decision , 32 ( 2 ): 269 - 274
王峰 , 程咏梅 . 2017 . 基于MSSTO与NSCT变换的可见光与红外图像增强融合 . 控制与决策 , 32 ( 2 ): 269- 274 [ DOI: 10.13195/j.kzyjc.2015.1406 http://dx.doi.org/10.13195/j.kzyjc.2015.1406 ]
Wei Y C , Yuan Q Q , Shen H F and Zhang L P . 2017 . Boosting the accuracy of multispectral image pansharpening by learning a deep residual network . IEEE Geoscience and Remote Sensing Letters , 14 ( 10 ): 1795 - 1799 [ DOI: 10.1109/lgrs.2017.2736020 http://dx.doi.org/10.1109/lgrs.2017.2736020 ]
Xu X , Chen Q , Sun H J and Xia D S . 2011 . Image visualization improvement based on gradient fusion . Journal of Image and Graphics , 16 ( 2 ): 278 - 286
许欣 , 陈强 , 孙怀江 , 夏德深 . 2011 . 基于梯度域融合的图像视觉效果改善 . 中国图象图形学报 , 16 ( 2 ): 278- 286 [ DOI: 10.11834/jig.20110221 http://dx.doi.org/10.11834/jig.20110221 ]
Zhong J Y , Yang B , Huang G Y , Zhong F and Chen Z Z . 2016 . Remote sensing image fusion with convolutional neural network . Sensing and Imaging , 17 ( 1 ): 10 [ DOI: 10.1007/s11220-016-0135-6 http://dx.doi.org/10.1007/s11220-016-0135-6 ]
相关作者
相关机构
京公网安备11010802024621
