Pan-sharpening by deep recursive residual network

Fen WANG; Qing GUO; Xiaoqing GE

doi:10.11834/jrs.20219250

Technologies and Methodologies | Views : 0 下载量: 970 CSCD: 5

PDF
Export
Share
Collection
Album

Pan-sharpening by deep recursive residual network
Vol. 25, Issue 6, Pages: 1244-1256(2021)
Received：11 July 2019，

Published：07 June 2021
DOI： 10.11834/jrs.20219250
稿件说明：

移动端阅览

王芬，郭擎，葛小青.2021.深度递归残差网络的遥感图像空谱融合.遥感学报，25（6）： 1244-1256 DOI： 10.11834/jrs.20219250.

Wang F，Guo Q and Ge X Q. 2021. Pan-sharpening by deep recursive residual network. National Remote Sensing Bulletin， 25（6）：1244-1256 DOI： 10.11834/jrs.20219250.

摘要

为了充分利用多光谱图像的光谱信息和全色图像的空间信息，本文提出一种基于深度递归残差网络的遥感图像空谱融合方法。方法将残差网络和递归网络相结合，利用残差网络学习低空间分辨率多光谱图像与高空间分辨率多光谱图像之间的残差，同时结合全局残差和局部残差，加快网络的收敛速度，解决深层次网络容易出现的梯度消失和梯度爆炸问题；利用递归神经网络在不增加权重参数、减轻过拟合的情况下通过提高网络层数来提高精度，得到更好的图像融合效果。为了验证本文方法的有效性，应用遥感图像进行模拟实验、真实实验和泛化实验，实验结果与传统方法和现有深度学习方法进行对比分析。主观视觉和客观定量评价表明，本文方法很好地改善了传统方法存在的光谱失真现象，并且较现有深度学习方法学习到更深层次丰富的图像特征，更好地保留了图像的空谱信息，同时泛化实验也说明本文的网络具有较好的泛化能力。

Abstract

Pan-sharpening is a task in the field of remote sensing data fusion

in which multispectral (MS) images with rich spectral information but low spatial resolution and panchromatic (PAN) images with rich spatial details but only grey information are fused to yield images with high spatial and spectral resolution. Traditional Component Substitution (CS) methods replace a particular component of the MS image transformation with a PAN image

and then inversely transforms it to obtain the final fused image. The traditional MultiResolution Analysis (MRA) methods first extract spatial structures from the PAN image by using MRA transforms

and then the extracted spatial structure information is injected into the up-sampled MS images to obtain the fused image. The whole fusing process of the CS and MRA methods can be described as linear functions. However

the performance of such linear models are limited by their linearity

which often has spectral distortion. In recent years

many advanced nonlinear deep learning models have been proposed. However

those existing deep learning fusion models are relatively simple and pose difficultly in learn in-depth features. To overcome the shortcomings of the current models

we propose a deep recursive residual network that is specifically designed for the pan-sharpening task.

Considering that the low-resolution input image and the high-resolution output image have high similarity

learning the relationship between input and output is highly redundant and difficult. If the sparse residual features between input and output are learned directly

then the network convergence can be significantly improved. Thus

the residual learning introduces the network structure

in which the introduced residuals include global residuals and local residuals. Such a structure is conducive to learning and not prone to overfitting. Moreover

the residual network can solve the problem of deep network gradient disappearance and gradient explosion well. Recursive network improves accuracy by increasing the number of network layers without increasing weight parameters. Specifically

as we use the residual network globally

recursive learning is introduced into residual learning by constructing recursive blocks structure

whereas multiple local residual units are stacked together in the recursive block. Through such an end-to-end network design

a better image fusion effect is obtained.

Given that no ideal fusion result has been used as a label

we made a data set according to Wald’s protocol using the original MS as the ideal fused image

downsampling and then upsampling the MS as the MS of the network input

and the downsampled PAN as the PAN of the network input. To comprehensively analyze our experimental results

we performed a large number of simulation experiments and real experiments on the 4-band GaoFen-1 data and 8-band WorldView-2 data with abundant feature types. We then generalized them to 4-band GeoEye data and 8-band WorldView-3 data. Experimental results are compared with traditional methods and the existing deep learning methods. The subjective visual analysis and objective evaluation indicators show that the proposed method reduces the spectral distortion phenomenon of traditional methods and preserves the spectrum of an image better than the existing deep learning method does.

The deep network designed in this paper has learned more in-depth and more luxurious image features and has achieved better fusion effects than existing methods. It uses a residual network to solve profound network gradient disappearance

gradient explosion

and degradation problems. In addition

the weight parameters are reduced by the design of the recurrent recursive block

and the network speed is improved. The generalization experiment shows that our network has a good generalization ability.

关键词

Keywords

references

Azarang A and Ghassemian H . 2017 . A new pansharpening method using multi resolution analysis framework and deep neural networks // Proceedings of the 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA) . Shahrekord, Iran : IEEE : 1 - 6 [ DOI: 10.1109/PRIA.2017.7983017 http://dx.doi.org/10.1109/PRIA.2017.7983017 ]

Chavez P S and Kwarteng A Y . 1988 . Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis . Photogrammetric Engineering and Remote Sensing , 55 ( 3 ): 339 - 348

Choi J , Yu K and Kim Y . 2011 . A new adaptive component-substitution-based satellite image fusion by using partial replacement . IEEE Transactions on Geoscience and Remote Sensing , 49 ( 1 ): 295 - 309 [ DOI: 10.1109/TGRS.2010.2051674 http://dx.doi.org/10.1109/TGRS.2010.2051674 ]

Easley G , Labate D and Lim W Q . 2008 . Sparse directional image representations using the discrete shearlet transform . Applied and Computational Harmonic Analysis , 25 ( 1 ): 25 - 46 [ DOI: 10.1016/j.acha.2007.09.003 http://dx.doi.org/10.1016/j.acha.2007.09.003 ]

Ehlers M . 1991 . Multisensor image fusion techniques in remote sensing . ISPRS Journal of Photogrammetry and Remote Sensing , 46 ( 1 ): 19 - 30 [ DOI: 10.1016/0924-2716(91)90003-e http://dx.doi.org/10.1016/0924-2716(91)90003-e ]

Koutsias N , Karteris M and Chuvieco E . 2000 . The use of intensity-hue-saturation transformation of landsat-5 thematic mapper data for burned land mapping . Photogrammetric Engineering and Remote Sensing , 66 ( 7 ): 829 - 839

Laben C A and Brower B V . 2000 . Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S ., No . 6011875

Le Pennec E and Mallat S . 2000 . Image compression with geometrical wavelets // Proceedings 2000 International Conference on Image Processing . Vancouver, Canada : IEEE : 661 - 664 [ DOI: 10.1109/ICIP.2000.901045 http://dx.doi.org/10.1109/ICIP.2000.901045 ]

Liang M and Hu X L . 2015 . Recurrent convolutional neural network for object recognition // Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Boston, MA, USA : IEEE : 3367 - 3375 [ DOI: 10.1109/cvpr.2015.7298958 http://dx.doi.org/10.1109/cvpr.2015.7298958 ]

Mallat S G . 1989 . Multifrequency channel decompositions of images and wavelet models . IEEE Transactions on Acoustics, Speech, and Signal Processing , 37 ( 12 ): 2091 - 2110 [ DOI: 10.1109/29.45554 http://dx.doi.org/10.1109/29.45554 ]

Masi G , Cozzolino D , Verdoliva L and Scarpa G . 2016 . Pansharpening by convolutional neural networks . Remote Sensing , 8 ( 7 ): 594 [ DOI: 10.3390/rs8070594 http://dx.doi.org/10.3390/rs8070594 ]

Micheal A A and Vani K . 2017 . Multi-sensor image fusion of the lunar image data using DT-CWT and curvelet transform // Proceedings of the 2017 4th International Conference on Electronics and Communication Systems (ICECS) . Coimbatore, India : IEEE : 49 - 53 [ DOI: 10.1109/ECS.2017.8067835 http://dx.doi.org/10.1109/ECS.2017.8067835 ]

Piella G . 2009 . Image fusion for enhanced visualization: A variational approach . International Journal of Computer Vision , 83 ( 1 ): 1 - 11 [ DOI: 10.1007/s11263-009-0206-4 http://dx.doi.org/10.1007/s11263-009-0206-4 ]

Rao Y Z , He L and Zhu J W . 2017 . A residual convolutional neural network for pan-shaprening // Proceedings of 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP) . Shanghai, China : IEEE : 1 - 4 [ DOI: 10.1109/RSIP.2017.7958807 http://dx.doi.org/10.1109/RSIP.2017.7958807 ]

Simonyan K and Zisserman A . 2014 . Very deep convolutional networks for large-scale image recognition . arXiv: 1409.1556 .

Socolinsky D A and Wolff L B . 2002 . Multispectral image visualization through first-order fusion . IEEE Transactions on Image Processing , 11 ( 8 ): 923 - 931 [ DOI: 10.1109/tip.2002.801588 http://dx.doi.org/10.1109/tip.2002.801588 ]

Tai Y , Yang J and Liu X M . 2017 . Image super-resolution via deep recursive residual network // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu, HI, USA : IEEE : 2790 - 2798 [ DOI: 10.1109/cvpr.2017.298 http://dx.doi.org/10.1109/cvpr.2017.298 ]

Tan H , Huang X H , Tan H C and He C T . 2012 . A comparative analysis of pixel-level image fusion based on sparse representation // Proceedings of 2012 International Conference on Computational Problem-Solving (ICCP) . Leshan, China : IEEE : 332 - 334 [ DOI: 10.1109/ICCPS.2012.6384257 http://dx.doi.org/10.1109/ICCPS.2012.6384257 ]

Wald L , Ranchin T and Mangolini M . 1997 . Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images . Photogrammetric Engineering and Remote Sensing , 63 ( 6 ): 691 - 699

Wang F and Cheng Y M . 2017 . Visible and infrared image enhanced fusion based on MSSTO and NSCT transform . Control and Decision , 32 ( 2 ): 269 - 274

王峰 , 程咏梅 . 2017 . 基于MSSTO与NSCT变换的可见光与红外图像增强融合 . 控制与决策 , 32 ( 2 ): 269- 274 [ DOI: 10.13195/j.kzyjc.2015.1406 http://dx.doi.org/10.13195/j.kzyjc.2015.1406 ]

Wei Y C , Yuan Q Q , Shen H F and Zhang L P . 2017 . Boosting the accuracy of multispectral image pansharpening by learning a deep residual network . IEEE Geoscience and Remote Sensing Letters , 14 ( 10 ): 1795 - 1799 [ DOI: 10.1109/lgrs.2017.2736020 http://dx.doi.org/10.1109/lgrs.2017.2736020 ]

Xu X , Chen Q , Sun H J and Xia D S . 2011 . Image visualization improvement based on gradient fusion . Journal of Image and Graphics , 16 ( 2 ): 278 - 286

许欣 , 陈强 , 孙怀江 , 夏德深 . 2011 . 基于梯度域融合的图像视觉效果改善 . 中国图象图形学报 , 16 ( 2 ): 278- 286 [ DOI: 10.11834/jig.20110221 http://dx.doi.org/10.11834/jig.20110221 ]

Zhong J Y , Yang B , Huang G Y , Zhong F and Chen Z Z . 2016 . Remote sensing image fusion with convolutional neural network . Sensing and Imaging , 17 ( 1 ): 10 [ DOI: 10.1007/s11220-016-0135-6 http://dx.doi.org/10.1007/s11220-016-0135-6 ]

Alert me when the article has been cited

提交

Multimodal object detection method using adaptive dusion of infrared and visible features

Fusion of panchromatic and multispectral remote sensing images using a dual-branch generative adversarial network combined with Transformer

MtSCCD： Land-use scene classification and change-detection dataset for deep learning

Hyperspectral unmixing network considering spectral information and superpixel segmentation

Spectral-spatial attention bilateral network for hyperspectral image classification