Précédent|Prochain

Pan-sharpening by residual network with dense convolution for remote sensing images

  • role： First author第一作者
  • Affiliation：

    Key Laboratory of Electronics Engineering, College of Heilongjiang Province, Heilongjiang University, Harbin 150080, China

    Key Laboratory of Information Fusion Estimation and Detection, Heilongjiang Province, Heilongjiang University, Harbin 150080, China

  • Email：13059027068@163.com
  • Introduction：陈毛毛，1994年生，女，硕士研究生，研究方向为遥感图像处理、深度学习。E-mail：13059027068@163.com
CHEN Maomao13,  
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

GUO Qing2,  
  • role： Corresponding author通信作者
  • Affiliation：

    Key Laboratory of Electronics Engineering, College of Heilongjiang Province, Heilongjiang University, Harbin 150080, China

    Key Laboratory of Information Fusion Estimation and Detection, Heilongjiang Province, Heilongjiang University, Harbin 150080, China

  • Email：2002039@hlju.edu.cn
  • Introduction：刘明亮，1980年生，男，教授，研究方向为智能检测、故障诊断和信号处理。E-mail：2002039@hlju.edu.cn
LIU Mingliang13*,  
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

LI An2

résumé

Pan-sharpening (also known as remote sensing image fusion) aims to generate Multi-Spectral (MS) images with high spatial resolution and high spectral resolution by fusing high spatial resolution panchromatic (PAN) images and high spectral resolution MS images with low spatial resolution. Traditional pan-sharpening methods mainly include the component substitution method, the multiresolution analysis method, and the model-based optimization method. These fusion methods involve linear models, which are difficult to use in achieving the appropriate trade-off between spatial improvement and spectral preservation. In addition, they often introduce spectral or spatial distortion. Recently, many fusion methods based on deep learning have been proposed. However, their network depth is relatively shallow, and detailed information is inevitably lost during feature transfer. Hence, we propose a deep residual network with dense convolution for pan-sharpening.As the network becomes deep, the features of different levels become complementary to one other. However, most fusion methods based on deep learning ignore making full use of the information of each convolution layer. The densely connected convolutional network allows the features of all previous layers to be used as input for each layer in one densely connected block. To fully utilize the features learned from all convolution layers, we establish the multiple densely convolutional blocks to reuse features. Moreover, the information flow is accelerated by the transition layer between every two blocks. These maximize the use of features and extract rich features. Given the great correlation between deep features and shallow features, residual learning is used to supervise the densely convolutional structure to learn the difference between them, that is, residual features. Thus, residual learning combines shallow features and residual features to obtain further advanced information from MS and PAN images, which prepares for obtaining fused images with high spatial and spectral resolution.To evaluate the effectiveness of the proposed method, we conduct simulated and real-image experiments on the 4-band GaoFen-1 data and 8-band WorldView-2 data with multiple land types. The trained network is generalized well to WorldView-3 images without pre-training. The visual and the quantitative assessment results show that the high-resolution fused images obtained by using the proposed method are superior to the results produced by the traditional and deep learning methods. The proposed approach achieves high spectral fidelity and enhances spatial details by reusing features.The proposed method makes comprehensive use of the advantages of densely convolutional blocks and residual learning. In the feature extraction stage, different levels of features are connected in series through the densely convolutional blocks. This characteristic makes the transmission of features and gradients effective in alleviating the gradient disappearance problem and provides rich spatial and spectral feature for fusion results. In the feature fusion stage, residual learning is used to learn the difference between deep features and shallow features, that is, residual feature. Hence, the convergence speed of the network is accelerated. The experiment result shows that our network has good fusion and generalization abilities.

mots-clés

pan-sharpening;remote sensing image fusion;deep learning;densely connected convolutional network;densely convolutional blocks

References

  1. 1.
    Aharon M, Elad M and Bruckstein A. 2006. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11): 4311-4322
  2. 2.
    Aiazzi B, Alparone L, Baronti S, Garzelli A and Selva M. 2006. MTF-tailored multiscale fusion of high-resolution MS and pan imagery. Photogrammetric Engineering and Remote Sensing, 72(5): 591-596
  3. 3.
    Alparone L, Wald L, Chanussot J, Thomas C, Gamba P and Bruce L M. 2007. Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Transactions on Geoscience and Remote Sensing, 45(10): 3012-3021
  4. 4.
    Cetin M and Musaoglu N. 2009. Merging hyperspectral and panchromatic image data: qualitative and quantitative analysis. International Journal of Remote Sensing, 30(7): 1779-1804
  5. 5.
    Chen Q J, Li Y and Chai Y Z. 2018. Remote sensing image fusion based on deep learning non-subsampled shearlet. Journal of Applied Optics, 39(5): 655-666
  6. 6.
    Cheng J, Liu H J, Liu T, Wang F and Li H S. 2015. Remote sensing image fusion via wavelet transform and sparse representation. ISPRS Journal of Photogrammetry and Remote Sensing, 104: 158-173
  7. 7.
    Choi J, Yu K Y and Kim Y. 2011. A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing, 49(1): 295-309
  8. 8.
    Garzelli A, Nencini F and Capobianco L. 2008. Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, 46(1): 228-236
  9. 9.
    Gillespie A R, Kahle A B and Walker R E. 1987. Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sensing of Environment, 22(3): 343-365
  10. 10.
    He K M and Sun J. 2015. Convolutional neural networks at constrained time cost//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE: 5353-5360
  11. 11.
    He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE: 770-778
  12. 12.
    Huang G, Liu Z, Van Der Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE: 2261-2269
  13. 13.
    Jia Y Q, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S and Darrell T. 2014. Caffe: convolutional architecture for fast feature embedding//Proceedings of the 22nd ACM International Conference on Multimedia. Orlando, Florida, USA: Association for Computing Machinery: 675-678
  14. 14.
    Jiang C, Zhang H Y, Shen H F and Zhang L P. 2014. Two-step sparse coding for the pan-sharpening of remote sensing images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(5): 1792-1805
  15. 15.
    Kuang P, Ma T S, Chen Z W and Li F. 2018. Image super-resolution with densely connected convolutional networks. Applied Intelligence, 49(1): 125-136
  16. 16.
    Laben C A and Brower B V. 2000. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S., No. 6,011,875
  17. 17.
    Li N, Huang N and Xiao L. 2017. PAN-Sharpening via residual deep learning//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium. Fort Worth, Texas, USA: IEEE: 5133-5136
  18. 18.
    Loncan L, de Almeida L B, Bioucas-Dias J M, Briottet X, Chanussot J, Dobigeon N, Fabre S, Liao W Z, Licciardi G A, Simões M, Tourneret J Y, Veganzones M A, Vivone G, Wei Q and Yokoya N. 2015. Hyperspectral pansharpening: a review. IEEE Geoscience and Remote Sensing Magazine, 3(3): 27-46
  19. 19.
    Masi G, Cozzolino D, Verdoliva L and Scarpa G. 2016. Pansharpening by convolutional neural networks. Remote Sensing, 8(7): 594
  20. 20.
    Otazu X, Gonzalez-Audicana M, Fors O and Nunez J. 2005. Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10): 2376-2385
  21. 21.
    Tong T, Li G, Liu X J and Gao Q Q. 2017. Image super-resolution using dense skip connections//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE: 4799-4807
  22. 22.
    Tu T M, Huang P S, Hung C L and Chang C P. 2004. A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geoscience and Remote Sensing Letters, 1(4): 309-312
  23. 23.
    Vivone G, Alparone L, Chanussot J, Mura M D, Garzelli A, Licciardi G A, Restaino R and Wald L. 2015. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53(5): 2565-2586
  24. 24.
    Wald L, Ranchin T and Mangolini M. 1997. Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6): 691-699
  25. 25.
    Wang F and Cheng Y M. 2017. Visible and infrared image enhanced fusion based on MSSTO and NSCT transform. Control and Decision, 32(2): 269-274
  26. 26.
    Wei Y C, Yuan Q Q, Shen H F and Zhang L P. 2017. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geoscience and Remote Sensing Letters, 14(10): 1795-1799
  27. 27.
    Wen R, Fu K, Sun H, Sun X and Wang L. 2018. Image superresolution using densely connected residual networks. IEEE Signal Processing Letters, 25(10): 1565-1569
  28. 28.
    Xu J, Chae Y, Stenger B and Datta A. 2018. Dense bynet: residual dense network for image super resolution//Proceedings of the 25th IEEE International Conference on Image Processing (ICIP). Athens: IEEE: 4809-4817
  29. 29.
    Zhang Y L, Tian Y P, Kong Y, Zhong B N and Fu Y. 2018. Residual dense network for image super-resolution//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE: 2472-2481

Lire l'article complet

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website