Anterior|Siguiente

Remote sensing image super-resolution guided by multi-level supervision paradigm

  • role： First author第一作者
  • Affiliation：

    School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China

  • Email：limingkai@bit.edu.cn
  • Introduction：李明锴，研究方向为深度学习、空天智能计算。E-mail： limingkai@bit.edu.cn
LI Mingkai,  
  • role： Corresponding author通信作者
  • Affiliation：

    School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China

  • Email：qizhi@bit.edu.cn
  • Introduction：徐其志，研究方向为多源遥感数据融合、目标检测跟踪、在轨信息智能处理。E-mail： qizhi@bit.edu.cn
XU Qizhi*

resumen

Super-resolution improves the spatial resolution of remote sensing images, providing detailed information for various satellite applications. However, existing methods often generate pseudo-detail and lose true detail in reconstructed images due to insufficient supervision images. To address this issue, a progressive super-resolution method based on multilevel supervision structure (MSSR) was proposed.First, the MSSR introduced ground truth images as guides, which reduced the loss of true detail and mitigated the appearance of pseudo-detail in an output image. The MSSR network consists of several basic super-resolution components (BSRCs) and multilevel supervision. The overall super-resolution scale factor of the MSSR network can be set flexibly. The BSRCs can be increased or decreased, similar to building blocks, and BSRCs decrease with decreasing overall super-resolution scale factor and increase with increasing overall super-resolution scale factor. The scale factor of each BSRC is determined by the overall scale factor and the number of BSRCs. Second, a scale-factor-adjustable and lightweight basic super-resolution component was designed to enable the construction of multilevel supervision networks with different number of BSRCs and different scale factors, such as building blocks. The BSRC consists of a multiscale feature extraction module, a global feature extraction module and an image reconstruction module. Given that the scale factor of each BSRC should have a degree of flexibility, the network structure of each BSRC is the same except for the image reconstruction module. This approach shortens the overall training time of the network. Finally, a method of dividing super-resolution overall scale factor was proposed, and the effects of different number and different scale factors of BSRCs on the performance of multilevel supervision network were explored. For the super-resolution process with a certain scale factor, we need a method to divide the overall scale factor into each BSRC. The number of supervision increases with the number of BSRCs, and the super-resolution ill-posedness is reduced. In addition, the total number of network layers and the number of computations increase. We determined the optimal number of BSRCs and their respective super-resolution scale factors by comparing the super-resolution effects of multiple BSRC combinations. Additionally, a new remote sensing dataset containing worldwide scenes was constructed for the super-resolution task in this paper.To adequately train and test the proposed super-resolution method and existing methods, we used our datasets and two existing super-resolution datasets: the UCMerced and AID datasets. We compared our method with the state-of-the-art methods: VDSR, SRGAN, RDN, RCAN, DRN, and TransENet. The experiment results on three datasets demonstrated that our MSSR network outperformed these methods.The progressive network and multilevel supervision structure can effectively suppress super-resolution ill-posedness and reduce pseudo-detail and detail loss in super-resolution results. By further analyzing the experimental results, we found that multilevel supervision has greater performance gains in super-resolution tasks with a larger scale factor than other methods. We speculated that the multilevel supervision network exhibit improved performance in super-resolution tasks at 4× magnification. In future research, we will explore multilevel supervision networks in super-resolution tasks at large magnifications (i.e., 5× and 8×).

palabra clave

remote sensing image;deep learning;multi-level supervision;super-resolution;multi-scale feature extration;progressive network;parameter sharing;transfer learning

References

  1. 1.
    Arun P V, Buddhiraju K M, Porwal A and Chanussot J. 2020. CNN based spectral super-resolution of remote sensing images. Signal Processing, 169: 107394
  2. 2.
    Chen H and Luo B. 2021. Multi-angle remote sensing images super-resolution reconstruction using dynamic upsampling filter deep network. Geomatics and Information Science of Wuhan University, 46(11): 1716-1726
  3. 3.
    Cheng G and Han J W. 2016. A survey on object detection in optical remote sensing images. ISPRS Journal of Photogrammetry and Remote Sensing, 117: 11-28
  4. 4.
    Cheng G, Han J W and Lu X Q. 2017. Remote sensing image scene classification: benchmark and state of the art. Proceedings of the IEEE, 105(10): 1865-1883
  5. 5.
    Deng Z P, Sun H, Zhou S L, Zhao J P, Lei L and Zou H X. 2018. Multi-scale object detection in remote sensing imagery with convolutional neural networks. ISPRS Journal of Photogrammetry and Remote Sensing, 145: 3-22
  6. 6.
    Diakogiannis F I, Waldner F, Caccetta P and Wu C. 2020. ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162: 94-114
  7. 7.
    Dong C, Loy C C, He K M and Tang X O. 2014. Learning a deep convolutional network for image super-resolution//13th European Conference on Computer Vision. Zurich: Springer: 184-199
  8. 8.
    Dong C, Loy C C and Tang X O. 2016. Accelerating the super-resolution convolutional neural network//14th European Conference on Computer Vision. Amsterdam: Springer: 391-407
  9. 9.
    Gong Y F, Liao P Y, Zhang X D, Zhang L F, Chen G Z, Zhu K, Tan X L and Lv Z Y. 2021. Enlighten-GAN for super resolution reconstruction in mid-resolution remote sensing images. Remote Sensing, 13(6): 1104
  10. 10.
    Guo Y, Chen J, Wang J D, Chen Q, Cao J Z, Deng Z S, Xu Y W and Tan M K. 2020. Closed-loop matters: dual regression networks for single image super-resolution//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE: 5406-5415
  11. 11.
    Hu J F, Huang T Z, Deng L J, Dou H X, Hong D F and Vivone G. 2022. Fusformer: a transformer-based fusion network for hyperspectral image super-resolution. IEEE Geoscience and Remote Sensing Letters, 19: 6012305
  12. 12.
    Im J, Park H and Takeuchi W. 2019. Advances in remote sensing-based disaster monitoring and assessment. Remote Sensing, 11(18): 2181
  13. 13.
    Jiang K, Wang Z Y, Yi P, Wang G C, Lu T and Jiang J J. 2019. Edge-enhanced GAN for remote sensing image superresolution. IEEE Transactions on Geoscience and Remote Sensing, 57(8): 5799-5812
  14. 14.
    Johnson J, Alahi A and Fei-Fei L. 2016. Perceptual losses for real-time style transfer and super-resolution//14th European Conference on Computer Vision. Amsterdam: Springer: 694-711
  15. 15.
    Kampffmeyer M, Salberg A B and Jenssen R. 2016. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Las Vegas: IEEE: 680-688
  16. 16.
    Kim J, Lee J K and Lee K M. 2016. Accurate image super-resolution using very deep convolutional networks//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 1646-1654
  17. 17.
    Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z H and Shi W Z. 2017. Photo-realistic single image super-resolution using a generative adversarial network//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE: 105-114
  18. 18.
    Lei S, Shi Z W and Mo W J. 2022. Transformer-based multistage enhancement for remote sensing image super-resolution. IEEE Transactions on Geoscience and Remote Sensing, 60: 5615611
  19. 19.
    Li K, Wan G, Cheng G, Meng L Q and Han J W. 2020. Object detection in optical remote sensing images: a survey and a new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 159: 296-307
  20. 20.
    Liang J Y, Cao J Z, Sun G L, Zhang K, Van Gool L and Timofte R. 2021. SwinIR: image restoration using swin transformer//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE: 1833-1844
  21. 21.
    Lim B, Son S, Kim H, Nah S and Lee K M. 2017. Enhanced deep residual networks for single image super-resolution//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE: 1132-1140
  22. 22.
    Liu Z, Lin Y T, Cao Y, Hu H, Wei Y X, Zhang Z, Lin S and Guo B N. 2021. Swin transformer: hierarchical vision transformer using shifted windows//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE: 9992-10002
  23. 23.
    Lu X Q, Zheng X T and Yuan Y. 2017. Remote sensing scene classification by unsupervised representation learning. IEEE Transactions on Geoscience and Remote Sensing, 55(9): 5148-5157
  24. 24.
    Ma J Q, Liang Z T and Zhang L. 2022. A text attention network for spatial deformation robust scene text image super-resolution//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE: 5901-5910
  25. 25.
    Mei Y Q, Fan Y C and Zhou Y Q. 2021. Image super-resolution with non-local sparse attention//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE: 3516-3525
  26. 26.
    Shi W Z, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D and Wang Z H. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 1874-1883
  27. 27.
    Tang X T, Yang X, Li F, Ma J and Liang L. Super-resolution reconstruction of hyper-temporal remote sensing images based on self-attention. National Remote Sensing Bulletin (唐晓天, 杨雪, 李峰, 马骏, 梁亮. 基于自注意力的超时相遥感图像超分辨率重建. 遥感学报 [DOI: 10.11834/jrs.20221825]
  28. 28.
    Tian C W, Yuan Y X, Zhang S C, Lin C W, Zuo W M and Zhang D. 2022. Image super-resolution with an enhanced group convolutional neural network. Neural Networks, 153: 373-385
  29. 29.
    Twumasi N Y D, Shao Z F and Orhan A. 2019. Remote sensing and GIS methods in urban disaster monitoring and management–an overview. International Journal of Trend in Scientific Research and Development, 3(4): 918-926
  30. 30.
    Voinov S, Schwarz E, Krause D and Berg M. 2018. Processing framework to support maritime surveillance applications based on optical remote sensing images//Proceedings Volume 10773, Sixth International Conference on Remote Sensing and Geoinformation of the Environment. Paphos: SPIE: 268-273
  31. 31.
    Wang P, Yao H Y and Zhang G. 2021. Super-resolution flood inundation mapping for multispectral image based on super-pixel scale. National Remote Sensing Bulletin, 25(2): 641-652
  32. 32.
    Wang Q, Zhang X D, Chen G Z, Dai F, Gong Y F and Zhu K. 2018. Change detection based on Faster R-CNN for high-resolution remote sensing images. Remote Sensing Letters, 9(10): 923-932
  33. 33.
    Wang T F, Xie J X, Sun W X, Yan Q and Chen Q F. 2021. Dual-camera super-resolution with aligned attention modules//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE: 1981-1990
  34. 34.
    Xia G S, Hu J W, Hu F, Shi B G, Bai X, Zhong Y F, Zhang L P and Lu X Q. 2017. AID: a benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7): 3965-3981
  35. 35.
    Xie J, He N J, Fang L Y and Plaza A. 2019. Scale-free convolutional neural network for remote sensing scene classification. IEEE Transactions on Geoscience and Remote Sensing, 57(9): 6916-6928
  36. 36.
    Xu Y Y, Luo W, Hu A N, Xie Z, Xie X J and Tao L F. 2022. TE-SAGAN: an improved generative adversarial network for remote sensing super-resolution images. Remote Sensing, 14(10): 2425
  37. 37.
    Yang Y and Newsam S. 2010. Bag-of-visual-words and spatial extensions for land-use classification//Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems. San Jose: ACM: 270-279
  38. 38.
    Ye P. 2022. Remote sensing approaches for meteorological disaster monitoring: recent achievements and new challenges. International Journal of Environmental Research and Public Health, 19(6): 3701
  39. 39.
    Yuan X H, Shi J F and Gu L C. 2021. A review of deep learning methods for semantic segmentation of remote sensing imagery. Expert Systems with Applications, 169: 114417
  40. 40.
    Zhang K, Liang J Y, Van Gool L and Timofte R. 2021. Designing a practical degradation model for deep blind image super-resolution//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE: 4771-4780
  41. 41.
    Zhang Y, Tian Y, Kong Y, Zhong B and Fu Y. 2018a. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition: 2472-2481
  42. 42.
    Zhang S M, Wu R Z, Xu K Y, Wang J M and Sun W W. 2019. R-CNN-based ship detection from high resolution remote sensing imagery. Remote Sensing, 11(6): 631
  43. 43.
    Zhang Y, Li K, Li K, Wang L, Zhong B and Fu Y. 2018b. Image super-resolution using very deep residual channel attention networks//Proceedings of the European conference on computer vision (ECCV). 286-301
  44. 44.
    Zhen J N, Jiang X P, Zhao D M, Wang J J, Miao J and Wu G F. 2022. Retrieving canopy nitrogen content of mangrove forests from Sentinel-2 super-resolution reconstruction data. National Remote Sensing Bulletin, 26(6): 1206-1219

Leer el texto completo

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website