浏览全部资源
扫码关注微信
Received：06 July 2023，
Published：07 July 2024
移动端阅览
超分辨率技术可提升遥感图像空间分辨率，为基于遥感图像的目标检测、场景分类等任务提供更加清晰的数据集，具有广泛的应用价值。然而，现有基于深度学习的超分辨率方法存在监督次数不足的问题，导致超分辨率重建图像中易出现细节损失和伪细节。针对这一问题，本文提出基于分级监督的遥感图像超分辨率方法（MSSR）。首先，提出了一个分级监督网络架构，通过引入多级真值图像作为监督，为超分辨率过程提供充足的图像细节恢复指引，进而减少超分辨率结果中细节损失和伪细节的出现。其次，为了便于构建级数可变、超分辨率倍数可变的分级监督网络，设计了一个轻量化的、超分辨率倍数可灵活调整的同构超分辨率模块（BSRC）。各级BSRC的网络结构基本相同，便于迁移网络权重，缩短训练时间。最后，针对分级网络超分辨率倍数一定时，网络级数及各级分辨率倍数有多种组合方式的问题，对比多种分级方式下的超分辨率结果，给出最佳网络分级方式。此外，构建了一个包含世界各地复杂细节地面场景的遥感图像数据集（RSSRD）。在该数据集和UCMerced、AID两个公开数据集上进行超分辨率实验，实验结果显示本文方法优于现有常用超分辨率方法。
Super-resolution improves the spatial resolution of remote sensing images
providing detailed information for various satellite applications. However
existing methods often generate pseudo-detail and lose true detail in reconstructed images due to insufficient supervision images. To address this issue
a progressive super-resolution method based on multilevel supervision structure (MSSR) was proposed.
First
the MSSR introduced ground truth images as guides
which reduced the loss of true detail and mitigated the appearance of pseudo-detail in an output image. The MSSR network consists of several basic super-resolution components (BSRCs) and multilevel supervision. The overall super-resolution scale factor of the MSSR network can be set flexibly. The BSRCs can be increased or decreased
similar to building blocks
and BSRCs decrease with decreasing overall super-resolution scale factor and increase with increasing overall super-resolution scale factor. The scale factor of each BSRC is determined by the overall scale factor and the number of BSRCs. Second
a scale-factor-adjustable and lightweight basic super-resolution component was designed to enable the construction of multilevel supervision networks with different number of BSRCs and different scale factors
such as building blocks. The BSRC consists of a multiscale feature extraction module
a global feature extraction module and an image reconstruction module. Given that the scale factor of each BSRC should have a degree of flexibility
the network structure of each BSRC is the same except for the image reconstruction module. This approach shortens the overall training time of the network. Finally
a method of dividing super-resolution overall scale factor was proposed
and the effects of different number and different scale factors of BSRCs on the performance of multilevel supervision network were explored. For the super-resolution process with a certain scale factor
we need a method to divide the overall scale factor into each BSRC. The number of supervision increases with the number of BSRCs
and the super-resolution ill-posedness is reduced. In addition
the total number of network layers and the number of computations increase. We determined the optimal number of BSRCs and their respective super-resolution scale factors by comparing the super-resolution effects of multiple BSRC combinations. Additionally
a new remote sensing dataset containing worldwide scenes was constructed for the super-resolution task in this paper.
To adequately train and test the proposed super-resolution method and existing methods
we used our datasets and two existing super-resolution datasets: the UCMerced and AID datasets. We compared our method with the state-of-the-art methods: VDSR
SRGAN
RDN
RCAN
DRN
and TransENet. The experiment results on three datasets demonstrated that our MSSR network outperformed these methods.
The progressive network and multilevel supervision structure can effectively suppress super-resolution ill-posedness and reduce pseudo-detail and detail loss in super-resolution results. By further analyzing the experimental results
we found that multilevel supervision has greater performance gains in super-resolution tasks with a larger scale factor than other methods. We speculated that the multilevel supervision network exhibit improved performance in super-resolution tasks at 4× magnification. In future research
we will explore multilevel supervision networks in super-resolution tasks at large magnifications (i.e.
5× and 8×).
Arun P V , Buddhiraju K M , Porwal A and Chanussot J . 2020 . CNN based spectral super-resolution of remote sensing images . Signal Processing , 169 : 107394 [ DOI: 10.1016/j.sigpro.2019.107394 http://dx.doi.org/10.1016/j.sigpro.2019.107394 ]
Chen H and Luo B . 2021 . Multi-angle remote sensing images super-resolution reconstruction using dynamic upsampling filter deep network . Geomatics and Information Science of Wuhan University , 46 ( 11 ): 1716 - 1726
陈行 , 罗斌 . 2021 . 利用动态上采样滤波深度网络进行多角度遥感影像超分辨率重建 . 武汉大学学报 (信息科学版) , 46 ( 11 ): 1716 - 1726 [ DOI: 10.13203/j.whugis20200651 http://dx.doi.org/10.13203/j.whugis20200651 ]
Cheng G and Han J W . 2016 . A survey on object detection in optical remote sensing images . ISPRS Journal of Photogrammetry and Remote Sensing , 117 : 11 - 28 [ DOI: 10.1016/j.isprsjprs.2016.03.014 http://dx.doi.org/10.1016/j.isprsjprs.2016.03.014 ]
Cheng G , Han J W and Lu X Q . 2017 . Remote sensing image scene classification: benchmark and state of the art . Proceedings of the IEEE , 105 ( 10 ): 1865 - 1883 [ DOI: 10.1109/JPROC.2017.2675998 http://dx.doi.org/10.1109/JPROC.2017.2675998 ]
Deng Z P , Sun H , Zhou S L , Zhao J P , Lei L and Zou H X . 2018 . Multi-scale object detection in remote sensing imagery with convolutional neural networks . ISPRS Journal of Photogrammetry and Remote Sensing , 145 : 3 - 22 [ DOI: 10.1016/j.isprsjprs.2018.04.003 http://dx.doi.org/10.1016/j.isprsjprs.2018.04.003 ]
Diakogiannis F I , Waldner F , Caccetta P and Wu C . 2020 . ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data . ISPRS Journal of Photogrammetry and Remote Sensing , 162 : 94 - 114 [ DOI: 10.1016/j.isprsjprs.2020.01.013 http://dx.doi.org/10.1016/j.isprsjprs.2020.01.013 ]
Dong C , Loy C C , He K M and Tang X O . 2014 . Learning a deep convolutional network for image super-resolution // 13th European Conference on Computer Vision . Zurich : Springer: 184 - 199 [ DOI: 10.1007/978-3-319-10593-2_13 http://dx.doi.org/10.1007/978-3-319-10593-2_13 ]
Dong C , Loy C C and Tang X O . 2016 . Accelerating the super-resolution convolutional neural network // 14th European Conference on Computer Vision . Amsterdam : Springer: 391 - 407 [ DOI: 10.1007/978-3-319-46475-6_25 http://dx.doi.org/10.1007/978-3-319-46475-6_25 ]
Gong Y F , Liao P Y , Zhang X D , Zhang L F , Chen G Z , Zhu K , Tan X L and Lv Z Y . 2021 . Enlighten-GAN for super resolution reconstruction in mid-resolution remote sensing images . Remote Sensing , 13 ( 6 ): 1104 [ DOI: 10.3390/rs13061104 http://dx.doi.org/10.3390/rs13061104 ]
Guo Y , Chen J , Wang J D , Chen Q , Cao J Z , Deng Z S , Xu Y W and Tan M K . 2020 . Closed-loop matters: dual regression networks for single image super-resolution // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE: 5406 - 5415 [ DOI: 10.1109/CVPR42600.2020.00545 http://dx.doi.org/10.1109/CVPR42600.2020.00545 ]
Hu J F , Huang T Z , Deng L J , Dou H X , Hong D F and Vivone G . 2022 . Fusformer: a transformer-based fusion network for hyperspectral image super-resolution . IEEE Geoscience and Remote Sensing Letters , 19 : 6012305 [ DOI: 10.1109/LGRS.2022.3194257 http://dx.doi.org/10.1109/LGRS.2022.3194257 ]
Im J , Park H and Takeuchi W . 2019 . Advances in remote sensing-based disaster monitoring and assessment . Remote Sensing , 11 ( 18 ): 2181 [ DOI: 10.3390/rs11182181 http://dx.doi.org/10.3390/rs11182181 ]
Jiang K , Wang Z Y , Yi P , Wang G C , Lu T and Jiang J J . 2019 . Edge-enhanced GAN for remote sensing image superresolution . IEEE Transactions on Geoscience and Remote Sensing , 57 ( 8 ): 5799 - 5812 [ DOI: 10.1109/TGRS.2019.2902431 http://dx.doi.org/10.1109/TGRS.2019.2902431 ]
Johnson J , Alahi A and Fei-Fei L . 2016 . Perceptual losses for real-time style transfer and super-resolution // 14th European Conference on Computer Vision . Amsterdam : Springer: 694 - 711 [ DOI: 10.1007/978-3-319-46475-6_43 http://dx.doi.org/10.1007/978-3-319-46475-6_43 ]
Kampffmeyer M , Salberg A B and Jenssen R . 2016 . Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Las Vegas : IEEE: 680 - 688 [ DOI: 10.1109/CVPRW.2016.90 http://dx.doi.org/10.1109/CVPRW.2016.90 ]
Kim J , Lee J K and Lee K M . 2016 . Accurate image super-resolution using very deep convolutional networks // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE: 1646 - 1654 [ DOI: 10.1109/CVPR.2016.182 http://dx.doi.org/10.1109/CVPR.2016.182 ]
Ledig C , Theis L , Huszár F , Caballero J , Cunningham A , Acosta A , Aitken A , Tejani A , Totz J , Wang Z H and Shi W Z . 2017 . Photo-realistic single image super-resolution using a generative adversarial network // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu : IEEE: 105 - 114 [ DOI: 10.1109/CVPR.2017.19 http://dx.doi.org/10.1109/CVPR.2017.19 ]
Lei S , Shi Z W and Mo W J . 2022 . Transformer-based multistage enhancement for remote sensing image super-resolution . IEEE Transactions on Geoscience and Remote Sensing , 60 : 5615611 [ DOI: 10.1109/TGRS.2021.3136190 http://dx.doi.org/10.1109/TGRS.2021.3136190 ]
Li K , Wan G , Cheng G , Meng L Q and Han J W . 2020 . Object detection in optical remote sensing images: a survey and a new benchmark . ISPRS Journal of Photogrammetry and Remote Sensing , 159 : 296 - 307 [ DOI: 10.1016/j.isprsjprs.2019.11.023 http://dx.doi.org/10.1016/j.isprsjprs.2019.11.023 ]
Liang J Y , Cao J Z , Sun G L , Zhang K , Van Gool L and Timofte R . 2021 . SwinIR: image restoration using swin transformer // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops . Montreal : IEEE: 1833 - 1844 [ DOI: 10.1109/ICCVW54120.2021.00210 http://dx.doi.org/10.1109/ICCVW54120.2021.00210 ]
Lim B , Son S , Kim H , Nah S and Lee K M . 2017 . Enhanced deep residual networks for single image super-resolution // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Honolulu : IEEE: 1132 - 1140 [ DOI: 10.1109/CVPRW.2017.151 http://dx.doi.org/10.1109/CVPRW.2017.151 ]
Liu Z , Lin Y T , Cao Y , Hu H , Wei Y X , Zhang Z , Lin S and Guo B N . 2021 . Swin transformer: hierarchical vision transformer using shifted windows // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE: 9992 - 10002 [ DOI: 10.1109/ICCV48922.2021.00986 http://dx.doi.org/10.1109/ICCV48922.2021.00986 ]
Lu X Q , Zheng X T and Yuan Y . 2017 . Remote sensing scene classification by unsupervised representation learning . IEEE Transactions on Geoscience and Remote Sensing , 55 ( 9 ): 5148 - 5157 [ DOI: 10.1109/TGRS.2017.2702596 http://dx.doi.org/10.1109/TGRS.2017.2702596 ]
Ma J Q , Liang Z T and Zhang L . 2022 . A text attention network for spatial deformation robust scene text image super-resolution // Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans : IEEE: 5901 - 5910 [ DOI: 10.1109/CVPR52688.2022.00582 http://dx.doi.org/10.1109/CVPR52688.2022.00582 ]
Mei Y Q , Fan Y C and Zhou Y Q . 2021 . Image super-resolution with non-local sparse attention // Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville : IEEE: 3516 - 3525 [ DOI: 10.1109/CVPR46437.2021.00352 http://dx.doi.org/10.1109/CVPR46437.2021.00352 ]
Shi W Z , Caballero J , Huszár F , Totz J , Aitken A P , Bishop R , Rueckert D and Wang Z H . 2016 . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE: 1874 - 1883 [ DOI: 10.1109/CVPR.2016.207 http://dx.doi.org/10.1109/CVPR.2016.207 ]
Tang X T , Yang X , Li F , Ma J and Liang L . Super-resolution reconstruction of hyper-temporal remote sensing images based on self-attention . National Remote Sensing Bulletin (唐晓天, 杨雪, 李峰, 马骏, 梁亮. 基于自注意力的超时相遥感图像超分辨率重建. 遥感学报 [DOI: 10.11834/jrs.20221825]
Tian C W , Yuan Y X , Zhang S C , Lin C W , Zuo W M and Zhang D . 2022 . Image super-resolution with an enhanced group convolutional neural network . Neural Networks , 153 : 373 - 385 [ DOI: 10.1016/j.neunet.2022.06.009 http://dx.doi.org/10.1016/j.neunet.2022.06.009 ]
Twumasi N Y D , Shao Z F and Orhan A . 2019 . Remote sensing and GIS methods in urban disaster monitoring and management–an overview . International Journal of Trend in Scientific Research and Development , 3 ( 4 ): 918 - 926 [ DOI: 10.31142/ijtsrd23976 http://dx.doi.org/10.31142/ijtsrd23976 ]
Voinov S , Schwarz E , Krause D and Berg M . 2018 . Processing framework to support maritime surveillance applications based on optical remote sensing images // Proceedings Volume 10773, Sixth International Conference on Remote Sensing and Geoinformation of the Environment . Paphos : SPIE: 268 - 273 [ DOI: 10.1117/12.2326058 http://dx.doi.org/10.1117/12.2326058 ]
Wang P , Yao H Y and Zhang G . 2021 . Super-resolution flood inundation mapping for multispectral image based on super-pixel scale . National Remote Sensing Bulletin , 25 ( 2 ): 641 - 652
王鹏 , 姚红雨 , 张弓 . 2021 . 超像元尺度下的多光谱图像超分辨率洪水淹没制图 . 遥感学报 , 25 ( 2 ): 641 - 652 [ DOI: 10.11834/jrs.20210199 http://dx.doi.org/10.11834/jrs.20210199 ]
Wang Q , Zhang X D , Chen G Z , Dai F , Gong Y F and Zhu K . 2018 . Change detection based on Faster R-CNN for high-resolution remote sensing images . Remote Sensing Letters , 9 ( 10 ): 923 - 932 [ DOI: 10.1080/2150704X.2018.1492172 http://dx.doi.org/10.1080/2150704X.2018.1492172 ]
Wang T F , Xie J X , Sun W X , Yan Q and Chen Q F . 2021 . Dual-camera super-resolution with aligned attention modules // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE: 1981 - 1990 [ DOI: 10.1109/ICCV48922.2021.00201 http://dx.doi.org/10.1109/ICCV48922.2021.00201 ]
Xia G S , Hu J W , Hu F , Shi B G , Bai X , Zhong Y F , Zhang L P and Lu X Q . 2017 . AID: a benchmark data set for performance evaluation of aerial scene classification . IEEE Transactions on Geoscience and Remote Sensing , 55 ( 7 ): 3965 - 3981 [ DOI: 10.1109/TGRS.2017.2685945 http://dx.doi.org/10.1109/TGRS.2017.2685945 ]
Xie J , He N J , Fang L Y and Plaza A . 2019 . Scale-free convolutional neural network for remote sensing scene classification . IEEE Transactions on Geoscience and Remote Sensing , 57 ( 9 ): 6916 - 6928 [ DOI: 10.1109/TGRS.2019.2909695 http://dx.doi.org/10.1109/TGRS.2019.2909695 ]
Xu Y Y , Luo W , Hu A N , Xie Z , Xie X J and Tao L F . 2022 . TE-SAGAN: an improved generative adversarial network for remote sensing super-resolution images . Remote Sensing , 14 ( 10 ): 2425 [ DOI: 10.3390/rs14102425 http://dx.doi.org/10.3390/rs14102425 ]
Yang Y and Newsam S . 2010 . Bag-of-visual-words and spatial extensions for land-use classification // Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems . San Jose : ACM: 270 - 279 [ DOI: 10.1145/1869790.1869829 http://dx.doi.org/10.1145/1869790.1869829 ]
Ye P . 2022 . Remote sensing approaches for meteorological disaster monitoring: recent achievements and new challenges . International Journal of Environmental Research and Public Health , 19 ( 6 ): 3701 [ DOI: 10.3390/ijerph19063701 http://dx.doi.org/10.3390/ijerph19063701 ]
Yuan X H , Shi J F and Gu L C . 2021 . A review of deep learning methods for semantic segmentation of remote sensing imagery . Expert Systems with Applications , 169 : 114417 [ DOI: 10.1016/j.eswa.2020.114417 http://dx.doi.org/10.1016/j.eswa.2020.114417 ]
Zhang K , Liang J Y , Van Gool L and Timofte R . 2021 . Designing a practical degradation model for deep blind image super-resolution // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision . Montreal : IEEE: 4771 - 4780 [ DOI: 10.1109/ICCV48922.2021.00475 http://dx.doi.org/10.1109/ICCV48922.2021.00475 ]
Zhang Y , Tian Y , Kong Y , Zhong B and Fu Y . 2018a . Residual dense network for image super-resolution . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition : 2472 - 2481
Zhang S M , Wu R Z , Xu K Y , Wang J M and Sun W W . 2019 . R-CNN-based ship detection from high resolution remote sensing imagery . Remote Sensing , 11 ( 6 ): 631 [ DOI: 10.3390/rs11060631 http://dx.doi.org/10.3390/rs11060631 ]
Zhang Y , Li K , Li K , Wang L , Zhong B and Fu Y . 2018b . Image super-resolution using very deep residual channel attention networks/ /Proceedings of the European conference on computer vision (ECCV). 286 - 301
Zhen J N , Jiang X P , Zhao D M , Wang J J , Miao J and Wu G F . 2022 . Retrieving canopy nitrogen content of mangrove forests from Sentinel-2 super-resolution reconstruction data . National Remote Sensing Bulletin , 26 ( 6 ): 1206 - 1219
甄佳宁 , 蒋侠朋 , 赵德梅 , 王俊杰 , 苗菁 , 邬国锋 . 2022 . 利用Sentinel-2影像超分辨率重建的红树林冠层氮含量反演 . 遥感学报 , 26 ( 6 ): 1206 - 1219 [ DOI: 10.11834/jrs.20221461 http://dx.doi.org/10.11834/jrs.20221461 ]
相关文章
相关作者
相关机构
京公网安备11010802024621