Multi-task Learning for Unsupervised Domain Adaptive Semantic Segmentation of Remote Sensing Images

Wang Yu; Feng Yuting; Gong Sishi; Mao Yanqin; Li Shengwen; Fang Fang; Zhou Shunping

doi:10.11834/jrs.20254411

Views : 0 下载量: 0 CSCD: 0

R-PDF
PDF
Export
Share
Collection
Album

Multi-task Learning for Unsupervised Domain Adaptive Semantic Segmentation of Remote Sensing Images
Pages: 1-18(2025)
Received：23 September 2024，

Published Online：28 October 2025，
DOI： 10.11834/jrs.20254411
稿件说明：

移动端阅览

王渝，冯雨婷，龚思诗，毛彦琴，李圣文，方芳，周顺平.XXXX.多任务协同学习的无监督域自适应遥感图像语义分割.遥感学报，XX（XX）： 1-18 DOI： 10.11834/jrs.20254411.

Wang Yu，Feng Yuting，Gong Sishi，Mao Yanqin，Li Shengwen，Fang Fang，Zhou Shunping. XXXX. Multi-task Learning for Unsupervised Domain Adaptive Semantic Segmentation of Remote Sensing Images. National Remote Sensing Bulletin， XX（XX）：1-18 DOI： 10.11834/jrs.20254411.

摘要

遥感图像语义分割在土地覆盖与利用分类、城市规划以及变化检测等领域具有重要作用。作为一种极具潜力的无监督学习方法，域自适应技术大大推动了遥感图像语义分割的发展。然而，现有模型尚基于单任务来学习，学习到的分割特征不够充分，导致在分割过程中难以准确识别遥感图像中的复杂区域。为了解决这一问题，本文提出了一种多任务学习域自适应语义分割网络（MTLDANet），该网络通过协同学习遥感图像中的语义信息与高程信息，来提升分割特征的学习能力。具体来说，该方法将任务特定的语义特征和高程特征输入跨任务特征关联学习模块，挖掘任务之间的潜在相关性，从而获得更强的任务特定特征表达。通过伪标签引导的混合一致性学习模块提升伪标签质量，实现全局域对齐。此外，熵引导的类别级对齐进一步增强了难分类类别的区分性。本文基于ISPRS 2D和US3D数据集进行了四组跨场景遥感图像语义分割实验。结果表明，所提方法在多种复杂跨域场景下均显著优于现有的域自适应方法。

Abstract

Objective Semantic segmentation of remote sensing images (RSIs) plays a crucial role in land cover and land use classification

urban planning

and change detection. As a highly promising unsupervised learning method

domain adaptation has significantly accelerated the advancement of RSI segmentation. However

current models often rely on the limited feature learning capability of single-task approaches

making it difficult to accurately distinguish hard-to-classify regions in RSIs. To address this issue

a multi-task learning domain adaptive network (MTLDANet) is proposed

which jointly learns semantic and elevation information in RSIs

improving segmentation performance.Method The method feeds task-specific semantic and elevation features into a cross-task feature correlation learning module to explore latent correlations between tasks

thereby enhancing task-specific feature representations. A hybrid consistency learning module

guided by pseudo-labels

is employed to improve pseudo-label quality and achieve global domain alignment. Additionally

entropy-guided category-level alignment enhances the separability of challenging categories.Result The proposed method is evaluated on four cross-scene RSI segmentation experiments using the ISPRS 2D and US3D datasets.Conclusion Results show that the method outperforms existing domain adaptation approaches

demonstrating significant advantages in various complex cross-domain scenarios.

关键词

Keywords

references

Bhattacharjee D , Zhang T , Süsstrunk S and Salzmann M . Mult: An end-to-end multitask learning transformer [C ] . Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022 : 12031 - 12041 [ DOI: 10.1109/CVPR52688.2022.01172 http://dx.doi.org/10.1109/CVPR52688.2022.01172 ]

Chang W L , Wang H P , Peng W H and Chiu W C . All about structure: Adapting structural information across domains for boosting semantic segmentation [C ] . Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2019 : 1900 - 1909 [ DOI: 10.1109/CVPR.2019.00200 http://dx.doi.org/10.1109/CVPR.2019.00200 ]

Chen L C , Papandreou G , Kokkinos I , Murphy K and Yuille A L . Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs [J ] . IEEE transactions on pattern analysis and machine intelligence , 2017 , 40 ( 4 ): 834 - 848 [ DOI: 10.1109/TPAMI.2017.2699184 http://dx.doi.org/10.1109/TPAMI.2017.2699184 ]

Cipolla R , Gal Y and Kendall A . Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics [C ] . Proceedings of the IEEE conference on computer vision and pattern recognition . 2018 : 7482 - 7491 [ DOI: 10.1109/CVPR.2018.00781 http://dx.doi.org/10.1109/CVPR.2018.00781 ]

Gao L , Zhang L and Zhang Q . Addressing Domain Gap via Content Invariant Representation for Semantic Segmentation [C ] . Proceedings of the AAAI Conference on Artificial Intelligence : Vol . 35 . 2021 : 7528 - 7536 [ DOI: https://doi.org/10.1609/aaai.v35i9.16922 https://doi.org/10.1609/aaai.v35i9.16922 ]

Hu J , Shen L and Sun G . Squeeze-and-excitation networks [C ] . Proceedings of the IEEE conference on computer vision and pattern recognition . 2018 : 7132 - 7141 [ DOI: 10.1109/CVPR.2018.00745 http://dx.doi.org/10.1109/CVPR.2018.00745 ]

Kundu J N , Lakkakula N and Babu R V . Um-adapt: Unsupervised multi-task adaptation using adversarial cross-task distillation [C ] . Proceedings of the IEEE/CVF International Conference on Computer Vision . 2019 : 1436 - 1445 [ DOI: 10.1109/ICCV.2019.00152 http://dx.doi.org/10.1109/ICCV.2019.00152 ]

Kussul N , Lavreniuk M , Skakun S and Shelestov A . Deep learning classification of land cover and crop types using remote sensing data [J ] . IEEE Geoscience and Remote Sensing Letters , 2017 , 14 ( 5 ): 778 - 782 [ DOI: 10.1109/LGRS.2017.2681128 http://dx.doi.org/10.1109/LGRS.2017.2681128 ]

Le Saux B , Yokoya N , Hänsch R and Brown M . Data Fusion Contest 2019 (DFC2019) [M/OL ] . IEEE Dataport , 2019 . [ DOI: https://dx.doi.org/10.21227/c6tm-vw12 http://dx.doi.org/https://dx.doi.org/10.21227/c6tm-vw12 ]

Li R , Zheng S , Zhang C , Duan C and Wang L . ABCNet: Attentive bilateral contextual network for efficient semantic segmentation of Fine-Resolution remotely sensed imagery [J ] . ISPRS journal of photogrammetry and remote sensing , 2021 , 181 : 84 - 98 [ DOI: https://doi.org/10.1016/j.isprsjprs.2021.09.005 https://doi.org/10.1016/j.isprsjprs.2021.09.005 ]

Li Y , Shi T , Zhang Y , Chen W and Wang Z . Learning deep semantic segmentation network under multiple weakly-supervised constraints for cross-domain remote sensing image semantic segmentation [J ] . ISPRS Journal of Photogrammetry and Remote Sensing , 2021 , 175 : 20 - 33 [ DOI: https://doi.org/10.1016/j.isprsjprs.2021.02.009 https://doi.org/10.1016/j.isprsjprs.2021.02.009 ]

Long J , Shelhamer E and Darrell T . Fully convolutional networks for semantic segmentation [C ] . Proceedings of the IEEE conference on computer vision and pattern recognition . 2015 : 3431 - 3440 [ DOI: 10.1109/CVPR.2015.7298965 http://dx.doi.org/10.1109/CVPR.2015.7298965 ]

Luo M and Ji S . Cross-spatiotemporal land-cover classification from VHR remote sensing images with deep learning-based domain adaptation [J ] . ISPRS Journal of Photogrammetry and Remote Sensing , 2022 , 191 : 105 - 128 . [ DOI: https://doi.org/10.1016/j.isprsjprs.2022.07.011 https://doi.org/10.1016/j.isprsjprs.2022.07.011 ]

Maggiori E , Tarabalka Y , Charpiat , G and Alliez P . ( 2017 ). Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification . IEEE Transactions on Geoscience and Remote Sensing , 55 ( 2 ), 645 - 657 . [ DOI: 10.1109/TGRS.2016.2612821 http://dx.doi.org/10.1109/TGRS.2016.2612821 ]

Olsson V , Tranheden W , Pinto J and Svensson L . Classmix: Segmentation-based data augmentation for semi-supervised learning [C ] . Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . 2021 : 1369 - 1378[DOI: 10.1109/WACV48630.2021.00141] .

Rottensteiner F , Sohn G , Jung J , Gerke M , Baillard C , Benitez S and Breitkopf U . The ISPRS benchmark on urban object classification and 3D building reconstruction [J ] . ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences ; I-3, 2012 , 1 ( 1 ): 293 - 298 [ DOI: https://doi.org/10.5194/isprsannals-I-3-293-2012 https://doi.org/10.5194/isprsannals-I-3-293-2012 ]

Saha S , Obukhov A , Paudel D P , Kanakis M , Chen Y ; Georgoulis S and Gool L V . Learning to relate depth and semantics for unsupervised domain adaptation [C ] . Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2021 : 8197 - 8207 [ DOI: 10.1109/CVPR46437.2021.00810 http://dx.doi.org/10.1109/CVPR46437.2021.00810 ]

Saporta A , Vu T H , Cord M and Pérez P . Esl: Entropy-guided self-supervised learning for domain adaptation in semantic segmentation [J ] . arXiv preprint arXiv:2006. 08658 , 2020 [ DOI: https://doi.org/10.48550/arXiv.2006.08658 https://doi.org/10.48550/arXiv.2006.08658 ]

Song X , Zhao X , Fang L , Hu H and Yu Y . Edgestereo: An effective multi-task learning network for stereo matching and edge detection [J ] . International Journal of Computer Vision , 2020 , 128 : 910 - 930 [ DOI: https://doi.org/10.1007/s11263-019-01287-w https://doi.org/10.1007/s11263-019-01287-w ]

Vu T H , Jain H , Bucher M , Cord M and Pérez P . ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation [C ] . Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2019 : 2517 - 2526[DOI: 10.1109/CVPR.2019.00262] .

Vu T H , Jain H , Bucher M , Cord M and Pérez P . Dada: Depth-aware domain adaptation in semantic segmentation [C ] . Proceedings of the IEEE/CVF International Conference on Computer Vision . 2019 : 7364 - 7373 [ DOI: 10.1109/ICCV.2019.00746 http://dx.doi.org/10.1109/ICCV.2019.00746 ]

Wang Q , Dai D , Hoyer L , Van Gool L and Fink O . Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation [C ] . Proceedings of the IEEE/CVF International Conference on Computer Vision . 2021 : 8515 - 8525 [ DOI: 10.1109/ICCV48922.2021.00840 http://dx.doi.org/10.1109/ICCV48922.2021.00840 ]

Woo S , Park J , Lee J Y and Kweon I S . Cbam: Convolutional block attention module [C ] . Proceedings of the European conference on computer vision (ECCV) . 2018 : 3 - 19 [ DOI: https://doi.org/10.7717/peerj-cs.2100 https://doi.org/10.7717/peerj-cs.2100 ]

Xu H , Yang M , Deng L , Qian Y and Wang C . Neutral Cross-Entropy Loss Based Unsupervised Domain Adaptation for Semantic Segmentation [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 4516 - 4525 [ DOI: 10.1109/TIP.2021.3073285 http://dx.doi.org/10.1109/TIP.2021.3073285 ]

Xu Q , Yuan X and Ouyang C . Class-Aware Domain Adaptation for Semantic Segmentation of Remote Sensing Images [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2022 , 60 [ DOI: 10.1109/TGRS.2020.3031926 http://dx.doi.org/10.1109/TGRS.2020.3031926 ]

Yan L , Fan B , Xiang S and Pan C . CMT: Cross Mean Teacher Unsupervised Domain Adaptation for VHR Image Semantic Segmentation [J ] . IEEE Geoscience and Remote Sensing Letters , 2021 , 19 : 1 - 5 [ DOI: 10.1109/LGRS.2021.3065982 http://dx.doi.org/10.1109/LGRS.2021.3065982 ]

Yao X , Wang Y , Wu Y and Liang Z . Weakly-supervised domain adaptation with adversarial entropy for building segmentation in cross-domain aerial imagery [J ] . IEEE Journal of Selected

Topics in Applied Earth Observations and Remote Sensing , 2021 , 14 : 8407 - 8418 [ DOI: 10.1109/JSTARS.2021.3105421 http://dx.doi.org/10.1109/JSTARS.2021.3105421 ]

Yi Z , Zhang H , Tan P and Gong M . Dualgan: Unsupervised dual learning for image-to-image translation [C ] . Proceedings of the IEEE international conference on computer vision . 2017 : 2849 - 2857 [ DOI: 10.1109/ICCV.2017.310 http://dx.doi.org/10.1109/ICCV.2017.310 ]

Zhang B , Chen T and Wang B . Curriculum-style local-to-global adaptation for cross-domain remote sensing image segmentation [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2021 , 60 : 1 - 12 [ DOI: 10.1109/TGRS.2021.3117851 http://dx.doi.org/10.1109/TGRS.2021.3117851 ]

Zhang J , Lin S , Ding L and Bruzzone L . Multi-scale context aggregation for semantic segmentation of remote sensing images [J ] . Remote Sensing , 2020 , 12 ( 4 ): 701 [ DOI: https://doi.org/10.3390/rs12040701 https://doi.org/10.3390/rs12040701 ]

Zheng A. , Wang M. , Li C. , Tang J . and Luo, B ., 2021 . Entropy guided adversarial domain adaptation for aerial image semantic segmentation. IEEE Transactions on Geoscience and Remote Sensing , 60 , pp. 1 - 14 [ DOI: 10.1109/TGRS.2021.3113581 http://dx.doi.org/10.1109/TGRS.2021.3113581 ]

Zhu J Y , Park T , Isola P and Efros A A . Unpaired image-to-image translation using cycle-consistent adversarial networks [C ] . Proceedings of the IEEE international conference on computer vision . 2017 : 2223 - 2232 [ DOI: 10.1109/ICCV.2017.244 http://dx.doi.org/10.1109/ICCV.2017.244 ]

Zhu X X , Tuia D , Mou L , Xia G S ， Zhang L P , Feng X and Fraundorfer . Deep learning in remote sensing: A comprehensive review and list of resources [J ] . IEEE geoscience and remote sensing magazine , 2017 , 5 ( 4 ): 8 - 36 [ DOI: 10.1109/MGRS.2017.2762307 http://dx.doi.org/10.1109/MGRS.2017.2762307 ]

Zhu Y , Liang Z , Yan J , Chen G and Wang X Q . ED-Net: Automatic building extraction from high-resolution aerial images with boundary information [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2021 , 14 : 4595 - 4606 [ DOI: 10.1109/JSTARS.2021.3073994 http://dx.doi.org/10.1109/JSTARS.2021.3073994 ]

Alert me when the article has been cited

提交

Dual-branch remote sensing shadow detection with the fusion of Transformer and CNN

Change detection for high-resolution remote sensing images with multi-scale feature transformer

Mapping typical floodplain wetlands using an improved D-UNet model

Edge-perception enhanced segmentation method for high-resolution remote sensing image

Dual-task model for ground crack detection in the goaf of coal mines