Multi-Prior Driven Weakly-Supervised Instance Segmentation in Remote Sensing Images
- Pages: 1-15(2024)
Published Online： 12 March 2024
DOI： 10.11834/jrs.20243407
扫 描 看 全 文
浏览全部资源
扫码关注微信
- About Journal
- Journal Online
- Special Source
摘要
遥感图像的实例分割能够同时实现感兴趣物体的目标级定位与像素级分类，是一项重要且极具挑战性的任务。当前大多遥感图像实例分割方法依赖精细的像素级注释，其制作成本高昂。此外，遥感图像混杂的前背景和复杂的目标轮廓同样增加分割的难度。为了迎接这些挑战，本文构建出一套适用于遥感图像弱监督实例分割任务的先验信息驱动体系，并提出一种基于多重先验驱动的遥感图像弱监督实例分割网络。具体地，将遥感图像弱监督实例分割任务中的先验信息按照其来源分为任务先验和图像先验，其中任务先验源自与实例分割紧密相关的边界框检测任务，图像先验来源于对图像本身信息的归纳与挖掘。进一步地，设计框-掩码投影一致性约束、像素区分难度表征函数和中心位置先验约束三个具体组件实例化任务先验信息，驱动网络确定掩码的尺寸并使其充分关注图像中的重点像素和区域；设计邻域视觉一致性约束和梯度一致性约束两个组件构建图像先验信息，使网络高效地进行前背景区分且适应遥感图像中复杂的目标轮廓。在光学和SAR遥感图像数据集上的实验结果表明，所提方法无需任何像素级注释即可实现52.5和54.1的AP值，优于现有的弱监督分割方法，且达到全监督Mask R-CNN的89.3%和84.3%。该方法能够为遥感图像的细粒度解译提供一种高性能且低成本的解决方案。
Abstract
Objective Remote sensing image interpretation has essential application values in various fields
such as urban management
maritime monitoring
and resource planning. As an important and challenging task in remote sensing image interpretation
instance segmentation of remote sensing images can achieve target-level localization and pixel-level classification of objects of interest with fine granularity
making it a current research hotspot. However
most existing remote sensing image instance segmentation methods adopt the fully supervised paradigm and require expensive pixel-level labels. Moreover
remote sensing images often have issues such as mixed foreground and background and complex target contours
making segmentation challenging.Method To overcome these challenges
we propose a prior information-driven system suitable for weakly supervised instance segmentation tasks in remote sensing images and a multi-prior driven weakly supervised instance segmentation network (MPD-WSIS-Net) to address these challenges. The prior information of weakly supervised instance segmentation can be divided into task prior and image prior. The task prior mainly comes from the bounding box detection task
which is closely related to instance segmentation. Specifically
the paper obtains prior information from three components: box-mask projection consistency constraint
pixel discrimination difficulty representation function
and center position prior constraint. The image prior comes from summarizing or excavating information about the image itself. In this study
we focus on the relationships between neighboring pixels in the image and the gradient information of targets. By integrating these constraints and the pixel discrimination difficulty representation function
we establish a complete prior information driving system to effectively enable MPD-WSIS-Net to perform instance segmentation tasks under weakly supervised conditions in remote sensing images.
Result
2
MPD-WSIS-Net was compared with weakly supervised methods
hybrid supervised methods
and fully supervised methods on optical and SAR remote sensing image datasets. Compared to weakly supervised methods
MPD-WSIS-Net achieved better segmentation results. Compared to hybrid supervised methods
the segmentation performance of MPD-WSIS-Net on both optical and SAR remote sensing image datasets significantly surpasses that of Mask R-CNN and CondInst of 50% pixel-level annotations. It is also competitive compared to Mask R-CNN and CondInst under 75% pixel-level annotation conditions. Compared to fully supervised methods trained with pixel-level labels
MPD-WSIS-Net can achieve 89.3% of fully supervised Mask R-CNN's AP value on optical and 84.3% on SAR remote sensing image datasets. Furthermore
we have demonstrated the positive impact of each prior information component in MPD-WSIS-Net on instance segmentation performance in optical remote sensing images through ablation experiments.Conclusion This study constructs a prior information-driven system consisting of task and image priors through a detailed analysis of the prior information in weakly supervised instance segmentation tasks. The specification of prior information is achieved through the box-mask projection consistency constraint
pixel discrimination difficulty representation function
center position prior constraint
neighborhood visual consistency constraint
and gradient consistency constraint. This research can enable MPD-WSIS-Net to complete instance segmentation tasks without pixel-level annotations and provide high-performance and low-cost prescription for fine-grained interpretation of optical and SAR remote sensing images.
关键词
遥感图像实例分割细粒度解译弱监督学习先验信息驱动体系目标轮廓注释成本
Keywords
remote sensing imageinstance segmentationfine-grained interpretationweakly supervised learningpriori informationdriven systemtarget contourannotation cost
references
Ahn J, Cho S H Y and Kwak S. 2019. Weakly supervised learning of instance segmentation with inter-pixel relations//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE: 2204–2213 [DOI: 10.1109/CVPR.2019.00231http://dx.doi.org/10.1109/CVPR.2019.00231]
Arun A, Jawahar C V and Kumar M P. 2020. Weakly supervised instance segmentation by learning annotation consistent instances//2020 European Conference on Computer Vision (ECCV). Glasgow, UK: Springer: 254–270 [DOI: 10.1007/978-3-030-58604-1_16http://dx.doi.org/10.1007/978-3-030-58604-1_16]
Bolya D, Zhou C, Xiao F Y and Lee Y J. 2019. YOLACT: real-time instance segmentation//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE: 9156–9165 [DOI: 10.1109/ICCV.2019.00925http://dx.doi.org/10.1109/ICCV.2019.00925]
Cai Z W and Vasconcelos N. 2018. Cascade R-CNN: delving into high quality object detection//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, UT, USA: IEEE: 6154–6162 [DOI: 10.1109/CVPR.2018.00644http://dx.doi.org/10.1109/CVPR.2018.00644]
Chen M, Zhang Y. Chen E P. Hu Y H. Xie Y F and Pan Z S. 2023. Meta-knowledge guided weakly supervised instance segmentation for optical and SAR image interpretation. Remote Sensing, 15: 2357 [DOI: 10.3390/rs15092357http://dx.doi.org/10.3390/rs15092357]
Chen Z Q. Shang Y H. Python A. Cai Y X and Yin J W. 2022. DB-BlendMask: decomposed attention and balanced BlendMask for instance segmentation of high-resolution remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 60: 1–15 [DOI: 10.1109/TGRS.2021.3138913http://dx.doi.org/10.1109/TGRS.2021.3138913]
Cheng G. Zhou P C and Han J W. 2016. Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 54: 7405–7415 [DOI: 10.1109/TGRS.2016.2601622http://dx.doi.org/10.1109/TGRS.2016.2601622]
Hao S Y. Wang G A and Gu R S. 2021. Weakly supervised instance segmentation using multi-prior fusion. Computer Vision and Image Understanding, 211: 103261 [DOI: 10.1016/j.cviu.2021.103261http://dx.doi.org/10.1016/j.cviu.2021.103261]
He K M. Gkioxari G. Dollar P and Girshick R. 2020. Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42: 386–397 [DOI: 10.1109/TPAMI.2018.2844175http://dx.doi.org/10.1109/TPAMI.2018.2844175]
He K M. Zhang X Y. Ren S Q and Sun J. 2016. Deep residual learning for image recognition//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE: 770–778 [DOI: 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90]
Hsu C C. Hsu K J. Tsai C C. Lin Y Y and Chuang Y Y. 2019. Weakly supervised instance segmentation using the bounding box tightness prior//2019 Advances in Neural Information Processing Systems (NeurIPS). Vancouver, BC, Canada: Curran Associates Inc: 6586–6597 [DOI: 10.5555/3454287.3454878http://dx.doi.org/10.5555/3454287.3454878]
Huang L. Han X Y. Wang X L. Zhang Y D. Yang J F. Fen A P. Li J G and Zhu N H N W. 2022. Coupling with high-resolution remote sensing data to evaluate urban non-point source pollution in Tongzhou, China. Science of The Total Environment. 831: 154632 [DOI: 10.1016/j.scitotenv.2022.154632http://dx.doi.org/10.1016/j.scitotenv.2022.154632]
Lan S Y. Yu Z D. Choy C. Radhakrishnan S. Liu G L. Zhu Y K. Davis L S and Anandkumar A. 2021. DiscoBox: weakly supervised instance segmentation and semantic correspondence from box supervision//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada: IEEE: 3386–3396 [DOI: 10.1109/ICCV48922.2021.00339http://dx.doi.org/10.1109/ICCV48922.2021.00339]
Lee J. Yi J. Shin C and Yoon S. 2021. BBAM: bounding box attribution map for weakly supervised semantic and instance segmentation//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA: IEEE 2643–2651 [DOI: 10.1109/CVPR46437.2021.00267http://dx.doi.org/10.1109/CVPR46437.2021.00267]
Li M Q. Fang S B. Zhu Y C. Wu Y J. Cao Y. Zhuo W and E Y H. 2022. Spatial and temporal distributions of waterlogging disasters in the summer of 2021 in Mainland China and their possible impacts. National Remote Sensing Bulletin, 26(9): 1886–1894
李梦倩, 房世波, 朱永超, 武英洁, 曹云, 卓文, 俄有浩. 2022. 2021年夏季中国大陆涝渍灾害时空分布分析. 遥感学报, 26(9): 1886-1894 [DOI: 10.11834/jrs.20221782http://dx.doi.org/10.11834/jrs.20221782]
Li Y H. Xue Y. Li L F. Zhang X J and Qian X M. 2022. Domain adaptive box-supervised instance segmentation network for mitosis detection. IEEE Transactions on Medical Imaging, 41: 2469–2485 [DOI: 10.1109/TMI.2022.3165518http://dx.doi.org/10.1109/TMI.2022.3165518]
Li Z L. Wang L Y. Jiang S. Wu Y H and Zhang Q J. 2021. On orbit extraction method of ship target in SAR images based on ultra-lightweight network. National Remote Sensing Bulletin, 25(3): 765–775
李宗凌, 汪路元, 蒋帅, 吴雨航, 张庆君. 2021. 超轻量网络的SAR图像舰船目标在轨提取. 遥感学报, 25(3): 765-775 [DOI: 10.11834/jrs.20210160http://dx.doi.org/10.11834/jrs.20210160]
Lin T Y. Dollar P. Girshick R. He K M. Hariharan B and Belongie S. 2017. Feature pyramid networks for object detection//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI: IEEE: 936–944 [DOI: 10.1109/CVPR.2017.106http://dx.doi.org/10.1109/CVPR.2017.106]
Lin T Y. Maire M. Belongie S. Hays J. Perona P. Ramanan D. Dollár P and Zitnick C L. 2014. Microsoft COCO: common objects in context//2014 European Conference on Computer Vision (ECCV). Zurich, Switzerland: Springer: 740–755 [DOI: 10.1007/978-3-319-10602-1_48http://dx.doi.org/10.1007/978-3-319-10602-1_48]
Long J. Shelhamer E and Darrell T. 2015. Fully convolutional networks for semantic segmentation//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE: 3431–3440. [DOI: 10.1109/CVPR.2015.7298965http://dx.doi.org/10.1109/CVPR.2015.7298965]
Lyu J. Wang S Q. Sun D Y. Nie J W. Jiao H B. Zhang H L and Liang H W. 2022. Remote sensing of spatial and temporal variations of euphotic zone depth in Bohai Sea and Yellow Sea during recent 20 years (2002—2020). National Remote Sensing Bulletin, 26(12): 2507-2517
吕军, 王胜强, 孙德勇, 聂君伟, 焦红波, 张海龙, 梁涵玮. 2023. 近20年（2002—2020年）渤黄海真光层深度的时空变化遥感分析. 遥感学报, 26(12): 2507-2517 [DOI: 10.11834/jrs.20210414http://dx.doi.org/10.11834/jrs.20210414]
Lyu S Y. Li J T. A X H. Yang C. Yang R C and Shang X M. 2023. Res_ASPP_UNet++： Building an extraction network from remote sensing imagery combining depthwise separable convolution with atrous spatial pyramid pooling. National Remote Sensing Bulletin, 27(2): 502-519
吕少云, 李佳田, 阿晓荟, 杨超, 杨汝春, 尚晓梅 .2023. Res_ASPP_UNet++：结合分离卷积与空洞金字塔的遥感影像建 筑物提取网络. 遥感学报, 27(2): 502-519 [DOI: 10.11834/jrs.20210477http://dx.doi.org/10.11834/jrs.20210477]
Ma T Q. Wang Q L. Zhang H Z and Zuo W M. 2022. Delving deeper into pixel prior for box-supervised semantic segmentation. IEEE Transactions on Image Process, 31: 1406–1417 [DOI: 10.1109/TIP.2022.3141878http://dx.doi.org/10.1109/TIP.2022.3141878]
Miao W. Geng J and Jiang W. 2022. Semi-supervised remote-sensing image scene classification using representation consistency siamese network. IEEE Transactions on Geoscience and Remote Sensing, 60: 1–14 [DOI: 10.1109/TGRS.2022.3140485http://dx.doi.org/10.1109/TGRS.2022.3140485]
Milletari F. Navab N and Ahmadi S A. 2016. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation//2016 Fourth International Conference on 3D Vision (3DV). Stanford, CA, USA: IEEE: 565–571 [DOI: 10.1109/3DV.2016.79http://dx.doi.org/10.1109/3DV.2016.79]
Pont-Tuset J. Arbelaez P. Barron J T. Marques F and Malik J. 2017. Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39: 128–140. [DOI: 10.1109/TPAMI.2016.2537320http://dx.doi.org/10.1109/TPAMI.2016.2537320]
Rother C. Kolmogorov V and Blake A. 2004. “GrabCut”: interactive foreground extraction using iterated graph cuts. ACM Transactions Graphics, 23: 309–314 [DOI: 10.1145/1015706.1015720http://dx.doi.org/10.1145/1015706.1015720]
Su H. Wei S J. Liu S. Liang J D. Wang C. Shi J and Zhang X L. 2020. HQ-ISNet: high-quality instance segmentation for remote sensing imagery. Remote Sensing, 12: 989 [DOI: 10.3390/rs12060989http://dx.doi.org/10.3390/rs12060989]
Tian Z. Shen C H and Chen H. 2020. Conditional convolutions for instance segmentation//2020 European Conference on Computer Vision (ECCV), 16th European Conference. Glasgow, UK: Springer: 254–270 [DOI: 10.1007/978-3-030-58580-7_16http://dx.doi.org/10.1007/978-3-030-58580-7_16]
Tian Z. Shen C H. Chen H and He T. 2019. FCOS: fully convolutional one-stage object detection//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE: 9626 [DOI: 10.1109/ICCV.2019.00969]
Tian Z. Shen C H. Wang X L and Chen H. 2021. BoxInst: high-performance instance segmentation with box annotations//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA: IEEE: 5439–5448 [DOI: 10.1109/CVPR46437.2021.00540http://dx.doi.org/10.1109/CVPR46437.2021.00540]
Wang X L. Liang Z Y and Liu T. 2023. Feature attention pyramid-based remote sensing image object detection method. National Remote Sensing Bulletin, 27(2): 492–501
汪西莉, 梁正印, 刘涛. 2023. 基于特征注意力金字塔的遥感图像目标检测方法. 遥感学报, 27(2): 492-501 [DOI: 10.11834/jrs.20235011http://dx.doi.org/10.11834/jrs.20235011]
Waqas Z S. Arora A. Gupta A. Khan S. Sun G. Shahbaz K F. Zhu F.Shao F. Xia G S and Bai X. 2019. iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images. arXiv:1905.12886. [DOI: 10.48550/arXiv.1905.12886http://dx.doi.org/10.48550/arXiv.1905.12886]
Wei S J. Zeng X F. Zhang H. Zhou Z C. Shi J and Zhang X L. 2022. LFG-Net: low-level feature guided network for precise ship instance segmentation in SAR images. IEEE Transactions on Geoscience and Remote Sensing, 60: 1–17 [DOI: 10.1109/TGRS.2022.3188677http://dx.doi.org/10.1109/TGRS.2022.3188677]
Wu Q Q. Wang S. Wang B and Wu Y L. 2022. Road extraction method of high-resolution remote sensing image on the basis of the spatial information perception semantic segmentation model. National Remote Sensing Bulletin, 26: 1872–1885
吴强强, 王帅, 王彪, 吴艳兰. 2022. 空间信息感知语义分割模型的高分辨率遥感影像道路提取. 遥感学报, 26(9): 1872-1885 [DOI: 10.11834/jrs.20210021http://dx.doi.org/10.11834/jrs.20210021]
Yang F. Yuan X Y. Ran J. Shu W Q. Zhao Y. Qin A Y and Gao C Q. 2021. Accurate instance segmentation for remote sensing images via adaptive and dynamic feature learning. Remote Sensing, 13: 4774 [DOI: 10.3390/rs13234774http://dx.doi.org/10.3390/rs13234774]
Zhang T. Yang X G. Lu X Q. Lu R T and Zhang S X. 2022. Ship detection in remote sensing image based on dense RFB and LSTM. National Remote Sensing Bulletin, 26(9): 1859-1871
张涛, 杨小冈, 卢孝强, 卢瑞涛, 张胜修. 2022. Dense RFB和LSTM遥感图像舰船目标检测. 遥感学报, 26(9): 1859-1871 [DOI: 10.11834/jrs.20211042http://dx.doi.org/10.11834/jrs.20211042]
Zhang T Y. Zhang X R. Zhu P. Tang X. Li C. Jiao L C and Zhou H Y. 2022. Semantic attention and scale complementary network for instance segmentation in remote sensing images. IEEE Transactions on Cybernetics, 52: 10999–11013 [DOI: 10.1109/TCYB.2021.3096185http://dx.doi.org/10.1109/TCYB.2021.3096185]
Zhao D P. Zhu C B. Qi J. Qi X H. Su Z H and Shi Z W. 2021. Synergistic attention for ship instance segmentation in SAR images. Remote Sensing, 13: 4384 [DOI: 10.3390/rs13214384http://dx.doi.org/10.3390/rs13214384]
Alert me when the article has been cited
提交
相关文章
相关作者
暂无数据
相关机构
Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology
School of Computer, Nanjing University of Information Science and Technology
School of Electronic Information, Northwestern Polytechnical University
School of Automation, Northwestern Polytechnical University
State Key Laboratory of Complex Electromagnetic Environment Effects on Electronics and Information System (CEMEE)
选择翻译语言
- Postal code：100190
- Tel：010-58887052 Email：nrsb@aircas.ac.cn
- Technical support is provided by Beijing Founder electronics co., LTD 京ICP备20021838号-9
京公网安备11010802024621 - It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
- Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
0
Bulk Citation
Export to
TOP