سابِق|تال

Multi-information supervision in optical remote sensing images

  • role： First author第一作者
  • Affiliation：

    School of Automation, Northwestern Polytechnical University, Xi'an 710129, China

  • Email：jbwang@mail.nwpu.edu.cn
  • Introduction：王家宝， 研究方向为高分辨率遥感图像理解。E-mail： jbwang@mail.nwpu.edu.cn
WANG Jiabao,  
  • role： Corresponding author通信作者
  • Affiliation：

    School of Automation, Northwestern Polytechnical University, Xi'an 710129, China

  • Email：gcheng@nwpu.edu.cn
  • Introduction：程塨， 研究方向为高分辨率遥感图像理解。E-mail： gcheng@nwpu.edu.cn
CHENG Gong*,  
  • Affiliation：

    School of Automation, Northwestern Polytechnical University, Xi'an 710129, China

XIE Xingxing,  
  • Affiliation：

    School of Automation, Northwestern Polytechnical University, Xi'an 710129, China

YAO Yanqing,  
  • Affiliation：

    School of Automation, Northwestern Polytechnical University, Xi'an 710129, China

HAN Junwei

ملخص

Oriented object detection is a basic task in the interpretation of high-resolution remote sensing images. Compared with general detectors, oriented detectors can locate instances with oriented bounding boxes, which are consistent with arbitrary-oriented ground truths in remote sensing images. Currently, oriented object detection has greatly progressed with the development of the convolutional neural network. However, this task is still challenging because of the extreme variation in object scales and arbitrary orientations. Most oriented detectors are evolved from horizontal detectors. They first generate horizontal proposals using the Region Proposal Network (RPN). Then, they classify these proposals into different categories and transform them into oriented bounding boxes. Despite their success, these detectors exploit only the annotations at the end of the network and do not fully utilize the angle and semantic information.This work proposes an Angle-based Region Proposal Network (ARPN), which learns the angle of objects and generates oriented proposals. The structure of ARPN is the same as that of RPN. However, for each proposal, instead of outputting four parameters for regression, ARPN generates five parameters, which are the center (x, y), shape (w, h), and angle (t). In the training, we first assign anchors with ground truths by the Intersection of Unions. Then, we directly supervise the ARPN with the shape and angle information of ground truths. We also propose a semantic branch to output image semantic results for utilizing the advantage of the semantic information. The semantic branch consists of two convolutional layers and is parallel with the detection head. We first assign objects to different scale levels according to their areas. Then, we create semantic labels in each scale and use them to supervise the semantic branch. With the semantic information supervision, the model will learn translation-variant features and improve accuracy. Moreover, the outputs of the semantic branch indicate the objectness in each place, which can filter out false positives of final predictions.We conduct comprehensive experiments on the DOTA dataset to validate the effectiveness of the proposed methods. In the data preparation, we first crop original images into 1024×1024 patches with the stride of 824. Compared with the baseline, the ARPN achieves a 2.2% increase in mAP, while the semantic branch contributes an additional 0.8% improvement in mAP. Finally, we combine both methods and achieve a 74.64% mAP, which is competitive with those obtained by other oriented object detectors. We visualize some results on the DOTA dataset. The results show that our method is highly effective for small objects and densely packed objects.We proposed ARPN and the semantic branch to utilize the multi-information in remote sensing images. The ARPN can directly generate oriented proposals, which can lead to better recall of oriented objects. The semantic branch increases the translation-variant property of the features. Experiments demonstrate the effectiveness of our method, which achieves a 74.64% mAP on the DOTA dataset. In the future works, we will focus on the model efficiency and the inference speed.

مفهوم

object detection;Oriented object detection;Region proposal generation;multi-information;remote sensing images

References

  1. 1.
    Cao Q, Ma A L, Zhong Y F, Zhao J, Zhao B and Zhang L P. 2019. Urban classification by multi-feature fusion of hyperspectral image and LiDAR data. Journal of Remote Sensing, 23(5): 892-903
  2. 2.
    Chen K Q, Gao X, Yan M L, Zhang Y and Sun X. 2020. Building extraction in pixel level from aerial imagery with a deep encoder-decoder network. Journal of Remote Sensing (Chinese), 24(9): 1134-1142
  3. 3.
    Cheng G, Zhou P C and Han J W. 2016. Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 54(12): 7405-7415
  4. 4.
    Ding J, Xue N, Long Y, Xia G S and Lu Q K. 2019. Learning RoI transformer for oriented object detection in aerial images//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE: 2844-2853
  5. 5.
    Girshick R. 2015. Fast R-CNN//2015 IEEE International Conference on Computer Vision (ICCV). Santiago: IEEE: 1440-1448
  6. 6.
    Girshick R, Donahue J, Darrell T and Malik J. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE: 580-587
  7. 7.
    Gong J Y and Zhong Y F. 2016. Survey of intelligent optical remote sensing image processing. Journal of Remote Sensing, 20(5): 733-747
  8. 8.
    Han J M, Ding J, Li J and Xia G S. 2022. Align deep features for oriented object detection. IEEE Transactions on Geoscience and Remote Sensing, 60: 5602511
  9. 9.
    He K M, Gkioxari G, Dollar P and Girshick R. 2020. Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2): 386-397
  10. 10.
    He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE: 770-778
  11. 11.
    Huang Z C, Li W, Xia X G, Wang H, Jie F R and Tao R. 2022a. LO-Det: lightweight oriented object detection in remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 60: 5603515
  12. 12.
    Huang Z C, Li W, Xia X G, Wu X, Cai Z Q and Tao R. 2022b. A novel nonlocal-aware pyramid and multiscale multitask refinement detector for object detection in remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 60: 5601920
  13. 13.
    Ioffe S and Szegedy C. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift//Proceedings of the 32nd International Conference on Machine Learning. Lille: JMLR.org: 448-456
  14. 14.
    Krizhevsky A, Sutskever I and Hinton G E. 2017. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84-90
  15. 15.
    Lin T Y, Goyal P, Girshick R, He K M and Dollar P. 2020. Focal loss for dense object detection. IEEE Trans actionson Pattern Analysis and Machine Intelligence, 42(2): 318-327
  16. 16.
    Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P and Zitnick C L. 2014. Microsoft COCO: common objects in context//13th European Conference on Computer Vision. Zurich: Springer: 740-755
  17. 17.
    Ma J Q, Shao W Y, Ye H, Wang L, Wang H, Zheng Y B and Xue X Y. 2018. Arbitrary-oriented scene text detection via rotation proposals. IEEE Transactions on Multimedia, 20(11): 3111-3122
  18. 18.
    Ming Q, Zhou Z Q, Miao L J, Zhang H W and Li L H. 2020. Dynamic anchor learning for arbitrary-oriented object detection. arXiv preprint arXiv: 2012.04150
  19. 19.
    Pang J M, Chen K, Shi J P, Feng H J, Ouyang W L and Lin D H. 2019. Libra R-CNN: towards balanced learning for object detection//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE: 821-830
  20. 20.
    Redmon J, Divvala S, Girshick R and Farhadi A. 2016. You only look once: unified, real-time object detection//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE: 779-788
  21. 21.
    Redmon J and Farhadi A. 2017. YOLO9000: better, faster, stronger//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE: 6517-6525
  22. 22.
    Redmon J and Farhadi A. 2018. YOLOv3: an incremental improvement. arXiv preprint arXiv: 1804.02767
  23. 23.
    Ren S Q, He K M, Girshick R and Sun J. 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 1137-1149
  24. 24.
    Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
  25. 25.
    Song G L, Liu Y and Wang X G. 2020. Revisiting the sibling head in object detector//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE: 11560-11569
  26. 26.
    Sun X, Liang W, Diao W H, Cao Z Y, Feng Y C, Wang B and Fu K. 2020. Progress and challenges of remote sensing edge intelligence technology. Journal of Image and Graphics, 25(9): 1719-1738
  27. 27.
    Szegedy C, Ioffe S, Vanhoucke V and Alemi A A. 2017. Inception-v4, inception-ResNet and the impact of residual connections on learning//Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence. San Francisco: AAAI Press: 4278-4284
  28. 28.
    Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V and Rabinovich A. 2015. Going deeper with convolutions//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE: 1-9
  29. 29.
    Szegedy C, Vanhoucke V, Ioffe S, Shlens J and Wojna Z. 2016. Rethinking the inception architecture for computer vision//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE: 2818-2826
  30. 30.
    Wang J Q, Chen K, Yang S, Loy C C and Lin D H. 2019. Region proposal by guided anchoring//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE: 2960-2969
  31. 31.
    Wu Y, Chen Y P, Yuan L, Liu Z C, Wang L J, Li H Z and Fu Y. 2020. Rethinking classification and localization for object detection//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE: 10183-10192
  32. 32.
    Xia G S, Bai X, Ding J, Zhu Z, Belongie S, Luo J B, Datcu M, Pelillo M and Zhang L P. 2018. DOTA: a large-scale dataset for object detection in aerial images//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE: 3974-3983
  33. 33.
    Yang X, Yan J C, Feng Z M and He T. 2021. R3Det: refined single-stage detector with feature refinement for rotating object. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4): 3163-3171
  34. 34.
    Yang X, Yang J R, Yan J C, Zhang Y, Zhang T F, Guo Z, Sun X and Fu K. 2019. SCRDet: towards more robust detection for small, cluttered and rotated objects//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE: 8231-8240
  35. 35.
    Yao H G, Wang C, Yu J, Bai X J and Li W. 2020. Recognition of small-target ships in complex satellite images. Journal of Remote Sensing (Chinese), 24(2): 116-125
  36. 36.
    Yao Y Q, Cheng G, Xie X X and Han J W. 2021. Optical remote sensing image object detection based on multi-resolution feature fusion. National Remote Sensing Bulletin, 25(5): 1124-1137
  37. 37.
    Zhou P C, Cheng G, Yao X W and Han J W. 2021. Machine learning paradigms in high-resolution remote sensing image interpretation. National Remote Sensing Bulletin, 25(1): 182-197

قراءة النص الكامل

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website