Hybrid-Gird： An explainable method for fine-grained classification of remote sensing images

ZHU Kaiwen; YOU Yanan; CAO Jingyi; MENG Gang; QIAO Yuanyuan; YANG Jie

doi:10.11834/jrs.20243252

Satellite Information Intelligent Processing Technology | Views : 0 下载量: 2100 CSCD: 2

R-PDF
PDF
Export
Share
Collection
Album

Hybrid-Gird： An explainable method for fine-grained classification of remote sensing images
Vol. 28, Issue 7, Pages: 1722-1734(2024)
Received：11 July 2023，

Published：07 July 2024
DOI： 10.11834/jrs.20243252
稿件说明：

移动端阅览

朱凯雯，尤亚楠，曹婧宜，孟钢，乔媛媛，杨洁.2024.Hybrid-Gird：遥感图像细粒度分类可解释方法.遥感学报，28（7）： 1722-1734 DOI： 10.11834/jrs.20243252.

Zhu K W，You Y N，Cao J Y，Meng G，Qiao Y Y and Yang J. 2024. Hybrid-Gird： An explainable method for fine-grained classification of remote sensing images. National Remote Sensing Bulletin， 28（7）：1722-1734 DOI： 10.11834/jrs.20243252.

摘要

基于遥感图像的目标细粒度分类深度神经网络已技术日益成熟，网络决策的可解释性研究是当前细粒度分类深度学习算法进一步提高决策可信度的关键问题。为精确表征对模型决策起决定性作用的本质特征，本文基于博弈竞争理论对遥感图像目标细粒度分类任务进行建模，分析了IG、SmoothGrad、Grad-CAM等可解释性方法在遥感图像目标细粒度分类网络上的适用性，提出了一种尺度自适应的目标细粒度分类本质特征可解释性分析方法Hybrid-Grid，使用像素级与局部特征关系融合算法提高对支撑网络决策的目标本质特征的精确描述能力。结果表明：本文提出的Hybrid-Grid对目标细粒度分类网络的解释效果在ADCC量化评估指标上达到78.87，相较Score-CAM有大幅提升；与SmoothGrad、Grad-CAM的解释结果相对比，本文方法在删除及精度损失实验上表现最好，使EFM-Net的Top-1准确率、Top-5准确率、F1得分分别损失了16.92%、1.61%、17.21%，证明Hybrid-Grid准确解释了对细粒度分类网络决策贡献最大的目标本质特征。本文提出的可解释性分析方法能够更精准地揭示当前目标细粒度分类网络的决策特征依据。

Abstract

At present

the deep neural network for targets fine-grained classification based on remote sensing images has been widely applied in military and civilian fields. The “black box” problem of deep learning network models still makes it difficult for people to understand the decision-making basis of the network in fine-grained classification tasks. This not only limits the possibility of deep neural networks optimizing and improving through feedback guidance

but also makes them unable to be fully trusted by humans and applied in important fields such as military and medical fields. Thus

how to carry out explainability research on its internal decision-making mechanism is the key problem for the current fine-grained algorithm to further improve the credibility of decision-making basis. This article summarized the commonly used explainable methods in existing fine-grained classification networks for remote sensing image targets. On this basis

a mathematical model for the problem of target fine-grained classification is established. It also models the fine-grained classification task of remote sensing image targets based on game competition theory

analyzes the applicability of explainable methods such as IG

SmoothGrad

and Grad CAM on the fine-grained classification network of remote sensing image targets

and proposes a scale adaptive method for analyzing the explainability of essential features of fine-grained classification of targets

that is

Hybrid Grid. The fusion algorithm of pixel level and local feature relationships is used to improve the accurate description ability of essential features of targets that support network decision-making. The typical explaining methods involved in this article were evaluated using indicators such as Average Drop

Coherence

Complexity

ADCC

and Deletion and precision loss

confirming the accuracy of the proposed method. The experimental results show that the Hybrid-Grid proposed in this paper achieves a score of 78.87 on the quantitative evaluation index of ADCC for target fine-grained classification networks

which is significantly improved compared to Score-CAM. Compared with the explanation results of SmoothGrad and Grad-CAM

our method performed the best in deletion and accuracy loss experiments

resulting in a loss of 16.92%

1.61%

and 17.21% in the Top-1 accuracy

Top-5 accuracy

and F1 score of EFM Net

respectively. This proves that the Hybrid Grid accurately explains the essential feature of the target that contributes the most to fine-grained classification network decision-making. Based on several typical fine-grained classification networks

the feasibility and universality of the explaining method proposed in this paper have been confirmed. The explaining method proposed in this article can more accurately reveal the decision feature basis of the current target fine-grained classification network. Future work will further enhance the visualization and interpretation accuracy of the essential features of the target

assisting in improving the classification performance and decision-making credibility of fine-grained classification networks.

关键词

Keywords

references

Bach S , Binder A , Montavon G , Klauschen F , Müller K R and Samek W . 2015 . On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation . PLoS One , 10 ( 7 ): e 0130140 [ DOI: 10.1371/journal.pone.0130140 http://dx.doi.org/10.1371/journal.pone.0130140 ]

Cheng X , Chu C , Zheng Y , Ren J and Zhang Q S . 2021 . A game-theoretic taxonomy of visual concepts in DNNs. arXiv preprint arXiv: 2106 . 10938

Deng H , Zou N , Du M , Chen W , Feng G , Yang Z , Li Z and Zhang Q . 2023 . Understanding and unifying fourteen attribution methods with taylor interactions . arXiv preprint arXiv: 2303.01506 .

Du R , Xie J , Ma Z , Chang D , Song Y Z and Guo J . 2021 . Progressive learning of category-consistent multi-granularity features for fine-grained visual classification . IEEE Transactions on Pattern Analysis and Machine Intelligence , 44 ( 12 ): 9521 - 9535 .

Fu K , Dai W , Zhang Y , Wang Z R , Yan M L and Sun X . 2019 . MultiCAM: multiple class activation mapping for aircraft recognition in remote sensing images . Remote Sensing , 11 ( 5 ): 544 [ DOI: 10.3390/rs11050544 http://dx.doi.org/10.3390/rs11050544 ]

Gong J Y , Huan L X and Zheng X W . 2022 . Deep learning interpretability analysis methods in image interpretation . Acta Geodaetica et Cartographica Sinica , 51 ( 6 ): 873 - 884

龚健雅 , 宦麟茜 , 郑先伟 . 2022 . 影像解译中的深度学习可解释性分析方法 . 测绘学报 , 51 ( 6 ): 873 - 884 [ DOI: 10.11947/j.AGCS.2022.20220106 http://dx.doi.org/10.11947/j.AGCS.2022.20220106 ]

Han Y Q , Yang X Y , Pu T and Peng Z M . 2022 . Fine-grained recognition for oriented ship against complex scenes in optical remote sensing images . IEEE Transactions on Geoscience and Remote Sensing , 60 : 5612318 [ DOI: 10.1109/TGRS.2021.3123666 http://dx.doi.org/10.1109/TGRS.2021.3123666 ]

He K M , Zhang X Y , Ren S Q and Sun J . 2016 . Deep residual learning for image recognition/ /Proceedings of the IEEE conference on computer vision and pattern recognition: 770 - 778 [ DOI: 10.1109/CVPR.2016.90 http://dx.doi.org/10.1109/CVPR.2016.90 ]

Liang W , Li J H , Diao W H , Sun X , Fu K and Wu Y R . 2020 . FGATR-Net: automatic network architecture design for fine-grained aircraft type recognition in remote sensing images . Remote Sensing , 12 ( 24 ): 4187 [ DOI: 10.3390/rs12244187 http://dx.doi.org/10.3390/rs12244187 ]

Lundberg S M and Lee S I . 2017 . A unified approach to interpreting model predictions // Proceedings of the 31st International Conference on Neural Information Processing Systems . Long Beach : Curran Associates, Inc.: 4768 - 4777

Nie Y , Bian C J and Li L G . 2022 . Adap-EMD: adaptive EMD for aircraft fine-grained classification in remote sensing . IEEE Geoscience and Remote Sensing Letters , 19 : 6509005 [ DOI: 10.1109/LGRS.2022.3168581 http://dx.doi.org/10.1109/LGRS.2022.3168581 ]

Petsiuk V , Das A and Saenko K . 2018 . RISE: randomized input sampling for explanation of black-box models // Proceedings of the British Machine Vision Conference 2018 . Newcastle : BMVA Press: 151

Poppi S , Cornia M , Baraldi L and Cucchiara R . 2021 . Revisiting the evaluation of class activation mapping for explainability: A novel metric and experimental analysis/ /Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 2299 - 2304

Ribeiro M T , Singh S and Guestrin C . 2016 . Why should I trust you?": explaining the predictions of any classifier // Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . San Francisco : ACM: 1135 - 1144 [ DOI: 10.1145/2939672.2939778 http://dx.doi.org/10.1145/2939672.2939778 ]

Chen S , Wang H , Xu F and Jin Y Q . 2016 . Target classification using the deep convolutional networks for sar images . IEEE transactions on geoscience and remote sensing , 54 ( 8 ): 4806 - 4817

Selvaraju R R , Cogswell M , Das A , Vedantam R , Parikh D and Batra D . 2017 . Grad-cam: Visual explanations from deep networks via gradient-based localization/ /Proceedings of the IEEE international conference on computer vision: 618 - 626

Selvaraju R R , Das A , Vedantam R , Cogswell M , Parikh D and Batra D . 2016 . Grad-CAM: Why did you say that?. arXiv preprint arXiv: 1611 . 07450

Simonyan K , Vedaldi A and Zisserman A . 2013 . Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 1312 . 6034

Smilkov D , Thorat N , Kim B , Viégas F and Wattenberg M . 2017 . Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 1706 . 03825

Springenberg J T , Dosovitskiy A , Brox T and Riedmiller M . 2014 . Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412 . 6806

Springenberg J T , Dosovitskiy A , Brox T and Riedmiller M . Striving for simplicity: The all convolutional net . arXiv preprint arXiv: 1412.6806 , 2014 .

Sundararajan M , Taly A , Yan Q . Axiomatic attribution for deep networks // International conference on machine learning . PMLR, 2017 : 3319 - 3328 .

Wang H F , Wang Z F , Du M N , Yang F , Zhang Z J , Ding S R , Mardziel P and Hu X . 2020 . Score-CAM: Score-weighted visual explanations for convolutional neural networks // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops . 2020: 24 - 25 [D0I: arXiv: 1910. 01279 v 2 ]

Xiong W , Xiong Z Y and Cui Y Q . 2022 . An explainable attention network for fine-grained ship classification using remote-sensing images . IEEE Transactions on Geoscience and Remote Sensing , 60 : 5620314 [ DOI: 10.1109/TGRS.2022.3162195 http://dx.doi.org/10.1109/TGRS.2022.3162195 ]

Yang P B , Sang J T , Zhang B , Feng Y G and Yu J . 2023 . Survey on interpretability of deep models for image classification . Journal of Software , 34 ( 1 ): 230 - 254

杨朋波 , 桑基韬 , 张彪 , 冯耀功 , 于剑 . 2023 . 面向图像分类的深度模型可解释性研究综述 . 软件学报 , 34 ( 1 ): 230 - 254 [ DOI: 10.13328/j.cnki.jos.006415 http://dx.doi.org/10.13328/j.cnki.jos.006415 ]

Yi Y H , You Y N , Li C and Zhou W L . 2023 . EFM-net: An essential feature mining network for target fine-grained classification in optical remote sensing images . IEEE Transactions on Geoscience and Remote Sensing . [ DOI: 10.1109/TGRS.2023.3265669 http://dx.doi.org/10.1109/TGRS.2023.3265669 ]

Zeiler M D and Fergus R . 2014 . Visualizing and understanding convolutional networks // Computer Vision-ECCV 2014: 13th European Conference , Zurich, Switzerland, September 6 - 12 , 2014, Proceedings, Part I 13 . Springer International Publishing, 2014 : 818 - 833

Alert me when the article has been cited

提交

Low-Cost Fine-Grained interpretation method of remote sensing images based on weakly supervised instance segmentation

Dual-branch remote sensing shadow detection with the fusion of Transformer and CNN

ADC-CPANet：A remote sensing image classification method based on local-global feature fusion

Remote sensing image detection method based on brain-inspired spiking neural networks

Few-shot object detection in optical remote sensing images