Deep capsule network combined with spatial-spectral information for hyperspectral image classification

Kuiliang GAO; Xuchu YU; Zhihang SONG; Jin ZHANG; Bing LIU; Yifan SUN

doi:10.11834/jrs.20210309

Technologies and Methodologies | Views : 0 下载量: 1386 CSCD: 5

PDF
Export
Share
Collection
Album

Deep capsule network combined with spatial-spectral information for hyperspectral image classification
Vol. 25, Issue 6, Pages: 1257-1269(2021)
Received：18 August 2020，

Published：07 June 2021
DOI： 10.11834/jrs.20210309
稿件说明：

移动端阅览

高奎亮，余旭初，宋治杭，张晋，刘冰，孙一帆.2021.联合空谱信息的高光谱影像深度胶囊网络分类.遥感学报，25（6）： 1257-1269 DOI： 10.11834/jrs.20210309.

Gao K L，Yu X C，Song Z H，Zhang J，Liu B and Sun Y F. 2021. Deep capsule network combined with spatial-spectral information for hyperspectral image classification. National Remote Sensing Bulletin， 25（6）：1257-1269 DOI： 10.11834/jrs.20210309.

摘要

向量化的胶囊神经元和动态路由式的信息传递机制赋予了胶囊网络更强的特征表示能力。在遥感领域，基于胶囊网络的高光谱影像分类方法已经获得了较传统深度学习模型更为优异的分类结果。针对现有胶囊分类模型中存在的网络浅层、空谱联合信息利用不足等问题，本文利用卷积胶囊层、残差连接、三维卷积胶囊层构建了一种用于高光谱影像分类的新型深度胶囊网络。具体地，本文方法直接以高维数据立方体作为网络输入，并利用胶囊残差块逐层提取数据中的深层抽象特征。为了更加充分地利用影像中的空谱联合特征，在深层次的胶囊残差块中引入三维卷积胶囊层，以进一步提高分类精度。为了验证本文方法的有效性，选择University of Pavia、Indian Pines和Salinas等3个常用高光谱数据集和一个大规模机载高光谱数据集Chikusei进行实验。结果表明，与现有深度学习模型相比，本文方法能够获得更为优异的分类效果，在4个数据集上分别获得了99.43%、98.85%、97.14%和97.43%的总体分类精度。

Abstract

Owing to vectorized capsule neurons and dynamic routing algorithm

capsule network possesses stronger feature representation capability than traditional convolutional neural networks. In the field of remote sensing

hyperspectral image (HSI) classification methods based on capsule network have obtained better classification results than traditional deep learning models. Aiming at the problems in the capsule classification models

such as shallow network layers and insufficient utilization of spatial–spectral information

this paper constructs a new deep capsule network for HSI classification. The designed network utilizes convolutional capsule layer

residual connection

and three dimensional convolutional capsule layer to further improve classification accuracy.

The proposed method includes a convolutional layer

four capsule residual blocks

a class capsule layer

and a reconstruction network. First

the proposed method takes HSI data cubes as the input directly to retain the spatial–spectral details in the HSI. Then

the deep features in the input data are extracted layer by layer with the capsule residual blocks. Finally

the three dimensional convolutional capsule layer is introduced to make full use of the spatial–spectral information

so as to improve the classification accuracy.

Three public HSI data sets including University of Pavia

Indian Pines and Salinas

as well as a large-scale hyperspectral data set Chikusei

are selected for the experiments. The results demonstrate that the proposed method outperforms the existing deep learning classification methods. Compared with the shallow capsule network model and the deep three dimensional convolutional network with residual structure

the proposed method improves the overall classification accuracy by 1%—3%

2%—5%

1%—2%

and 2%—4% on four different data sets. In addition

hyperparameters such as learning rate

capsule neurons dimension

network structure

and spatial neighborhood have been analyzed in detail. The effectiveness of three dimensional convolutional capsule layer has also been proven by conducting ablation studies.

Compared with the existing HSI classification model based on capsule network

the proposed method has the following advantages. (1) The high-dimensional cubes are used for input data without any dimension reduction process

thus enabling the model to make full use of spatial–spectral details. (2) A deep network is constructed utilizing convolutional capsule layers and residual connection

thus allowing the model to extract more robust and abstract deep features. (3) The introduction of three dimensional convolutional capsule layer can make full use of spatial–spectral information in HSI to further improve the classification accuracy.

关键词

Keywords

references

Benediktsson J A , Palmason J A and Sveinsson J R . 2005 . Classification of hyperspectral data from urban areas based on extended morphological profiles . IEEE Transactions on Geoscience and Remote Sensing , 43 ( 3 ): 480 - 491 [ DOI: 10.1109/tgrs.2004.842478 http://dx.doi.org/10.1109/tgrs.2004.842478 ]

Chen Y S , Lin Z H , Zhao X , Wang G and Gu Y F . 2014 . Deep learning-based classification of hyperspectral data . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 7 ( 6 ): 2094 - 2107 [ DOI: 10.1109/JSTARS.2014.2329330 http://dx.doi.org/10.1109/JSTARS.2014.2329330 ]

Chen Y S , Zhao X and Jia X P . 2015 . Spectral–spatial classification of hyperspectral data based on deep belief network . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 8 ( 6 ): 2381 - 2392 [ DOI: 10.1109/JSTARS.2015.2388577 http://dx.doi.org/10.1109/JSTARS.2015.2388577 ]

Chen Y S , Jiang H L , Li C Y , Jia X P and Ghamisi P . 2016 . Deep feature extraction and classification of hyperspectral images based on convolutional neural networks . IEEE Transactions on Geoscience and Remote Sensing , 54 ( 10 ): 6232 - 6251 [ DOI: 10.1109/TGRS.2016.2584107 http://dx.doi.org/10.1109/TGRS.2016.2584107 ].

Dong X F , Gan F P , Li N , Yan B K , Zhang L , Zhao J Q , Yu J C , Liu R Y and Ma Y N . 2020 . Fine mineral identification of GF-5 hyperspectral image . Journal of Remote Sensing , 24 ( 4 ): 454 - 464

董新丰 , 甘甫平 , 李娜 , 闫柏琨 , 张磊 , 赵佳琪 , 于峻川 , 刘镕源 , 马燕妮 . 2020 . 高分五号高光谱影像矿物精细识别 . 遥感学报 , 24 ( 4 ): 454- 464 [ DOI: 10.11834/jrs.20209194 http://dx.doi.org/10.11834/jrs.20209194 ]

Gao K L , Zhang P Q , Yu X C , Tan X and Liu B . 2019 . Classification method of hyperspectral image based on Network In Network structure . Journal of Geomatics Science and Technology , 36 ( 5 ): 500 - 504, 510

高奎亮 , 张鹏强 , 余旭初 , 谭熊 , 刘冰 . 2019 . 基于Network In Network网络结构的高光谱影像分类方法 . 测绘科学技术学报 , 36 ( 5 ): 500 - 504 , 510

Gao K L , Liu B , Yu X C , Qin J C , Zhang P Q and Tan X . 2020 . Deep relation network for hyperspectral image few-shot classification . Remote Sensing , 12 ( 6 ): 923 [ DOI: 10.3390/rs12060923 http://dx.doi.org/10.3390/rs12060923 ]

He L , Li J , Liu C Y and Li S T . 2018 . Recent advances on spectral-spatial hyperspectral image classification: an overview and new guidelines . IEEE Transactions on Geoscience and Remote Sensing , 56 ( 3 ): 1579 - 1597 [ DOI: 10.1109/TGRS.2017.2765364 http://dx.doi.org/10.1109/TGRS.2017.2765364 ]

Hu W , Huang Y Y , Li W , Zhang F and Li H C . 2015 . Deep convolutional neural networks for hyperspectral image classification . Journal of Sensors , 2015 : 258619 [ DOI: 10.1155/2015/258619 http://dx.doi.org/10.1155/2015/258619 ]

Jiang X F , Liu W B , Zhang Y , Liu J R , Li S Y and Lin J Z . 2020 . Spectral–spatial hyperspectral image classification using dual-channel capsule networks . IEEE Geoscience and Remote Sensing Letters [ DOI: 10.1109/LGRS.2020.2991405 http://dx.doi.org/10.1109/LGRS.2020.2991405 ]

Kang X D , Zhuo B B and Duan P H . 2019 . Semi-supervised deep learning for hyperspectral image classification . Remote Sensing Letters , 10 ( 4 ): 353 - 362

Lee H and Kwon H . 2017 . Going deeper with contextual CNN for hyperspectral image classification . IEEE Transactions on Image Processing , 26 ( 10 ): 4843 - 4855 [ DOI: 10.1109/tip.2017.2725580 http://dx.doi.org/10.1109/tip.2017.2725580 ]

Li J , Marpu P R , Plaza A , Bioucas-Dias J M and Benediktsson J A . 2013 . Generalized composite kernel framework for hyperspectral image classification . IEEE Transactions on Geoscience and Remote Sensing , 51 ( 9 ): 4816 - 4829 [ DOI: 10.1109/TGRS.2012.2230268 http://dx.doi.org/10.1109/TGRS.2012.2230268 ]

Li Y , Zhang H K and Shen Q . 2017 . Spectral–spatial classification of hyperspectral imagery with 3D convolutional neural network . Remote Sensing , 9 ( 1 ): 67 [ DOI: 10.3390/rs9010067 http://dx.doi.org/10.3390/rs9010067 ]

Liu B , Yu X C , Zhang P Q , Yu A Z , Fu Q Y and Wei X P . 2018a . Supervised deep feature extraction for hyperspectral image classification . IEEE Transactions on Geoscience and Remote Sensing , 56 ( 4 ): 1909 - 1921 [ DOI: 10.1109/TGRS.2017.2769673 http://dx.doi.org/10.1109/TGRS.2017.2769673 ]

Liu B , Yu X C , Yu A Z and Wan G . 2018b . Deep convolutional recurrent neural network with transfer learning for hyperspectral image classification . Journal of Applied Remote Sensing , 12 ( 2 ): 026028 [ DOI: 10.1117/1.JRS.12.026028 http://dx.doi.org/10.1117/1.JRS.12.026028 ]

Liu B , Yu X C , Zhang P Q and Tan X . 2019 . Deep 3D convolutional network combined with spatial-spectral features for hyperspectral image classification . Acta Geodaetica et Cartographica Sinica , 48 ( 1 ): 53 - 63

刘冰 , 余旭初 , 张鹏强 . 2019 . 联合空-谱信息的高光谱影像深度三维卷积网络分类 . 测绘学报 . 48 ( 1 ): 53 - 63

Paoletti M E , Haut J M , Fernandez-Beltran R , Plaza J , Plaza A , Li J and Pla F . 2019 . Capsule networks for hyperspectral image classification . IEEE Transactions on Geoscience and Remote Sensing , 57 ( 4 ): 2145 - 2160 [ DOI: 10.1109/TGRS.2018.2871782 http://dx.doi.org/10.1109/TGRS.2018.2871782 ]

Rajasegaran J , Jayasundara V , Jayasekara S , Jayasekara H , Seneviratne S and Rodrigo R . 2019 . DeepCaps: going deeper with capsule networks // Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Long Beach, CA, USA : IEEE : 10717 - 10725 [ DOI: 10.1109/CVPR.2019.01098 http://dx.doi.org/10.1109/CVPR.2019.01098 ]

Sabour S , Frosst N and Hinton G E . 2017 . Dynamic routing between capsules . arXiv : 1710 . 09829

Tarabalka Y , Fauvel M , Chanussot J and Benediktsson J A . 2010 . SVM- and MRF-based method for accurate classification of hyperspectral images . IEEE Geoscience and Remote Sensing Letters , 7 ( 4 ): 736 - 740 [ DOI: 10.1109/LGRS.2010.2047711 http://dx.doi.org/10.1109/LGRS.2010.2047711 ]

Xi E , Bing S and Jin Y . 2017 . Capsule network performance on complex data . arXiv : 1712 . 03480

Xu Q , Xiao Y , Wang D Y and Luo B . 2020 . CSA-MSO3DCNN: multiscale octave 3D CNN with channel and spatial attention for hyperspectral image classification . Remote Sensing , 12 ( 1 ): 188 [ DOI: 10.3390/rs12010188 http://dx.doi.org/10.3390/rs12010188 ]

Yin J H , Li S , Zhu H M and Luo X Y . 2019 . Hyperspectral image classification using CapsNet with well-initialized shallow layers . IEEE Geoscience and Remote Sensing Letters , 16 ( 7 ): 1095 - 1099 [ DOI: 10.1109/lgrs.2019.2891076 http://dx.doi.org/10.1109/lgrs.2019.2891076 ]

Zhang M M , Li W and Du Q . 2018 . Diverse region-based CNN for hyperspectral image classification . IEEE Transactions on Image Processing , 27 ( 6 ): 2623 - 2634 [ DOI: 10.1109/TIP.2018.2809606 http://dx.doi.org/10.1109/TIP.2018.2809606 ]

Zhang L , Zhang J Y , Wei W and Zhang Y N . 2019 . Learning discriminative compact representation for hyperspectral imagery classification . IEEE Transactions on Geoscience and Remote Sensing , 57 ( 10 ): 8276 - 8289 [ DOI: 10.1109/TGRS.2019.2919938 http://dx.doi.org/10.1109/TGRS.2019.2919938 ]

Zhi L , Yu X C , Liu B and Wei X P . 2019 . A dense convolutional neural network for hyperspectral image classification . Remote Sensing Letters , 10 ( 1 ): 59 - 66 [ DOI: 10.1080/2150704X.2018.1526424 http://dx.doi.org/10.1080/2150704X.2018.1526424 ]

Zhu K Q , Chen Y S , Ghamisi P , Jia X P and Benediktsson J A . 2019 . Deep convolutional capsule network for hyperspectral image spectral and spectral-spatial classification . Remote Sensing , 11 ( 3 ): 223 [ DOI: 10.3390/rs11030223 http://dx.doi.org/10.3390/rs11030223 ]

Alert me when the article has been cited

提交

Fine 3D mapping and vegetation spatial differentiation study based on UAV LiDAR point cloud in karst wetland

LAE-Transformer： An airborne LiDAR point cloud segmentation network incorporating local-aware enhancement

A multi-scale InSAR phase filtering method incorporating adaptive enhancement and dynamic screening mechanisms

Multimodal object detection method using adaptive dusion of infrared and visible features

Research progress of generative adversarial networks in remote sensing image fusion