Joint Classification of Hyperspectral and LiDAR Data Based on Inter-Modality Match Learning

HANG Renlong; SUN Yu; LIU Qingshan

doi:10.11834/jrs.20232635

Views : 0 下载量: 30 CSCD: 0

R-PDF
PDF
Export
Share
Collection
Album

Joint Classification of Hyperspectral and LiDAR Data Based on Inter-Modality Match Learning
Pages: 1-15(2023)
DOI： 10.11834/jrs.20232635

扫描看全文

Quote
杭仁龙，孙瑜，刘青山.XXXX.基于模态间匹配学习的高光谱和激光雷达联合分类.遥感学报，XX（XX）： 1-15

HANG Renlong，SUN Yu，LIU Qingshan. XXXX. Joint Classification of Hyperspectral and LiDAR Data Based on Inter-Modality Match Learning. National Remote Sensing Bulletin， XX（XX）：1-15
杭仁龙，孙瑜，刘青山.XXXX.基于模态间匹配学习的高光谱和激光雷达联合分类.遥感学报，XX（XX）： 1-15 DOI： 10.11834/jrs.20232635.

HANG Renlong，SUN Yu，LIU Qingshan. XXXX. Joint Classification of Hyperspectral and LiDAR Data Based on Inter-Modality Match Learning. National Remote Sensing Bulletin， XX（XX）：1-15 DOI： 10.11834/jrs.20232635.

摘要

如何有效地提取和融合不同模态的特征是高光谱图像和激光雷达数据联合分类的关键。近年来，得益于深度学习强大的特征学习能力，其在高光谱图像和激光雷达数据联合分类领域受到了越来越多的关注。然而，现有的深度学习模型大多基于监督学习的模式，分类性能依赖标注样本的数量和质量。为此，本文提出了一种基于模态间匹配学习的联合分类方法，充分利用未标注样本的信息，减少对标注信息的依赖性。具体而言，本文首先通过高光谱图像和激光雷达数据之间的匹配关系和KMeans聚类算法，构造模态匹配标签。然后，利用该标签训练含有多个卷积层的匹配学习网络。该网络由两个并行分支构成，每个分支负责提取单个模态的特征。最后，以该网络为基础，构造高光谱图像和激光雷达数据联合分类模型。该模型的参数由匹配学习网络进行初始化，因而只需要少量标注样本进行微调即可达到理想的分类效果。为了验证本文方法的有效性，在Houston和MUUFL两个常用的高光谱图像和激光雷达数据联合分类数据集上进行了大量的实验。实验结果表明，与已有的分类模型相比，本文方法能够获得更高的分类性能。

Abstract

In recent years, there have been plenty of excellent models for joint classification of hyperspectral image and LiDAR data, which were designed on the basis of supervised learning methods such as convolutional neural networks. Their classification performance depends largely on the quantity and quality of training samples. Nevertheless, when the distribution of ground objects becomes increasingly complex and the resolutions of remote sensing images grow higher and higher, it is hard to obtain high-quality labels only with limited cost and manpower. To this end, many scholars made efforts to learn features directly from unlabeled samples. For instance, the theory of autoencoder was applied to multimodal joint classification, achieving satisfactory performance. Although those methods based on reconstruction idea reduce the dependence on labeled information to a certain extent, there still exist several problems to be settled. For example, such methods pay more attention to data reconstruction, but fail to guarantee that the extracted features have sufficient discriminant ability, thus affecting the performance of joint classification.To address the aforementioned issue, this paper proposes an effective model named ,Joint Classification of Hyperspectral and LiDAR Data Based on Inter-Modality Match Learning,. Different from feature extraction models based on reconstruction idea, the proposed model tends to compare the matching relationship between samples from different modality, hence enhancing the discriminative ability of features. Specifically, it is composed of inter-modality matching learning network and multimodal joint classification network. The former one is prone to identify whether the input patch pairs of hyperspectral image and LiDAR data match, so reasonable construction of matching labels is essential. To this end, spatial positions of center pixels in cropped patches and KMeans clustering method are employed. Such constructed labels and the patch pairs are combined together to train the network. It is worth noting that this process does not use manual labeled information and is capable of directly extracting features from abundant unlabeled samples. Furthermore, in the joint classification stage, the structure and trained parameters of matching learning network are transferred, and then a small number of manually labeled training samples are used to finetune the model parameters.Eventually, extensive experiments were conducted on two widely used datasets named Houston and MUUFL to verify the effectiveness of the proposed model, including comparison experiments with several state-of-the-art models, hyper-parameter analysis experiments and ablation study. Take the first experiment as an example, when compared with other models such as CNN, EMFNet, AE_H, AE_HL, CAE_H, CAE_HL, IP-CNN and PToP CNN, the proposed model in this paper is able to achieve higher performance on both datasets, whose OAs are 88.39% and 81.46%, respectively. In summary, our model enables to reduce the dependence on manually labeled data and improve the joint classification accuracy in the case of limited training samples. In the future, a better model structure and more testing datasets will be explored to make further improvements.

关键词

高光谱图像激光雷达数据深度学习匹配学习联合分类

Keywords

Hyperspectral ImageLiDAR DataDeep LearningMatch LearningJoint Classification

references

Abbasi A N and He M Y. 2019. Convolutional Neural Networ-k with PCA and Batch Normalization for Hyperspectral Image Classification. 2019 IEEE International Geoscience and Remote Sensing Symposium, 959-962 [DOI:10.1109/IGARSS.2019.8899329http://dx.doi.org/10.1109/IGARSS.2019.8899329]

Cao Q, Ma A L, Zhong Y F, Zhao J, Zhao B and Zhang L P. 2019. Urban classification by multi-feature fusion of h-yperspectral image and LiDAR data. Journal of Remote Sensing, 23(5): 892-903

曹琼, 马爱龙, 钟燕飞, 赵济, 赵贝, 张良培. 2019. 高光谱-LiDAR多级融合城区地表覆盖分类. 遥感学报, 23(5): 892-903 [DOI:10.11834/jrs.20197512http://dx.doi.org/10.11834/jrs.20197512]

Chen Y S, Lin Z H, Zhao X, Wang G and Gu Y F. 2014. D-eep Learning-Based Classification of Hyperspectral Data. IEEE Journal of Selected Topics in Applied Earth Obser-vations and Remote Sensing, 7(6): 2094-2107 [DOI:10.1109/JSTARS.2014.2329330http://dx.doi.org/10.1109/JSTARS.2014.2329330]

Chen Y S, Jiang H L, Li C Y, Jia X P and Ghamisi P. 2016.Deep Feature Extraction and Classification of Hyperspectr-al Images Based on Convolutional Neural Networks. IEE-E Transactions on Geoscience and Remote Sensing, 54(10): 6232-6251 [DOI: 10.1109/TGRS.2016.2584107http://dx.doi.org/10.1109/TGRS.2016.2584107]

Chen Y S, Li C Y, Ghamisi P, Jia X P and Gu Y F. 2017. Deep Fusion of Remote Sensing Data for Accurate Class-ification. IEEE Geoscience and Remote Sensing Letters, 14(8): 1253-1257 [DOI: 10.1109/LGRS.2017.2704625http://dx.doi.org/10.1109/LGRS.2017.2704625]

Dalponte M, Bruzzone L and Gianelle D. 2008. Fusion of Hy-perspectral and LIDAR Remote Sensing Data for Classif-ication of Complex Forest Areas. IEEE Transactions on Geoscience and Remote Sensing, 46(5): 1416-1427 [DOI:10.1109/TGRS.2008.916480http://dx.doi.org/10.1109/TGRS.2008.916480]

Debes C, Merentitis A, Heremans R, Hahn J, Frangiadakis Net al. 2014. Hyperspectral and LiDAR Data Fusion: Outc-ome of the 2013 GRSS Data Fusion Contest. IEEE Jour-nal of Selected Topics in Applied Earth Observations an-d Remote Sensing, 7(6): 2405-2418 [DOI: 10.1109/JSTARS.2014.2305441http://dx.doi.org/10.1109/JSTARS.2014.2305441]

Gader P, Zare A, Close R, Aitken J and Tuell G. 2013. MUU-FL Gulfport Hyperspectral and Lidar Airborne Data Set.Technical Report, University of Florida, 570.

Ge C R, Du Q, Sun W W, Wang K Y, Li J J and Li Y S. 2021. Deep Residual Network-Based Fusion Framework f-or Hyperspectral and LiDAR Data. IEEE Journal of Sele-cted Topics in Applied Earth Observations and Remote Sensing, 14: 2458-2472 [DOI:10.1109/JSTARS.2021.3054392http://dx.doi.org/10.1109/JSTARS.2021.3054392]

Hang R L, Li Z, Ghamisi P, Hong D F, Xia G Y and Liu QS. 2020. Classification of Hyperspectral and LiDAR DataUsing Coupled CNNs. IEEE Transactions on Geoscience and Remote Sensing, 58(7): 4939-4950 [DOI: 10.1109/TGRS.2020.2969024http://dx.doi.org/10.1109/TGRS.2020.2969024]

He X, Chen Y S, and Lin Z H. 2021. Spatial-Spectral Transf-ormer for Hyperspectral Image Classification. Remote Sen-sing, 13(3): 498 [DOI: 10.3390/rs13030498http://dx.doi.org/10.3390/rs13030498]

Hong D F, Gao L R, Yokoya N, Yao J, Chanussot J, Du Q a-nd Zhang B. 2021. More Diverse Means Better: Multim-odal Deep Learning Meets Remote Sensing Imagery Cla-ssification. IEEE Transactions on Geoscience and Remote Sensing, 59(5): 4340-4354 [DOI: 10.1109/TGRS.2020.3016820http://dx.doi.org/10.1109/TGRS.2020.3016820]

Hong D F, Gao L R, Hang R L, Zhang B and Chanussot J. 2022. Deep Encoder-Decoder Networks for Classification of Hyperspectral and LiDAR Data. IEEE Geoscience and Remote Sensing Letters, 19: 1-5 [DOI: 10.1109/LGRS.2020.3017414http://dx.doi.org/10.1109/LGRS.2020.3017414]

Huang R and Zhu J T. 2013. Using Random Forest to integra-te lidar data and hyperspectral imagery for land cover c-lassification. 2013 IEEE International Geoscience and Re-mote Sensing Symposium, 3978-3981 [DOI: 10.1109/IGARSS.2013.6723704http://dx.doi.org/10.1109/IGARSS.2013.6723704]

Jahan F, Zhou J, Awrangjeb M and Gao Y S. 2020. Inverse Coefficient of Variation Feature and Multilevel Fusion Te-chnique for Hyperspectral and LiDAR Data Classificatio-n. IEEE Journal of Selected Topics in Applied Earth Ob-servations and Remote Sensing, 13: 367-381 [DOI:10.110http://dx.doi.org/10.110

9/JSTARS.2019.2962659]

Kemker R and Kanan C. 2017. Self-Taught Feature Learning for Hyperspectral Image Classification. IEEE Transactio-ns on Geoscience and Remote Sensing, 55(5): 2693-2705 [DOI:10.1109/TGRS.2017.2651639http://dx.doi.org/10.1109/TGRS.2017.2651639]

Li C X, Hang R L and Rasti B. 2021. EMFNet: Enhanced Multisource Fusion Network for Land Cover Classificati-on. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 4381-4389 [DOI:10.1109/JSTARS.2021.3073719http://dx.doi.org/10.1109/JSTARS.2021.3073719]

Li H, Ghamisi P, Soergel U and Zhu X X. 2018. Hyperspec-tral and LiDAR Fusion Using Deep Three-Stream Con-volutional Neural Networks. Remote Sensing, 10(10): 1649 [DOI: 10.3390/rs10101649http://dx.doi.org/10.3390/rs10101649]

Liu Q C, Xiao L, Yang J X and Wei Z H. 2021. CNN-Enh-anced Graph Convolutional Network With Pixel-and Su-perpixel-Level Feature Fusion for Hyperspectral Image Classification. IEEE Transactions on Geoscience and Re-mote Sensing, 59(10): 8657-8671 [DOI: 10.1109/TGRS.2020.3037361http://dx.doi.org/10.1109/TGRS.2020.3037361]

Liu W Q, Wang C, Zang Y, Hu Q, Yu S S and Lai B Q. 2022. A survey on information extraction technology ba-sed on remote sensing big data. Big Data Research, 8(2): 28-57

刘伟权, 王程, 臧彧, 胡倩, 于尚书, 赖柏锜.2022. 基于遥感大数据的信息提取技术综述. 大数据, 8(2): 28-57 [DOI: 10.11959/j.issn.2096-0271.2022014http://dx.doi.org/10.11959/j.issn.2096-0271.2022014]

Ni L, Gao L R, Li S S, Li J and Zhang B. 2014. Edge-con-strained Markov random field classification by integrati-ng hyperspectral image with LiDAR data over urban a-reas. Journal of Applied Remote Sensing, 8(1): 085089[DOI: 10.1117/1.JRS.8.085089http://dx.doi.org/10.1117/1.JRS.8.085089]

Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z M, Gimelshein N, Antiga Let al. 2019. Pytorch: An imperative style, high-performance d-eep learning library. Advances in Neural Information Pr-ocessing Systems, 32: 8026-8037.

Prabha K A and Visalakshi N K. 2014. Improved Particle S-warm Optimization Based K-Means Clustering. 2014 Int-ernational Conference on Intelligent Computing Applica-tions, 59-63 [DOI: 10.1109/ICICA.2014.21http://dx.doi.org/10.1109/ICICA.2014.21]

Qu Y, Baghbaderani R K, Li W, Gao L R, Zhang Y X and Qi H R. 2021. Physically Constrained Transfer Learnin-g Through Shared Abundance Space for Hyperspectral I-mage Classification. IEEE Transactions on Geoscience and Remote Sensing, 59(12): 10455-10472 [DOI: 10.1109/TGRS.2020.3045790http://dx.doi.org/10.1109/TGRS.2020.3045790]

Tong Q X, Zhang B and Zhang L F. 2016. Current progress of hyperspectral remote sensing in China. Journal of Re-mote Sensing, 20(5): 689-707

童庆禧, 张兵, 张立福. 2016. 中国高光谱遥感的前沿进展. 遥感学报, 20(5): 689-707 [DOI: 10.11834/jrs.20166264http://dx.doi.org/10.11834/jrs.20166264]

Wang X H, Feng Y N, Song R X, Mu Z H and Song C M. 2022. Multi-attentive hierarchical dense fusion net for fus-ion classification of hyperspectral and LiDAR data. Info-rmation Fusion, 82: 1-18 [DOI: 10.1016/j.inffus.2021.12.008http://dx.doi.org/10.1016/j.inffus.2021.12.008].

Xia J S, Ghamisi P, Yokoya N and Iwasaki A. 2018. RandomForest Ensembles and Extended Multi-Extinction Profiles for Hyperspectral Image Classification. IEEE Transactionson Geoscience and Remote Sensing, 56(1): 202-216 [DOI

1109/TGRS.2017.2744662]

Xu X D, Li W, Ran Q, Du Q, Gao L R and Zhang B. 2018.Multisource Remote Sensing Data Classification Based onConvolutional Neural Network. IEEE Transactions on Ge-oscience and Remote Sensing, 56(2): 937-949 [DOI: 10.1109/TGRS.2017.2756851http://dx.doi.org/10.1109/TGRS.2017.2756851]

Zhang L P and Shen H F. 2016. Progress and future of remot-e sensing data fusion. Journal of Remote Sensing, 20(5):1050-1061

张良培, 沈焕锋. 2016. 遥感数据融合的进展与前瞻. 遥感学报, 20(5): 1050–1061 [DOI: 10.11834/jrs.20166243http://dx.doi.org/10.11834/jrs.20166243]

Zhang M M, Li W, Du Q, Gao L R and Zhang B. 2020. Fea-ture Extraction for Classification of Hyperspectral and L-iDAR Data Using Patch-to-Patch CNN. IEEE Transaction-s on Cybernetics, 50(1): 100-111 [DOI: 10.1109/TCYB.2018.2864670http://dx.doi.org/10.1109/TCYB.2018.2864670]

Zhang M M, Li W, Tao R, Li H C and Du Q. 2022. Informa-tion Fusion for Classification of Hyperspectral and LiD-AR Data Using IP-CNN. IEEE Transactions on Geoscien-ce and Remote Sensing, 60: 1-12 [DOI:10.1109/TGRS.2021.3093334http://dx.doi.org/10.1109/TGRS.2021.3093334]

Zhao W D，Li S S，Li A，Zhang B and Chen J. 2021. Deepfusion of hyperspectral images and multi-source remote s-ensing data for classification with convolutional neural n-etwork. National Remote Sensing Bulletin, 25(7): 1489-1502

赵伍迪, 李山山, 李安, 张兵, 陈俊. 2021. 结合深度学习的高光谱与多源遥感数据融合分类.遥感学报, 25(7): 1489-1502 [DOI：10.11834/jrs.20219117http://dx.doi.org/10.11834/jrs.20219117]

Alert me when the article has been cited

提交

Hyperspectral unmixing based on adversarial autoencoder network

Variety identification model for maize seeds using hyperspectral pixel-level information combined with convolutional neural network

CNN with local binary patterns for hyperspectral images classification

Time series classification of remote sensing data based on temporal self-attention mechanism

A new method for high precision processing of multi-system Earth observation satellite data