Detail focused fusion of hyperspectral and multispectral images

FANG Shuai; YAN Mingchang; ZHANG Jing; CAO Yang

doi:10.11834/jrs.20210287

Models and Methods | Views : 0 下载量: 2406 CSCD: 2

R-PDF
PDF
Export
Share
Collection
Album

Detail focused fusion of hyperspectral and multispectral images
Vol. 26, Issue 12, Pages: 2594-2602(2022)
Received：22 July 2020，

Published：07 December 2022
DOI： 10.11834/jrs.20210287
稿件说明：

移动端阅览

方帅，闫明畅，张晶，曹洋.2022.基于细节关注的高光谱与多光谱图像融合算法.遥感学报，26（12）： 2594-2602 DOI： 10.11834/jrs.20210287.

Fang S，Yan M C，Zhang J and Cao Y. 2022. Detail focused fusion of hyperspectral and multispectral images. National Remote Sensing Bulletin， 26（12）：2594-2602 DOI： 10.11834/jrs.20210287.

摘要

低分辨率高光谱图像（LR-HSI）与高分辨率多光谱图像（HR-MSI）融合技术，广泛用于解决图像空间分辨率与光谱分辨率无法同时保持高水平的矛盾。从融合效果上分析，现有算法的空间重建误差与光谱重建误差都主要体现在边缘和细节区域。因此，本文提出了基于细节关注的字典构建和图像重建的融合算法。在光谱特性保持方面，由于图像邻近效应导致在细节区域光谱分布复杂多样，本文提出对图像层和细节层分别进行字典学习。在空间特性增强方面，提出了细节感知误差和边缘方向自适应全变分约束，并将其与局部低秩约束结合在同一个融合框架用来估计稀疏系数。消融实验证明细节感知误差、细节感知字典和EADTV正则项的引入，在Pavia University数据集上分别将整体精度（PNSR）提升了0.0263、0.289和0.4121，光谱（SAM）精度分别提高了0.2%、4.6%和4.3%。在Pavia University数据集和Indian Pine数据集上的对比实验证明，本文算法相对于次优解，PNSR分别提高了0.4945和0.2345，实验结果印证了本文算法有效地提高了融合精度。通过实验对比，本文提出算法的融合结果在空间特性与光谱特性方面较其他算法有明显提升。

Abstract

Hyperspectral image (HSI) and multispectral image (MSI) are two types of images widely used in the field of remote sensing. These images are useful in certain applications

such as environmental monitoring

target detection

and mineral exploration. HSI contains a large amount of spectral information. Photons are typically collected in a larger spatial area on the sensor to ensure a sufficiently high signal-to-noise ratio (SNR). Accordingly

the HSI spatial resolution is much lower compared with MSI. This low spatial resolution greatly affects the practicality of HSI. Accordingly

fusing a low-spatial resolution HSI (LR-HSI) with a high-spatial resolution MSI (HR-MSI) in the same scene to obtain a high-resolution HSI (HR-HSI) is a method for solving such problems

which resolves the contradiction that the spatial resolution and the spectral resolution cannot simultaneously maintain a high level. From the analysis of fusion effect

the spatial and spectral reconstruction errors of the existing algorithms are mainly reflected in the edge and detail areas.

The method proposed in this work was a fusion algorithm for dictionary construction and image reconstruction based on detail attention. In terms of maintaining spectral characteristics

the spectral distribution in the detail area is complex and diverse because of the proximity effect of the image. This work proposes to perform dictionary learning on the image and detail layers. The detail perception error terms and a constraint of edge adaptive directional total variation are proposed for spatial characteristic enhancement

which is combined with a local low rank constraint in the same fusion framework to estimate the sparse coefficient.

Experiments were conducted on two datasets

namely

Pavia University and Indian Pine

to verify the effectiveness of the proposed method. The quantitative evaluation metrics contain peak SNR

relative dimensionless global error in synthesis

spectral angle map

and universal image quality index. Based on the experimental comparison

the fusion result of the algorithm proposed in this work is significantly improved compared with those of the other algorithms in terms of spatial and spectral characteristics.

This work uses dictionary learning to propose a fusion algorithm for dictionary construction and image reconstruction with attention to details through the analysis of the existing hyperspectral and multispectral image fusion algorithms. A hierarchical dictionary learning algorithm is proposed to address the problem of large reconstruction error in the detail part of the existing algorithms. The detail perception error term and the direction adaptive full variational regularization term are used to improve the spectral dictionary solution and coefficient estimation

respectively. The result of the fusion is the error in the spectral characteristics and spatial texture of the detail

which achieves an accurate representation of the edge detail.

关键词

Keywords

references

Afonso M V , Bioucas-Dias J M and Figueiredo M A T . 2011 . An augmented Lagrangian approach to the constrained optimization formulation of imaging inverse problems . IEEE Transactions on Image Processing , 20 ( 3 ): 681 - 695 [ DOI: 10.1109/TIP.2010.2076294 http://dx.doi.org/10.1109/TIP.2010.2076294 ]

Alparone L , Wald L , Chanussot J , Thomas C , Gamba P and Bruce L M . 2007 . Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest . IEEE Transactions on Geoscience and Remote Sensing , 45 ( 10 ): 3012 - 3021 [ DOI: 10.1109/TGRS.2007.904923 http://dx.doi.org/10.1109/TGRS.2007.904923 ]

Bayram İ and Kamasak M E . 2012 . A directional total variation // 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO) . Bucharest : IEEE: 265 - 269 [ DOI: 10.1109/LSP.2012.2220349 http://dx.doi.org/10.1109/LSP.2012.2220349 ]

Dian R W , Fang L Y and Li S T . 2017 . Hyperspectral image super-resolution via non-local sparse tensor factorization // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu : IEEE: 3862 - 3871 [ DOI: 10.1109/CVPR.2017.411 http://dx.doi.org/10.1109/CVPR.2017.411 ]

Dian R W , Li S T , Fang L Y and Bioucas-Dias J . 2018 . Hyperspectral image super-resolution via local low-rank and sparse representations // 2018 IEEE International Geoscience and Remote Sensing Symposium . Valencia : IEEE: 4003 - 4006 [ DOI: 10.1109/IGARSS.2018.8519213 http://dx.doi.org/10.1109/IGARSS.2018.8519213 ]

Dian R W , Li S T , Fang L Y , Lu T and Bioucas-Dias J M . 2020 . Nonlocal sparse tensor factorization for semiblind hyperspectral and multispectral image fusion . IEEE Transactions on Cybernetics , 50 ( 10 ): 4469 - 4480 [ DOI: 10.1109/TCYB.2019.2951572 http://dx.doi.org/10.1109/TCYB.2019.2951572 ]

Dong W S , Fu F Z , Shi G M , Cao X , Wu J J , Li G Y and Li X . 2016 . Hyperspectral image super-resolution via non-negative structured sparse representation . IEEE Transactions on Image Processing , 25 ( 5 ): 2337 - 2352 [ DOI: 10.1109/TIP.2016.2542360 http://dx.doi.org/10.1109/TIP.2016.2542360 ]

Dong W S , Li X , Zhang L and Shi G M . 2011 . Sparsity-based image denoising via dictionary learning and structural clustering // Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition . Colorado Springs : IEEE: 457 - 464 [ DOI: 10.1109/CVPR.2011.5995478 http://dx.doi.org/10.1109/CVPR.2011.5995478 ]

Han X H , Shi B X and Zheng Y Q . 2018 . Self-similarity constrained sparse representation for hyperspectral image super-resolution . IEEE Transactions on Image Processing , 27 ( 11 ): 5625 - 5637 [ DOI: 10.1109/TIP.2018.2855418 http://dx.doi.org/10.1109/TIP.2018.2855418 ]

Hardie R C , Eismann M T and Wilson G L . 2004 . MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor . IEEE Transactions on Image Processing , 13 ( 9 ): 1174 - 1184 [ DOI: 10.1109/TIP.2004.829779 http://dx.doi.org/10.1109/TIP.2004.829779 ]

Kanatsoulis C I , Fu X , Sidiropoulos N D and Ma W K . 2018 . Hyperspectral super-resolution: a coupled tensor factorization approach . IEEE Transactions on Signal Processing , 66 ( 24 ): 6503 - 6517 [ DOI: 10.1109/TSP.2018.2876362 http://dx.doi.org/10.1109/TSP.2018.2876362 ]

Li S T ， Li C Y and Kang X D . 2021 . Development status and future prospects of multi-source remote sensing image fusion . National Remote Sensing Bulletin ， 25 （ 1 ）： 148 - 166

李树涛，李聪妤，康旭东 . 2021 . 多源遥感图像融合发展现状与未来展望 . 遥感学报， 25 ( 1 ): 148 - 166 [ DOI: 10.11834/jrs.20210259 http://dx.doi.org/10.11834/jrs.20210259 ]

Li X H , Shen H F , Zhang L P , Zhang H Y , Yuan Q Q and Yang G . 2014 . Recovering quantitative remote sensing products contaminated by thick clouds and shadows using multitemporal dictionary learning . IEEE Transactions on Geoscience and Remote Sensing , 52 ( 11 ): 7086 - 7098 [ DOI: 10.1109/TGRS.2014.2307354 http://dx.doi.org/10.1109/TGRS.2014.2307354 ]

Lin C H , Ma F , Chi C Y and Hsieh C H . 2018 . A convex optimization-based coupled nonnegative matrix factorization algorithm for hyperspectral and multispectral data fusion . IEEE Transactions on Geoscience and Remote Sensing , 56 ( 3 ): 1652 - 1667 [ DOI: 10.1109/TGRS.2017.2766080 http://dx.doi.org/10.1109/TGRS.2017.2766080 ]

Lorente D , Aleixos N , Gómez-Sanchis J , Cubero S , García-Navarrete O L and Blasco J . 2012 . Recent advances and applications of hyperspectral imaging for fruit and vegetable quality assessment . Food and Bioprocess Technology , 5 ( 4 ): 1121 - 1142 [ DOI: 10.1007/s11947-011-0725-1 http://dx.doi.org/10.1007/s11947-011-0725-1 ]

Meng X C , Sun W W , Ren K , Yang G , Shao F and Fu R D . 2020 . Spatial-spectral fusion of GF-5/GF-1 remote sensing images based on multiresolution analysis . Journal of Remote Sensing . 24 ( 4 ): 379 - 387

孟祥超 , 孙伟伟 , 任凯 , 杨刚 , 邵枫 , 符冉迪 . 2020 . 基于多分辨率分析的GF-5和GF-1遥感影像空-谱融合 . 遥感学报 , 24 ( 4 ): 379 - 387 [ DOI: 10.11834/jrs.20209214 http://dx.doi.org/10.11834/jrs.20209214 ]

Palsson F , Sveinsson J R and Ulfarsson M O . 2017 . Multispectral and hyperspectral image fusion using a 3-D-convolutional neural network . IEEE Geoscience and Remote Sensing Letters , 14 ( 5 ): 639 - 643 [ DOI: 10.1109/LGRS.2017.2668299 http://dx.doi.org/10.1109/LGRS.2017.2668299 ]

Simões M , Bioucas-Dias J , Almeida L B and Chanussot J . 2015 . A convex formulation for hyperspectral image superresolution via subspace-based regularization . IEEE Transactions on Geoscience and Remote Sensing , 53 ( 6 ): 3373 - 3388 [ DOI: 10.1109/TGRS.2014.2375320 http://dx.doi.org/10.1109/TGRS.2014.2375320 ]

Veganzones M A , Simões M , Licciardi G , Yokoya N , Bioucas-Dias J M and Chanussot J . 2016 . Hyperspectral super-resolution of locally low rank images from complementary multisource data . IEEE Transactions on Image Processing , 25 ( 1 ): 274 - 288 [ DOI: 10.1109/TIP.2015.2496263 http://dx.doi.org/10.1109/TIP.2015.2496263 ]

Wald L . 2000 . Quality of high resolution synthesised images: is there a simple criterion? //Proceedings of the 3rd Conference “Fusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images”. Sophia Antipolis : HAL: 99 - 103

Wang X Y , Zhong Y F , Zhang L P and Xu Y Y . 2017 . Spatial group sparsity regularized nonnegative matrix factorization for hyperspectral unmixing . IEEE Transactions on Geoscience and Remote Sensing , 55 ( 11 ): 6287 - 6304 [ DOI: 10.1109/TGRS.2017.2724944 http://dx.doi.org/10.1109/TGRS.2017.2724944 ]

Wang Z and Bovik A C . 2002 . A universal image quality index . IEEE Signal Processing Letters , 9 ( 3 ): 81 - 84 [ DOI: 10.1109/97.995823 http://dx.doi.org/10.1109/97.995823 ]

Wei Q , Dobigeon N and Tourneret J Y . 2014 . Bayesian fusion of hyperspectral and multispectral images // Proceedings of 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Florence : IEEE: 3176 - 3180 [DOI: 10.1109/ICASSP.2014. 6854186 ]

Wu R Y , Ma W K , Fu X and Li Q . 2020 . Hyperspectral super-resolution via global-local low-rank matrix estimation . IEEE Transactions on Geoscience and Remote Sensing , 58 ( 10 ): 7125 - 7140 [ DOI: 10.1109/TGRS.2020.2979908 http://dx.doi.org/10.1109/TGRS.2020.2979908 ]

Xu G H , Liu Q H , Chen L F and Liu L Y . 2016 . Remote sensing for China’s sustainable development: opportunities and challenges . Journal of Remote Sensing , 20 ( 5 ): 679 - 688

徐冠华 , 柳钦火 , 陈良富 , 刘良云 . 2016 . 遥感与中国可持续发展: 机遇和挑战 . 遥感学报 , 20 ( 5 ): 679 - 688 [ DOI: 10.11834/jrs.20166308 http://dx.doi.org/10.11834/jrs.20166308 ]

Yang J X , Zhao Y Q and Chan J C W . 2018 . Hyperspectral and multispectral image fusion via deep two-branches convolutional neural network . Remote Sensing , 10 ( 5 ): 800 [ DOI: 10.3390/rs10050800 http://dx.doi.org/10.3390/rs10050800 ]

Yokoya N , Grohnfeldt C and Chanussot J . 2017 . Hyperspectral and multispectral data fusion: a comparative review of the recent literature . IEEE Geoscience and Remote Sensing Magazine , 5 ( 2 ): 29 - 56 [ DOI: 10.1109/MGRS.2016.2637824 http://dx.doi.org/10.1109/MGRS.2016.2637824 ]

Yokoya N , Yairi T and Iwasaki A . 2012 . Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion . IEEE Transactions on Geoscience and Remote Sensing , 50 ( 2 ): 528 - 537 [ DOI: 10.1109/TGRS.2011.2161320 http://dx.doi.org/10.1109/TGRS.2011.2161320 ]

Zhang B . 2016 . Advancement of hyperspectral image processing and information extraction . Journal of Remote Sensing , 20 ( 5 ): 1062 - 1090

张兵 . 2016 . 高光谱图像处理与信息提取前沿 . 遥感学报 , 20 ( 5 ): 1062 - 1090 [ DOI: 10.11834/jrs.20166179 http://dx.doi.org/10.11834/jrs.20166179 ]

Zhang H and Wang Y Q . 2013 . Edge adaptive directional total variation . The Journal of Engineering , 2013 ( 11 ): 61 - 62 [ DOI: 10.1049/joe.2013.0116 http://dx.doi.org/10.1049/joe.2013.0116 ]

Alert me when the article has been cited

提交

Class-independent domain adaptation for hyperspectral image classification

Spectral-frequency domain attribute pattern fusion for hyperspectral image change detection

Hyperspectral remote sensing image classification based on multidirectional adaptive aware network

生成对抗网络的无监督高光谱解混

Neural network generation of adaptive filter and new applications in remote sensing image processing