Convolutional neural network based single image pair method for spatiotemporal fusion

Yunfei LI; Jun LI; Lin HE

doi:10.11834/jrs.20219348

Remote Sensing Intelligent Interpretation | Views : 0 下载量: 920 CSCD: 3

R-PDF
PDF
Export
Share
Collection
Album

Convolutional neural network based single image pair method for spatiotemporal fusion
Vol. 26, Issue 8, Pages: 1614-1623(2022)
Received：18 October 2019，

Published：07 August 2022
DOI： 10.11834/jrs.20219348
稿件说明：

移动端阅览

李云飞，李军，贺霖.2022.单样本对卷积神经网络遥感图像时空融合.遥感学报，26（8）： 1614-1623 DOI： 10.11834/jrs.20219348.

Li Y F，Li J and He L. 2022. Convolutional neural network based single image pair method for spatiotemporal fusion. National Remote Sensing Bulletin， 26（8）：1614-1623 DOI： 10.11834/jrs.20219348.

摘要

遥感图像时空融合是一种生成兼具高时空分辨率的合成遥感数据的技术。近年来，产生了一些基于卷积神经网络的时空融合方法。这些方法效果良好，但需要较多的图像样本对训练模型，限制了它们的应用。针对此问题，本文提出了一种单样本对卷积神经网络时空融合方法（SS-CNN）。该方法以高空间分辨率图像的波段平均图像提供的空间信息激励卷积神经网络建立高、低空间分辨率图像间的超分关系，进而利用该超分关系映射求解目标高空间分辨率图像。在实验中使用两个模拟数据集和一个真实数据集对该方法进行了测试，并与两种常用的时空融合方法做了比较。实验结果表明，SS-CNN在单样本对训练的情况下，可以较好地预测地物的物候变化和类型的变化，且在异质性高、地块破碎的区域表现良好。其不足之处在于会在地物边界上会造成轻微的模糊，将来需针对此问题做进一步改进。

Abstract

Spatiotemporal fusion is a feasible way to provide synthetic satellite images with high spatial and high temporal resolution simultaneously. In recent years

some efficient STF methods based on Convolutional Neural Networks (CNNs) have been developed. However

these methods require a significant number of training image pairs

where each pair generally consists of a high spatial resolution image and a low spatial resolution image. Such a requirement limits the applicability of STF methods to actual scenarios because image pairs for training are not widely available in many cases. To overcome this important limitation

we introduce a CNN-based single image pair method for STF of remotely sensed images. Our method

called SS-CNN

uses the spatial information provided by the average image (obtained across available spectral bands) of the high spatial resolution image to perform CNN-based Super-Resolution Mapping (SRM) between the low and high spatial resolution images. The proposed SS-CNN has been tested in experiments using two simulated and one real dataset and compared with two commonly used spatiotemporal fusion methods. The experimental results show that SS-CNN can predict the phenological changes and land cover changes well. Plus

its performance in heterogeneous areas is remarkable. The disadvantage is that it will slightly blur the boundary

which needs to be further improved in the future.

关键词

Keywords

references

Boyte S P , Wylie B K , Rigge M B and Dahal D . 2018 . Fusing MODIS with Landsat 8 data to downscale weekly normalized difference vegetation index estimates for central great basin rangelands, USA . GIScience and Remote Sensing , 55 ( 3 ): 376 - 399 [ DOI: 10.1080/15481603.2017.1382065 http://dx.doi.org/10.1080/15481603.2017.1382065 ]

Gao F , Masek J , Schwaller M and Hall F . 2006 . On the blending of the Landsat and MODIS surface reflectance: predicting daily Landsat surface reflectance . IEEE Transactions on Geoscience and Remote Sensing , 44 ( 8 ): 2207 - 2218 [ DOI: 10.1109/TGRS.2006.872081 http://dx.doi.org/10.1109/TGRS.2006.872081 ]

Gevaert C M and García-Haro F J . 2015 . A comparison of STARFM and an unmixing-based algorithm for Landsat and MODIS data fusion . Remote Sensing of Environment , 156 : 34 - 44 [ DOI: 10.1016/j.rse.2014.09.012 http://dx.doi.org/10.1016/j.rse.2014.09.012 ]

Heimhuber V , Tulbure M G and Broich M . 2018 . Addressing spatio-temporal resolution constraints in Landsat and MODIS-based mapping of large-scale floodplain inundation dynamics . Remote Sensing of Environment , 211 : 307 - 320 [ DOI: 10.1016/j.rse.2018.04.016 http://dx.doi.org/10.1016/j.rse.2018.04.016 ]

Hilker T , Wulder M A , Coops N C , Linke J , McDermid G , Masek J G , Gao F and White J C . 2009 . A new data fusion model for high spatial- and temporal-resolution mapping of forest disturbance based on Landsat and MODIS . Remote Sensing of Environment , 113 ( 8 ): 1613 - 1627 [ DOI: 10.1016/j.rse.2009.03.007 http://dx.doi.org/10.1016/j.rse.2009.03.007 ]

Huang B , and Song H H . 2012 . Spatiotemporal reflectance fusion via sparse representation . IEEE Transactions on Geoscience and Remote Sensing , 50 ( 10 ): 3707 - 3716 [ DOI: 10.1109/TGRS.2012.2186638 http://dx.doi.org/10.1109/TGRS.2012.2186638 ]

Huang B , Zhao Y Q . 2017 . Research status and prospect of spatiotemporal fusion of multi-source satellite remote sensing imagery . Acta Geodaetica et Cartographica Sinica , 46 ( 10 ): 1492 - 1499

黄波 , 赵涌泉 . 2017 . 多源卫星遥感影像时空融合研究的现状及展 . 测绘学报 , 46 ( 10 ): 1492 - 1499 [ DOI: 10.11947/j.AGCS.2017.20170376 http://dx.doi.org/10.11947/j.AGCS.2017.20170376 ]

Ke Y H , Im J , Park S and Gong H L . 2016 . Downscaling of MODIS one kilometer evapotranspiration using Landsat-8 data and machine learning approaches . Remote Sensing , 8 ( 3 ): 215 [ DOI: 10.3390/rs8030215 http://dx.doi.org/10.3390/rs8030215 ]

Kingma D P and Ba J . 2015 . Adam: a method for stochastic optimization // 3rd International Conference on Learning Representations . San Diego : ICLR .

Knipper K R , Kustas W P , Anderson M C , Alfieri J G , Prueger J H , Hain C R , Gao F , Yang Y , McKee L G , Nieto H , Hipps L E , Alsina M M and Sanchez L . 2019 . Evapotranspiration estimates derived using thermal-based satellite remote sensing and data fusion for irrigation management in California vineyards . Irrigation Science , 37 ( 3 ): 431 - 449 [ DOI: 10.1007/s00271-018-0591-y http://dx.doi.org/10.1007/s00271-018-0591-y ]

Li D C , Li Y R , Yang W F , Ge Y Q , Han Q J , Ma L L , Chen Y H and Li X . 2018 . An enhanced single-pair learning-based reflectance fusion algorithm with spatiotemporally extended training samples . Remote Sensing , 10 ( 8 ): 1207 [ DOI: 10.3390/rs10081207 http://dx.doi.org/10.3390/rs10081207 ]

Liao C H , Wang J F , Dong T F , Shang J L , Liu J G and Song Y . 2019 . Using spatio-temporal fusion of Landsat-8 and MODIS data to derive phenology, biomass and yield estimates for corn and soybean . Science of the Total Environment , 650 : 1707 - 1721 [ DOI: 10.1016/j.scitotenv.2018.09.308 http://dx.doi.org/10.1016/j.scitotenv.2018.09.308 ]

Liu X , Deng X W , Chanussot J , Hong D F and Zhao B J . 2019 . StfNet : a two-stream convolutional neural network for spatiotemporal image fusion . IEEE Transactions on Geoscience and Remote Sensing , 57 ( 9 ): 6552 - 6564 [ DOI: 10.1109/TGRS.2019.2907310 http://dx.doi.org/10.1109/TGRS.2019.2907310 ]

Nduati E , Sofue Y , Matniyaz A , Park J G , Yang W and Kondoh A . 2019 . Cropland mapping using fusion of multi-sensor data in a complex Urban/Peri-Urban area . Remote Sensing , 2 ( 11 ): 207 [ DOI: 10.3390/rs11020207 http://dx.doi.org/10.3390/rs11020207 ]

Pan Y Q , Shen F and Wei X D . 2018 . Fusion of landsat-8/OLI and GOCI data for hourly mapping of suspended particulate matter at high spatial resolution: a case study in the yangtze (Changjiang) estuary . Remote Sensing , 2 ( 10 ): 158 [ DOI: 10.3390/rs10020158 http://dx.doi.org/10.3390/rs10020158 ]

Park B , Windham W R , Lawrence K C and Smith D P . 2007 . Contaminant classification of poultry hyperspectral imagery using a spectral angle mapper algorithm . Biosystems Engineering , 96 ( 3 ): 323 - 333 [ DOI: 10.1016/j.biosystemseng.2006.11.012 http://dx.doi.org/10.1016/j.biosystemseng.2006.11.012 ]

Renza D , Martinez E and Arquero A . 2013 . A new approach to change detection in multispectral images by means of ERGAS index . IEEE Geoscience and Remote Sensing Letters , 10 ( 1 ): 76 - 80 [ DOI: 10.1109/LGRS.2012.2193372 http://dx.doi.org/10.1109/LGRS.2012.2193372 ]

Song H H and Huang B . 2013 . Spatiotemporal satellite image fusion through one-pair image learning . IEEE Transactions on Geoscience and Remote Sensing , 51 ( 4 ): 1883 - 1896 [ DOI: 10.1109/TGRS.2012.2213095 http://dx.doi.org/10.1109/TGRS.2012.2213095 ]

Song H H , Liu Q S , Wang G J , Hang R L and Huang B . 2018 . Spatiotemporal satellite image fusion using deep convolutional neural networks . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 11 ( 3 ): 821 - 829 [ DOI: 10.1109/JSTARS.2018.2797894 http://dx.doi.org/10.1109/JSTARS.2018.2797894 ]

Wang Z , Bovik A C , Sheikh H R and Simoncelli E P . 2004 . Image quality assessment: from error visibility to structural similarity . IEEE Transactions on Image Processing , 13 ( 4 ): 600 - 612 [ DOI: 10.1109/TIP.2003.819861 http://dx.doi.org/10.1109/TIP.2003.819861 ]

Weng Q H , Peng F and Feng G . 2014 . Generating daily land surface temperature at Landsat resolution by fusing Landsat and MODIS data . Remote Sensing of Environment , 145 ( 8 ): 55 - 67 [ DOI: 10.1016/j.rse.2014.02.003 http://dx.doi.org/10.1016/j.rse.2014.02.003 ]

Wu B , Huang B and Zhang L P . 2015 . An error-bound-regularized sparse coding for spatiotemporal reflectance fusion . IEEE Transactions on Geoscience and Remote Sensing , 53 ( 12 ): 6791 - 6803 [ DOI: 10.1109/TGRS.2015.2448100 http://dx.doi.org/10.1109/TGRS.2015.2448100 ]

Wu M Q , Huang W J , Niu Z and Wang C Y . 2015 . Generating daily synthetic Landsat imagery by combining Landsat and MODIS data . Sensors , 15 ( 9 ): 24002 - 24025 [ DOI: 10.3390/s150924002 http://dx.doi.org/10.3390/s150924002 ]

Zhang M , Lin H , Wang G , Sun H and Cai Y T . 2019 . Estimation of vegetation productivity using a Landsat 8 time series in a heavily Urbanized area, central China . Remote Sensing , 11 ( 2 ): 133 [ DOI: 10.3390/rs11020133 http://dx.doi.org/10.3390/rs11020133 ]

Zhang W , Li A N , Jin H A , Bian J H , Zhang Z J , Lei G B , Qin Z H and Huang C Q . 2013 . An enhanced spatial and temporal data fusion model for fusing Landsat and MODIS surface reflectance to generate high temporal Landsat-like data . Remote Sensing , 5 ( 10 ): 5346 - 5368 [ DOI: 10.3390/rs5105346 http://dx.doi.org/10.3390/rs5105346 ]

Zhao C Y , Gao X B , Emery W J , Ying W and Li J . 2018 . An integrated spatio-spectral-temporal sparse representation method for fusing remote-sensing images with different resolutions . IEEE Transactions on Geoscience and Remote Sensing , 56 ( 6 ): 3358 - 3370 [ DOI: 10.1109/TGRS.2018.2798663 http://dx.doi.org/10.1109/TGRS.2018.2798663 ]

Zhu X L , Chen J , Gao F , Chen X H and Masek J G . 2010 . An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions . Remote Sensing of Environment , 114 ( 11 ): 2610 - 2623 [ DOI: 10.1016/j.rse.2010.05.032 http://dx.doi.org/10.1016/j.rse.2010.05.032 ]

Zhu X L , Helmer E H ,, Gao F , Liu D S, Chen J and Lefsky M A. 2016 . A flexible spatiotemporal method for fusing satellite images with different resolutions . Remote Sensing of Environment, 172 : 165 - 177 [ DOI: 10.1016/j.rse.2015.11.016 http://dx.doi.org/10.1016/j.rse.2015.11.016 ]

Alert me when the article has been cited

提交

Spatial reconstruction-aided spatio-temporal fusion of NDVI images

A review of SAR target recognition methods fusing electromagnetic scattering features

Net primary productivity estimation of Dongting Lake wetland

A review of spatiotemporal fusion methods for remotely sensed land surface temperature

Landslide susceptibility assessment of the Luding 9.5 earthquake area by fusing LT-1 ascending and descending ground deformation observations