Convolutional neural network based single image pair method for spatiotemporal fusion

Yunfei LI; Jun LI; Lin HE

doi:10.11834/jrs.20219348

Remote Sensing Intelligent Interpretation | Views : 0 下载量: 483 CSCD: 2 更多指标

R-PDF
PDF
Export
Share
Collection
Album

Convolutional neural network based single image pair method for spatiotemporal fusion
Vol. 26, Issue 8, Pages: 1614-1623(2022)
Published： 07 August 2022 ，
DOI： 10.11834/jrs.20219348

扫描看全文

李云飞，李军，贺霖.2022.单样本对卷积神经网络遥感图像时空融合.遥感学报，26（8）： 1614-1623

Li Y F，Li J and He L. 2022. Convolutional neural network based single image pair method for spatiotemporal fusion. National Remote Sensing Bulletin， 26（8）：1614-1623
李云飞，李军，贺霖.2022.单样本对卷积神经网络遥感图像时空融合.遥感学报，26（8）： 1614-1623 DOI： 10.11834/jrs.20219348.

Li Y F，Li J and He L. 2022. Convolutional neural network based single image pair method for spatiotemporal fusion. National Remote Sensing Bulletin， 26（8）：1614-1623 DOI： 10.11834/jrs.20219348.

摘要

遥感图像时空融合是一种生成兼具高时空分辨率的合成遥感数据的技术。近年来，产生了一些基于卷积神经网络的时空融合方法。这些方法效果良好，但需要较多的图像样本对训练模型，限制了它们的应用。针对此问题，本文提出了一种单样本对卷积神经网络时空融合方法（SS-CNN）。该方法以高空间分辨率图像的波段平均图像提供的空间信息激励卷积神经网络建立高、低空间分辨率图像间的超分关系，进而利用该超分关系映射求解目标高空间分辨率图像。在实验中使用两个模拟数据集和一个真实数据集对该方法进行了测试，并与两种常用的时空融合方法做了比较。实验结果表明，SS-CNN在单样本对训练的情况下，可以较好地预测地物的物候变化和类型的变化，且在异质性高、地块破碎的区域表现良好。其不足之处在于会在地物边界上会造成轻微的模糊，将来需针对此问题做进一步改进。

Abstract

Spatiotemporal fusion is a feasible way to provide synthetic satellite images with high spatial and high temporal resolution simultaneously. In recent years

some efficient STF methods based on Convolutional Neural Networks (CNNs) have been developed. However

these methods require a significant number of training image pairs

where each pair generally consists of a high spatial resolution image and a low spatial resolution image. Such a requirement limits the applicability of STF methods to actual scenarios because image pairs for training are not widely available in many cases. To overcome this important limitation

we introduce a CNN-based single image pair method for STF of remotely sensed images. Our method

called SS-CNN

uses the spatial information provided by the average image (obtained across available spectral bands) of the high spatial resolution image to perform CNN-based Super-Resolution Mapping (SRM) between the low and high spatial resolution images. The proposed SS-CNN has been tested in experiments using two simulated and one real dataset and compared with two commonly used spatiotemporal fusion methods. The experimental results show that SS-CNN can predict the phenological changes and land cover changes well. Plus

its performance in heterogeneous areas is remarkable. The disadvantage is that it will slightly blur the boundary

which needs to be further improved in the future.

关键词

遥感时空融合遥感图像单样本对卷积神经网络

Keywords

remote sensingspatio-temporal fusionremote sensing imagessingle image pairConvolutional Neural Networks (CNN)

references

Boyte S P, Wylie B K, Rigge M B and Dahal D. 2018. Fusing MODIS with Landsat 8 data to downscale weekly normalized difference vegetation index estimates for central great basin rangelands, USA. GIScience and Remote Sensing, 55(3): 376-399 [DOI: 10.1080/15481603.2017.1382065http://dx.doi.org/10.1080/15481603.2017.1382065]

Gao F, Masek J, Schwaller M and Hall F. 2006. On the blending of the Landsat and MODIS surface reflectance: predicting daily Landsat surface reflectance. IEEE Transactions on Geoscience and Remote Sensing, 44(8): 2207-2218 [DOI: 10.1109/TGRS.2006.872081http://dx.doi.org/10.1109/TGRS.2006.872081]

Gevaert C M and García-Haro F J. 2015. A comparison of STARFM and an unmixing-based algorithm for Landsat and MODIS data fusion. Remote Sensing of Environment, 156: 34-44 [DOI: 10.1016/j.rse.2014.09.012http://dx.doi.org/10.1016/j.rse.2014.09.012]

Heimhuber V, Tulbure M G and Broich M. 2018. Addressing spatio-temporal resolution constraints in Landsat and MODIS-based mapping of large-scale floodplain inundation dynamics. Remote Sensing of Environment, 211: 307-320 [DOI: 10.1016/j.rse.2018.04.016http://dx.doi.org/10.1016/j.rse.2018.04.016]

Hilker T, Wulder M A, Coops N C, Linke J, McDermid G, Masek J G, Gao F and White J C. 2009. A new data fusion model for high spatial- and temporal-resolution mapping of forest disturbance based on Landsat and MODIS. Remote Sensing of Environment, 113(8): 1613-1627 [DOI: 10.1016/j.rse.2009.03.007http://dx.doi.org/10.1016/j.rse.2009.03.007]

Huang B, and Song H H. 2012. Spatiotemporal reflectance fusion via sparse representation. IEEE Transactions on Geoscience and Remote Sensing, 50(10): 3707-3716 [DOI: 10.1109/TGRS.2012.2186638http://dx.doi.org/10.1109/TGRS.2012.2186638]

Huang B, Zhao Y Q. 2017. Research status and prospect of spatiotemporal fusion of multi-source satellite remote sensing imagery. Acta Geodaetica et Cartographica Sinica, 46(10): 1492-1499

黄波, 赵涌泉. 2017. 多源卫星遥感影像时空融合研究的现状及展. 测绘学报, 46(10): 1492-1499 [DOI: 10.11947/j.AGCS.2017.20170376http://dx.doi.org/10.11947/j.AGCS.2017.20170376]

Ke Y H, Im J, Park S and Gong H L. 2016. Downscaling of MODIS one kilometer evapotranspiration using Landsat-8 data and machine learning approaches. Remote Sensing, 8(3): 215 [DOI: 10.3390/rs8030215http://dx.doi.org/10.3390/rs8030215]

Kingma D P and Ba J. 2015. Adam: a method for stochastic optimization//3rd International Conference on Learning Representations. San Diego: ICLR.

Knipper K R, Kustas W P, Anderson M C, Alfieri J G, Prueger J H, Hain C R, Gao F, Yang Y, McKee L G, Nieto H, Hipps L E, Alsina M M and Sanchez L. 2019. Evapotranspiration estimates derived using thermal-based satellite remote sensing and data fusion for irrigation management in California vineyards. Irrigation Science, 37(3): 431-449 [DOI: 10.1007/s00271-018-0591-yhttp://dx.doi.org/10.1007/s00271-018-0591-y]

Li D C, Li Y R, Yang W F, Ge Y Q, Han Q J, Ma L L, Chen Y H and Li X. 2018. An enhanced single-pair learning-based reflectance fusion algorithm with spatiotemporally extended training samples. Remote Sensing, 10(8): 1207 [DOI: 10.3390/rs10081207http://dx.doi.org/10.3390/rs10081207]

Liao C H, Wang J F, Dong T F, Shang J L, Liu J G and Song Y. 2019. Using spatio-temporal fusion of Landsat-8 and MODIS data to derive phenology, biomass and yield estimates for corn and soybean. Science of the Total Environment, 650: 1707-1721 [DOI: 10.1016/j.scitotenv.2018.09.308http://dx.doi.org/10.1016/j.scitotenv.2018.09.308]

Liu X, Deng X W, Chanussot J, Hong D F and Zhao B J. 2019. StfNet: a two-stream convolutional neural network for spatiotemporal image fusion. IEEE Transactions on Geoscience and Remote Sensing, 57(9): 6552-6564 [DOI: 10.1109/TGRS.2019.2907310http://dx.doi.org/10.1109/TGRS.2019.2907310]

Nduati E, Sofue Y, Matniyaz A, Park J G, Yang W and Kondoh A. 2019. Cropland mapping using fusion of multi-sensor data in a complex Urban/Peri-Urban area. Remote Sensing, 2(11): 207 [DOI: 10.3390/rs11020207http://dx.doi.org/10.3390/rs11020207]

Pan Y Q, Shen F and Wei X D. 2018. Fusion of landsat-8/OLI and GOCI data for hourly mapping of suspended particulate matter at high spatial resolution: a case study in the yangtze (Changjiang) estuary. Remote Sensing, 2(10): 158 [DOI: 10.3390/rs10020158http://dx.doi.org/10.3390/rs10020158]

Park B, Windham W R, Lawrence K C and Smith D P. 2007. Contaminant classification of poultry hyperspectral imagery using a spectral angle mapper algorithm. Biosystems Engineering, 96(3): 323-333 [DOI: 10.1016/j.biosystemseng.2006.11.012http://dx.doi.org/10.1016/j.biosystemseng.2006.11.012]

Renza D, Martinez E and Arquero A. 2013. A new approach to change detection in multispectral images by means of ERGAS index. IEEE Geoscience and Remote Sensing Letters, 10(1): 76-80 [DOI: 10.1109/LGRS.2012.2193372http://dx.doi.org/10.1109/LGRS.2012.2193372]

Song H H and Huang B. 2013. Spatiotemporal satellite image fusion through one-pair image learning. IEEE Transactions on Geoscience and Remote Sensing, 51(4): 1883-1896 [DOI: 10.1109/TGRS.2012.2213095http://dx.doi.org/10.1109/TGRS.2012.2213095]

Song H H, Liu Q S, Wang G J, Hang R L and Huang B. 2018. Spatiotemporal satellite image fusion using deep convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3): 821-829 [DOI: 10.1109/JSTARS.2018.2797894http://dx.doi.org/10.1109/JSTARS.2018.2797894]

Wang Z, Bovik A C, Sheikh H R and Simoncelli E P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612 [DOI: 10.1109/TIP.2003.819861http://dx.doi.org/10.1109/TIP.2003.819861]

Weng Q H, Peng F and Feng G. 2014. Generating daily land surface temperature at Landsat resolution by fusing Landsat and MODIS data. Remote Sensing of Environment, 145(8): 55-67 [DOI: 10.1016/j.rse.2014.02.003http://dx.doi.org/10.1016/j.rse.2014.02.003]

Wu B, Huang B and Zhang L P. 2015. An error-bound-regularized sparse coding for spatiotemporal reflectance fusion. IEEE Transactions on Geoscience and Remote Sensing, 53(12): 6791-6803 [DOI: 10.1109/TGRS.2015.2448100http://dx.doi.org/10.1109/TGRS.2015.2448100]

Wu M Q, Huang W J, Niu Z and Wang C Y. 2015. Generating daily synthetic Landsat imagery by combining Landsat and MODIS data. Sensors, 15(9): 24002-24025 [DOI: 10.3390/s150924002http://dx.doi.org/10.3390/s150924002]

Zhang M, Lin H, Wang G, Sun H and Cai Y T. 2019. Estimation of vegetation productivity using a Landsat 8 time series in a heavily Urbanized area, central China. Remote Sensing, 11(2): 133 [DOI: 10.3390/rs11020133http://dx.doi.org/10.3390/rs11020133]

Zhang W, Li A N, Jin H A, Bian J H, Zhang Z J, Lei G B, Qin Z H and Huang C Q. 2013. An enhanced spatial and temporal data fusion model for fusing Landsat and MODIS surface reflectance to generate high temporal Landsat-like data. Remote Sensing, 5(10): 5346-5368 [DOI: 10.3390/rs5105346http://dx.doi.org/10.3390/rs5105346]

Zhao C Y, Gao X B, Emery W J, Ying W and Li J. 2018. An integrated spatio-spectral-temporal sparse representation method for fusing remote-sensing images with different resolutions. IEEE Transactions on Geoscience and Remote Sensing, 56(6): 3358-3370 [DOI: 10.1109/TGRS.2018.2798663http://dx.doi.org/10.1109/TGRS.2018.2798663]

Zhu X L, Chen J, Gao F, Chen X H and Masek J G. 2010. An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions. Remote Sensing of Environment, 114(11): 2610-2623 [DOI: 10.1016/j.rse.2010.05.032http://dx.doi.org/10.1016/j.rse.2010.05.032]

Zhu X L, Helmer E H,, Gao F, Liu D S, Chen J and Lefsky M A. 2016. A flexible spatiotemporal method for fusing satellite images with different resolutions. Remote Sensing of Environment, 172: 165-177 [DOI: 10.1016/j.rse.2015.11.016http://dx.doi.org/10.1016/j.rse.2015.11.016]

Alert me when the article has been cited

提交

Net primary productivity estimation of Dongting Lake wetland

A review of spatiotemporal fusion methods for remotely sensed land surface temperature

Transparent earth-observing： Exploring the new generation of earth observation system

Transferring deep convolutional neural network models for generalization mapping of autumn crops

Grid A*：A fast path planning algorithm for air-ground cooperative response of emergency rescue in the field