时空遥感云计算平台PIE-Engine Studio的研究与应用

程伟; 钱晓明; 李世卫; 马海波; 刘东升; 刘富乾; 梁军龙; 胡举

doi:10.11834/jrs.20211248

遥感大数据云计算 | 浏览量 : 0 下载量: 744 CSCD: 4 更多指标

PDF
导出
分享
收藏
专辑

时空遥感云计算平台PIE-Engine Studio的研究与应用
Research and application of PIE-Engine Studio for spatiotemporal remote sensing cloud computing platform
2022年26卷第2期页码：335-347
纸质出版日期： 2022-02-07 ，
DOI： 10.11834/jrs.20211248

扫描看全文

程伟，钱晓明，李世卫，马海波，刘东升，刘富乾，梁军龙，胡举.2022.时空遥感云计算平台PIE-Engine Studio的研究与应用.遥感学报，26（2）： 335-347

Cheng W， Qian X M， Li S W，Ma H B， Liu D S， Liu F Q， Liang J L and Hu J. 2022. Research and application of PIE-Engine Studio for spatiotemporal remote sensing cloud computing platform. National Remote Sensing Bulletin， 26（2）：335-347
程伟，钱晓明，李世卫，马海波，刘东升，刘富乾，梁军龙，胡举.2022.时空遥感云计算平台PIE-Engine Studio的研究与应用.遥感学报，26（2）： 335-347 DOI： 10.11834/jrs.20211248.

Cheng W， Qian X M， Li S W，Ma H B， Liu D S， Liu F Q， Liang J L and Hu J. 2022. Research and application of PIE-Engine Studio for spatiotemporal remote sensing cloud computing platform. National Remote Sensing Bulletin， 26（2）：335-347 DOI： 10.11834/jrs.20211248.

摘要

随着遥感大数据时代的到来，为快速处理和分析海量遥感数据，国内外涌现了众多遥感云计算平台，使得全球尺度、长时间序列遥感数据的快速分析和应用成为可能。本文在分析国内外遥感云计算平台现状的基础上，针对大数据时代国内缺少功能完备的遥感云计算平台，且国外遥感云计算平台对国产卫星数据支持不足等问题，基于容器云技术，构建了包含国产卫星数据且集数据、算力和技术于一体的时空遥感云计算平台PIE（Pixel Information Expert）-Engine Studio，实现了脚本驱动的遥感数据的按需获取以及海量数据的快速处理。采用Landsat 8数据，以生长季植被指数NDVI（Normalized Difference Vegetation Index）的计算为例，对比了本平台与GEE（Google Earth Engine）的数据处理能力。结果表明，由于计算资源的限制，本平台的计算和导出时间均比GEE稍长，但计算结果的空间分布一致，其中近68%的值均分布在（0.48，0.77），且二者差值的95.33%集中在（-0.13，0.13），结果较为可信。因此，本文构建的基于共享、开放的中国自主遥感云计算平台PIE-Engine Studio，可为地球科学领域的研究提供数据和算力支持，将有助于推进中国遥感云计算平台的发展进程，推动国产卫星数据在云计算平台上的应用。

Abstract

With the arrival of remote sensing big data era

numerous remote sensing cloud computing platforms have emerged inland and overseas to rapidly process and analyze massive remote sensing data. The emergence of remote sensing cloud computing platform makes it possible to quickly analyze and apply remote sensing data on a global scale or for longterm sequences. However

currently

there is lacking of remote sensing cloud computing platform with complete functions in domestic

while foreign remote sensing cloud computing platform has insufficient support for domestic satellite data. Based on this situation

we have independently developed a spatiotemporal remote sensing cloud computing platform

PIE (Pixel Information Expert) -Engine Studio. By adopting container cloud technology

this platform integrating data

computing power and technology

can implements on-demand acquisition of remote sensing data and rapid processing of massive data just driven by the script. (1) This study first introduced the system architecture of PIE-Engine Studio

and then described the data storage and access mode. (2) PIE-Engine Studio provides operations for multiple objects such as number

matrix

image

vector

list

dictionary

etc.

also machine learning algorithms and some special satellite algorithms. (3) Furthermore

this study illustrated the calculation flow of the platform in detail. Firstly

the user writes a script in the front-end to describe the calculation process of remote sensing data. Click the “Run” button

these codes automatically build the preliminary chained structure call syntax tree. Then the syntax tree is optimized in the back-end through filter the invalid calculation content. The computing tasks are then distributed to the computing services on multiple nodes through the scheduling center. Finally

the resulting visual map layer or data file is returned to the front-end interface triggered by specific front-end requests or operators (print

addLayer

export).(4) At last

an application case is presented

we adopted Landsat 8 data and taking the calculation of Normalized Difference Vegetation Index (NDVI) in the growing season as an example

the calculation results and running time of this platform are compared with Google Earth Engine (GEE). The results show that

due to the limitation of computing resources

the running and export time of this platform are slightly longer than that of GEE

but the spatial distribution of calculation results is consistent

among which about 68% values are distributed between (0.48

0.77)

and 95.33% of the difference between the two results is concentrated between (-0.13

0.13). It shows that the results are reliable. Therefore

the remote sensing cloud computing platform constructed by this paper

can provide data resources and computing power for research in the field of earth science

and will help promote the development of remote sensing cloud computing platform in China and the application of domestic satellite data in cloud computing platform.

关键词

遥感大数据遥感云计算平台分布式存储并行计算

Keywords

remote sensingbig dataremote sensing cloud computing platformdistributed storageparallel computing

references

Bastin J F, Berrahmouni N, Grainger A, Maniatis D, Mollicone D, Moore R, Patriarca C, Picard N, Sparrow B, Abraham E M, Aloui K, Atesoglu A, Attore F, Bassüllü Ç, Bey A, Garzuglia M, García-Montero L G, Groot N, Guerin G, Laestadius L, Lowe A J, Mamane B, Marchi G, Patterson P, Rezende M, Ricci S, Salcedo I, Diaz A S P, Stolle F, Surappaeva V and Castro R. 2017. The extent of forest in dryland biomes. Science, 356(6338): 635-638 [DOI: 10.1126/science.aam6527http://dx.doi.org/10.1126/science.aam6527]

Bastin J F, Finegold Y, Garcia C, Mollicone D, Rezende M, Routh D, Zohner C M and Crowther T W. 2019. The global tree restoration potential. Science, 365(6448): 76-79 [DOI: 10.1126/science.aax0848http://dx.doi.org/10.1126/science.aax0848]

Bogard M J, Kuhn C D, Johnston S E, Striegl R G, Holtgrieve G W, Dornblaser M M, Spencer R G M, Wickland K P and Butman D E. 2019. Negligible cycling of terrestrial carbon in many lakes of the arid circumpolar landscape. Nature Geoscience, 12(3): 180-185 [DOI: 10.1038/s41561-019-0299-5http://dx.doi.org/10.1038/s41561-019-0299-5]

Bureau of Industry and Security, Commerce. 2020. Addition of software specially designed to automate the analysis of geospatial imagery to the export control classification number 0Y521 series[EB/OL]. (2020-01-06)[2021-04-29]. https://www.federalregister.gov/documents/2020/01/06/2019-27649/addition-of-software-specially-designed-to-‍https://www.federalregister.gov/documents/2020/01/06/2019-27649/addition-of-software-specially-designed-to-‍automate-the-analysis-of-geospatial-imagery-to-the-export

Burke M and Lobell D B. 2017. Satellite-based assessment of yield variation and its determinants in smallholder African systems. Proceedings of the National Academy of Sciences of the United States of America, 114(9): 2189-2194 [DOI: 10.1073/pnas.1616919114http://dx.doi.org/10.1073/pnas.1616919114]

Ceccherini G, Duveiller G, Grassi G, Lemoine G, Avitabile V, Pilli R and Cescatti A. 2020. Abrupt increase in harvested forest area over Europe after 2015. Nature, 583(7814): 72-77 [DOI: 10.1038/s41586-020-2438-yhttp://dx.doi.org/10.1038/s41586-020-2438-y]

Chudley T R, Christoffersen P, Doyle S H, Bougamont M, Schoonman C M, Hubbard B and James M R. 2019. Supraglacial lake drainage at a fast-flowing Greenlandic outlet glacier. Proceedings of the National Academy of Sciences of the United States of America, 116(51): 25468-25477 [DOI: 10.1073/pnas.1913685116http://dx.doi.org/10.1073/pnas.1913685116]

Dong J W, Li S W, Zeng Y L, Yan K and Fu D J. 2020. Remote Sensing Cloud Computing and Scientific Analysis-Application and Practice. Beijing: Science Press: 10-18

董金玮, 李世卫, 曾也鲁, 闫凯, 付东杰. 2020. 遥感云计算与科学分析——应用与实践. 北京: 科学出版社: 10-18

Finer M, Novoa S, Weisse M J, Petersen R, Mascaro J, Souto T, Stearns F and Martinez R G. 2018. Combating deforestation: from satellite to intervention. Science, 360(6395): 1303-1305 [DOI: 10.1126/science.aat1203http://dx.doi.org/10.1126/science.aat1203]

Fu D J, Xiao H, Su F Z, Zhou C H, Dong J W, Zeng Y L, Yan K, Li S W, Wu J, Wu W Z and Yan F Q. 2021. Remote sensing cloud computing platform development and Earth science application. National Remote Sensing Bulletin, 25(1): 220-230

付东杰, 肖寒, 苏奋振, 周成虎, 董金玮, 曾也鲁, 闫凯, 李世卫, 吴进, 吴文周, 颜凤芹. 2021. 遥感云计算平台发展及地球科学应用. 遥感学报, 25(1): 220-230 [DOI: 10.11834/jrs.20210447http://dx.doi.org/10.11834/jrs.20210447]

Giezendanner J, Pasetto D, Perez-Saez J, Cerrato C, Viterbi R, Terzago S, Palazzi E and Rinaldo A. 2020. Earth and field observations underpin metapopulation dynamics in complex landscapes: near-term study on carabids. Proceedings of the National Academy of Sciences of the United States of America, 117(23): 12877-12884 [DOI: 10.1073/pnas.1919580117http://dx.doi.org/10.1073/pnas.1919580117]

Gorelick N, Hancher M, Dixon M, Ilyushchenko S, Thau D and Moore R. 2017. Google Earth Engine: planetary-scale geospatial analysis for everyone. Remote Sensing of Environment, 202: 18-27 [DOI: 10.1016/j.rse.2017.06.031http://dx.doi.org/10.1016/j.rse.2017.06.031]

Hansen A J, Burns P, Ervin J, Goetz S J, Hansen M, Venter O, Watson J E M, Jantz P A, Virnig A L S, Barnett K, Pillay R, Atkinson S, Supples C, Rodríguez-Buritica S and Armenteras D. 2020. A policy-driven framework for conserving the best of Earth’s remaining moist tropical forests. Nature Ecology and Evolution, 4(10): 1377-1384 [DOI: 10.1038/s41559-020-1274-7http://dx.doi.org/10.1038/s41559-020-1274-7]

Hansen M C, Potapov P V, Moore R, Hancher M, Turubanova S A, Tyukavina A, Thau D, Stehman S V, Goetz S J, Loveland T R, Kommareddy A, Egorov A, Chini L, Justice C O and Townshend J R G. 2013. High-resolution global maps of 21st-century forest cover change. Science, 342(6160): 850-853 [DOI: 10.1126/science.1244693http://dx.doi.org/10.1126/science.1244693]

Hendershot J N, Smith J R, Anderson C B, Letten A D, Frishkoff L O, Zook J R, Fukami T and Daily G C. 2020. Intensive farming drives long-term shifts in avian community composition. Nature, 579(7799): 393-396 [DOI: 10.1038/s41586-020-2090-6http://dx.doi.org/10.1038/s41586-020-2090-6]

Joshi A R, Dinerstein E, Wikramanayake E, Anderson M L, Olson D, Jones B S, Seidensticker J, Lumpkin S, Hansen M C, Sizer N C, Davis C L, Palminteri S and Hahn N R. 2016. Tracking changes and preventing loss in critical tiger habitat. Science Advances, 2(4): e1501675 [DOI: 10.1126/sciadv.1501675http://dx.doi.org/10.1126/sciadv.1501675]

Laskin D N, McDermid G J, Nielsen S E, Marshall S J, Roberts D R and Montaghi A. 2019. Advances in phenology are conserved across scale in present and future climates. Nature Climate Change, 9(5): 419-425 [DOI: 10.1038/s41558-019-0454-4http://dx.doi.org/10.1038/s41558-019-0454-4]

Li D R. 2016. Towards geo-spatial information science in big data era. Acta Geodaetica et Cartographica Sinica, 45(4): 379-384

李德仁. 2016. 展望大数据时代的地球空间信息学. 测绘学报, 45(4): 379-384 [DOI: 10.11947/j.AGCS.2016.20160057http://dx.doi.org/10.11947/j.AGCS.2016.20160057]

Li D R, Zhang P L and Xia G S. 2014. Automatic analysis and mining of remote sensing big data. Acta Geodaetica et Cartographica Sinica, 43(12): 1211-1216

李德仁, 张良培, 夏桂松. 2014. 遥感大数据自动分析与数据挖掘. 测绘学报, 43(12): 1211-1216 [DOI: 10.13485/j.cnki.11-2089.2014.0187http://dx.doi.org/10.13485/j.cnki.11-2089.2014.0187]

Liu J, Wang W and Zhong H. 2020a. EarthDataMiner: a cloud-based big earth data intelligence analysis platform. IOP Conference Series: Earth and Environmental Science, 509: 012032 [DOI: 10.1088/1755-1315/509/1/012032http://dx.doi.org/10.1088/1755-1315/509/1/012032]

Liu Y L, Kumar M, Katul G G, Feng X and Konings A G. 2020b. Plant hydraulics accentuates the effect of atmospheric moisture stress on transpiration. Nature Climate Change, 10(7): 691-695 [DOI: 10.1038/s41558-020-0781-5http://dx.doi.org/10.1038/s41558-020-0781-5]

Liu X P, Huang Y H, Xu X C, Li X C, Li X, Ciais P, Lin P R, Gong K, Ziegler A D, Chen A P, Gong P, Chen J, Hu G H, Chen Y M, Wang S J, Wu Q S, Huang K N, Estes L and Zeng Z Z. 2020c. High-spatiotemporal-resolution mapping of global urban change from 1985 to 2015. Nature Sustainability, 3(7): 564-570 [DOI: 10.1038/s41893-020-0521-xhttp://dx.doi.org/10.1038/s41893-020-0521-x]

Moore R T and Hansen M C. 2011. Google Earth Engine: a new cloud-computing platform for global-scale earth observation data and analysis//Proceedings of the American Geophysical Union, Fall Meeting 2011. [s.l.]: AGU

Myers-Smith I H, Kerby J T, Phoenix G K, Bjerke J W, Epstein H E, Assmann J J, John C, Andreu-Hayles L, Angers-Blondin S, Beck P S A, Berner L T, Bhatt U S, Bjorkman A D, Blok D, Bryn A, Christiansen C T, Cornelissen J H C, Cunliffe A M, Elmendorf S C, Forbes B C, Goetz S J, Hollister R D, de Jong R, Loranty M M, Macias-Fauria M, Maseyk K, Normand S, Olofsson J, Parker T C, Parmentier F J W, Post E, Schaepman-Strub G, Stordal F, Sullivan P F, Thomas H J D, Tømmervik H, Treharne R, Tweedie C E, Walker D A, Wilmking M and Wipf S. 2020. Complexity revealed in the greening of the Arctic. Nature Climate Change, 10(2): 106-117 [DOI: 10.1038/s41558-019-0688-1http://dx.doi.org/10.1038/s41558-019-0688-1]

Nemani R R, Votava P, Michaelis A, Melton F S, Hashimoto H, Milesi C, Wang W and Ganguly S. 2010. NASA earth exchange: a collaborative earth science platform//Proceedings of the American Geophysical Union, Fall Meeting 2010. [s.l.]: AGU

Qin Y W, Xiao X M, Dong J W, Zhang Y, Wu X C, Shimabukuro Y, Arai E, Biradar C, Wang J, Zou Z H, Liu F, Shi Z, Doughty R and Moore III B. 2019. Improved estimates of forest cover and loss in the Brazilian Amazon in 2000-2017. Nature Sustainability, 2(8): 764-772 [DOI: 10.1038/s41893-019-0336-9http://dx.doi.org/10.1038/s41893-019-0336-9]

Ryan J C, Smith L C, van As D, Cooley S W, Cooper M G, Pitcher L H and Hubbard A. 2019. Greenland Ice Sheet surface melt amplified by snowline migration and bare ice exposure. Science Advances, 5(3): (eaav3738 [DOI: 10.1126/sciadv.aav3738http://dx.doi.org/10.1126/sciadv.aav3738])

Tuckett P A, Ely J C, Sole A J, Livingstone S J, Davison B J, van Wessem J M and Howard J. 2019. Rapid accelerations of Antarctic Peninsula outlet glaciers driven by surface melt. Nature Communications, 10(1): 4311 [DOI: 10.1038/s41467-019-12039-2http://dx.doi.org/10.1038/s41467-019-12039-2]

Venter Z S, Aunan K, Chowdhury S and Lelieveld J. 2020. COVID-19 lockdowns cause global air pollution declines. Proceedings of the National Academy of Sciences of the United States of America, 117(32): 18984-18990 [DOI: 10.1073/pnas.2006853117http://dx.doi.org/10.1073/pnas.2006853117]

Walter T R, Haghighi M H, Schneider F M, Coppola D, Motagh M, Saul J, Babeyko A, Dahm T, Troll V R, Tilmann F, Heimann S, Valade S, Triyono R, Khomarudin R, Kartadinata N, Laiolo M, Massimetti F and Gaebler P. 2019. Complex hazard cascade culminating in the Anak Krakatau sector collapse. Nature Communications, 10(1): 4339 [DOI: 10.1038/s41467-019-12284-5http://dx.doi.org/10.1038/s41467-019-12284-5]

Watmough G R, Marcinko C L J, Sullivan C, Tschirhart K, Mutuo P K, Palm C A and Svenning J C. 2019. Socioecologically informed use of remote sensing data to predict rural household poverty. Proceedings of the National Academy of Sciences of the United States of America, 116(4): 1213-1218 [DOI: 10.1073/pnas.1812969116http://dx.doi.org/10.1073/pnas.1812969116]

Weiss D J, Nelson A, Gibson H S, Temperley W, Peedell S, Lieber A, Hancher M, Poyart E, Belchior S, Fullman N, Mappin B, Dalrymple U, Rozier J, Lucas T C D, Howes R E, Tusting L S, Kang S Y, Cameron E, Bisanzio D, Battle K E, Bhatt S and Gething P W. 2018. A global map of travel time to cities to assess inequalities in accessibility in 2015. Nature, 553(7688): 333-336 [DOI: 10.1038/nature25181http://dx.doi.org/10.1038/nature25181]

Wu X, Braun D, Schwartz J, Kioumourtzoglou M A and Dominici F. 2020. Evaluating the impact of long-term exposure to fine particulate matter on mortality among the elderly. Science Advances, 6(29): (eaba5692 [DOI: 10.1126/sciadv.aba5692http://dx.doi.org/10.1126/sciadv.aba5692])

Yeh C, Perez A, Driscoll A, Azzari G, Tang Z Y, Lobell D, Ermon S and Burke M. 2020. Using publicly available satellite imagery and deep learning to understand economic well-being in Africa. Nature Communications, 11(1): 2583 [DOI: 10.1038/s41467-020-16185-whttp://dx.doi.org/10.1038/s41467-020-16185-w]

文章被引用时，请邮件提醒。

提交