سابِق|تال

GeoCube： A spatio-temporal cube toward massive and multi-source EO data analysis

  • role： First author第一作者
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

  • Email：fgao@whu.edu.cn
  • Introduction：高凡，研究方向为时空大数据存储和计算。E-mail： fgao@whu.edu.cn
GAO Fan1,  
  • role： Corresponding author通信作者
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

    Hubei Province Engineering Center for Intelligent Geoprocessing (HPECIG), Wuhan 430079, China

  • Email：pyue@whu.edu.cn
  • Introduction：乐鹏，研究方向为地理信息系统与服务。E-mail：pyue@whu.edu.cn
YUE Peng12*,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

JIANG Liangcun1,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

CAO Zhipeng1,  
  • Affiliation：

    Guangdong Southern Digital Technology Co., Ltd., Guangzhou 510665, China

LIANG Zheheng3,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

SHANGGUAN Boyi1,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

HU Lei1,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

ZHAO Shuaifeng1

ملخص

The volume of Earth Observation (EO) data has tremendously increased after the establishment of EO system. Managing such big EO data and turning them into valuable information is a major challenge in EO domain. This study proposes a multisource EO cube toward large-scale analysis.The infrastructure accommodates multisource geospatial data including raster and vector data. A cube model is designed, and four dimensions including product, space, time, and band dimension are formalized. Several cube explore examples are presented. The infrastructure enables large-scale analysis based on cloud computing technology, and a set of distributed cube objects extending Spark Resilient Distributed Dataset for cube tiles is designed. The distributed cube objects are compatible with multiple data source including raster and vector data. A multi-thread computing method is used together with cloud computing, which forms a hybrid parallelism, to further improve data access and processing efficiency. Batch computation is also used to address the issue that massive number of tiles cannot be loaded into memory at one time. Moreover, a machine learning-based approach is integrated into the cube to enhance parallel geoprocessing. The computational intensity of tiles can be predicted and saved in databases in advance, which eliminates the extra time cost of computational intensity prediction on the fly for those commonly used products. The design and implementation for the cube infrastructure, named GeoCube, is provided. It covers the ingestion and management of multisource geospatial data in the cube, the processing of geospatial/EO queries against different cube dimensions, and high-performance cube computing of large-scale geospatial datasets. The creation of such a geospatial data cube help advance the EO data cube approach while keeping connections to the data cube in the BI domain.The performance on data query and access, data processing, and load balance is presented. Results demonstrate the advantage of GeoCube infrastructure. Several applications are presented including cube OLAP operations, large-scale time-series analysis, and multisource data cube analysis.In conclusion, compared with existing cube approaches, the proposed infrastructure emphasizes the accommodation of multisource geospatial data including raster and vector data in the cube, cube tile processing with cloud computing, and artificial intelligence machine learning-enabled cube computation. Such a cube can inherit not only the large-scale processing capabilities of EO data cubes but also the data management capabilities of BI data cubes.

مفهوم

remote sensing;EO data;spatio-temporal cube;artificial intelligence;large-scale analysis

References

  1. 1.
    Armstrong M P and Densham P J. 1992. Domain decomposition for parallel processing of spatial problems. Computers, Environment and Urban Systems, 16(6): 497-513
  2. 2.
    Bansal S, Kumar P and Singh K. 2006. An improved two-step algorithm for task and data parallel scheduling in distributed memory machines. Parallel Computing, 32(10): 759-774
  3. 3.
    Baumann P. 2017. The datacube manifesto[EB/OL]. .
  4. 4.
    Baumann P, Dehmel A, Furtado P, Ritsch R, Widmann N. 1999. Spatio-temporal retrieval with RasDaMan//Proceedings of the 25th International Conference on Very Large Data Bases. Edinburgh, Scotland, UK: Morgan Kaufmann Publishers Inc.: 746-749
  5. 5.
    Baumann P, Rossi A P, Bell B, Clements O, Evans B, Hoenig H, Hogan P, Kakaletris G, Koltsida P, Mantovani S, Marco Figuera R, Merticariu V, Misev D, Pham H B, Siemen S and Wagemann J. 2018. Fostering cross-disciplinary earth science through datacube analytics//Earth Observation Open Science and Innovation. Switzerland: Spring: 91-119
  6. 6.
    CEOS. 2020a. Committee on earth observation satellites[EB/OL].
  7. 7.
    CEOS. 2020b. CEOS Open Data Cube[EB/OL].
  8. 8.
    Chaudhuri S, Dayal U. 1997. An overview of data warehousing and OLAP technology. ACM SIGMOD Record, 26(1): 65-74
  9. 9.
    Crippen R E. 1990. Calculating the vegetation index faster. Remote Sensing of Environment, 34(1): 71-73
  10. 10.
    DASK. 2020. DASK: Scalable analytics in Python[EB/OL].
  11. 11.
    Gao B C. 1996. NDWI—A normalized difference water index for remote sensing of vegetation liquid water from space. Remote Sensing of Environment, 58(3): 257-266
  12. 12.
    Huete A, Didan K, Miura T, Rodriguez E P, Gao X and Ferreira L G. 2002. Overview of the radiometric and biophysical performance of the MODIS vegetation indices. Remote Sensing of Environment, 83(1/2): 195-213
  13. 13.
    Huete A R. 1988. A soil-adjusted vegetation index (SAVI). Remote Sensing of Environment, 25(3): 295-309
  14. 14.
    Kopp S, Becker P, Doshi A, Wright D J, Zhang K X and Xu H. 2019. Achieving the full vision of earth observation data cubes. Data, 4(3): 94
  15. 15.
    Lewis A, Lacey J, Mecklenburg S, Ross J, Siqueira A, Killough B, Szantoi Z, Tadono T, Rosenqvist A, Goryl P, Miranda N and Hosford S. 2018. CEOS analysis ready data for land (CARD4L) overview//2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia, Spain: IEEE: 7407-7410
  16. 16.
    Li D R, Zhang L P and Xia G S. 2014. Automatic analysis and mining of remote sensing big data. Acta Geodaetica et Cartographica Sinica, 43(12): 1211-1216
  17. 17.
    Liao X J. 2021. Scientific and technological progress and development prospect of the earth observation in China in the past 20 years. Journal of Remote Sensing, 25(1): 267-275
  18. 18.
    Mahecha M D, Gans F, Brandt G, Christiansen R, Cornell S E, Fomferra N, Kraemer G, Peters J, Bodesheim P, Camps-Valls G, Donges J F, Dorigo W, Estupinan-Suarez L M, Gutierrez-Velez V H, Gutwin M, Jung M, Londoño M C, Miralles D G, Papastefanou P and Reichstein M. 2020. Earth system data cubes unravel global multivariate dynamics. Earth System Dynamics, 11(1): 201-234
  19. 19.
    McFeeters S K. 1996. The use of the Normalized Difference Water Index (NDWI) in the delineation of open water features. International Journal of Remote Sensing, 17(7): 1425-1432
  20. 20.
    Mueller N, Lewis A, Roberts D, Ring S, Melrose R, Sixsmith J, Lymburner L, McIntyre A, Tan P, Curnow S and lp A. 2016. Water observations from space: mapping surface water from 25 years of Landsat imagery across Australia. Remote Sensing of Environment, 174: 341-352
  21. 21.
    Nativi S, Mazzetti P and Craglia M. 2017. A view-based model of data-cube to support big earth data systems interoperability. Big Earth Data, 1(1/2): 75-99
  22. 22.
    Ren Y B, Chen Z J, Chen G, Han Y and Wang Y J. 2017. A hybrid process/thread parallel algorithm for generating DEM from LiDAR points. ISPRS International Journal of Geo-Information, 6(10): 300
  23. 23.
    Sudmanns M, Tiede D, Lang S, Bergstedt H, Trost G, Augustin H, Baraldi A and Blaschke T. 2020. Big Earth data: disruptive changes in Earth observation data management and analysis?. International Journal of Digital Earth, 13(7): 832-850
  24. 24.
    Voidrot M F and Percivall G. 2020. OGC geospatial coverages data cube community practice. IOP Conference Series: Earth and Environmental Science, 509: 012058
  25. 25.
    Wagemann J, Clements O, Figuera R M, Rossi A P and Mantovani S. 2018. Geospatial web services pave new ways for server-based on-demand access and processing of Big Earth Data. International Journal of Digital Earth, 11(1): 7-25
  26. 26.
    Wang S W and Armstrong M P. 2009. A theoretical approach to the use of cyberinfrastructure in geographical analysis. International Journal of Geographical Information Systems, 23(2): 169-193
  27. 27.
    Wulder M A and Coops N C. 2014. Satellites: make Earth observations open access. Nature, 513(7516): 30-31
  28. 28.
    Xu H Q. 2006. Modification of normalised difference water index (NDWI) to enhance open water features in remotely sensed imagery. International Journal of Remote Sensing, 27(14): 3025-3033
  29. 29.
    You S M, Zhang J T and Gruenwald L. 2015. Large-scale spatial join query processing in cloud//The 31st IEEE International Conference on Data Engineering Workshops. Seoul, Korea (South): IEEE: 34-41
  30. 30.
    Yue P, Gao F, Shangguan B Y and Yan Z R. 2020. A machine learning approach for predicting computational intensity and domain decomposition in parallel geoprocessing. International Journal of Geographical Information Science, 34(11): 2243-2274
  31. 31.
    Yue P, Ramachandran R, Baumann P, Khalsa S J S, Deng M X and Jiang L C. 2016. Recent activities in earth data science [technical committees]. IEEE Geoscience and Remote Sensing Magazine, 4(4): 84-89
  32. 32.
    Yue P, Zhang M D and Tan Z Y. 2015. A geoprocessing workflow system for environmental monitoring and integrated modelling. Environmental Modelling and Software, 69: 128-140
  33. 33.
    Zaharia M, Chowdhury M, Franklin M J, Shenke S and Stoica I. 2010. Spark: cluster computing with working sets//Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing. Boston, MA: USENIX Association
  34. 34.
    Zhang L F, Chen H, Sun X J, Fu D J, Tong Q X. 2017. Designing spatial-temporal-spectral integrated storage structure of multi-dimensional remote sensing images. Journal of Remote Sensing, 21(1): 62-73

قراءة النص الكامل

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website