Anterior|Siguiente

GF quantitative remote sensing production system： Core design

  • role： First author第一作者
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

  • Email：zhangzheng@aircas.ac.cn
  • Introduction：张正，研究方向为遥感图像处理、大数据挖掘、高性能计算。E-mail： zhangzheng@aircas.ac.cn
ZHANG Zheng,  
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

LI Hongyi,  
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

HU Changmiao,  
  • role： Corresponding author通信作者
  • Affiliation：

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

  • Email：tangping@aircas.ac.cn
  • Introduction：唐娉，研究方向为遥感图像处理、人工智能与深度学习。E-mail： tangping@aircas.ac.cn
TANG Ping*

resumen

A fast development in remote sensing science and technologies has been witnessed in recent years with the launch of various remote sensing satellites and the establishment of numerous industrial application systems. Satellite series, like GaoFen, has pushed the richness of data to a new level. Moreover, quantitative product-driven application systems have become increasingly influential in many disciplines. By contrast, the bridge between data and application, namely, the production capability of quantitative products, seems weak, greatly limiting the usage and influence of GaoFen data. The improvement of production capability comes from multiple aspects, including product hierarchy, algorithm model, and production system. In this study, we propose, from the production system perspective, an integrated system design that responds to the four main challenges of the system: uniform data access, heterogeneous algorithm integration, layered workflow orchestration, and cloud infrastructure adaptation.The system is based on algorithm containerization, where executable algorithms and all their dependencies are encapsulated. Thus, it can run uniformly and consistently on different infrastructures without worrying about the complexity caused by deployment. This scenario helps the system to manage diverse remote sensing algorithms uniformly. We employ the Kubernetes container orchestration platform to automate the execution, scaling, and management of containerized algorithms. A containerized cluster consists of multiple master nodes, many computing nodes, and multiple data centers. Multiple algorithm repositories are constructed to support the system and cope with the high computing and data throughput density of remote sensing algorithms. Each algorithm repository is further divided into several subrepositories to improve load balancing. User-defined role-based access control for algorithms is set up to protect the intellectual properties of algorithm owners. A recommended algorithm image architecture is introduced to standardize algorithm encapsulation. A set of nine properties are abstracted to describe uniformly any data entity parameter of an algorithm. This approach ensures that suitable input data can be found for user-uploaded algorithms to run in the system. For data visualization and quantitative computing scenarios, a multiscenario data organization strategy is proposed to avoid excessive data operations, such as projection transform or subdivision. The business logic of the system, from user order creation to product calculation, is detailed for clear implementation. The production sometimes involves workflow batches. We propose a stratified workflow aggregation strategy to optimize workflow execution.The system has been used for large-scale production of various GF quantitative remote sensing products, including surface reflectance product, normalized difference vegetation index product, leaf area index product, and surface albedo product. These products fully cover China’s area for eight successive years from 2013 to 2020, with quantities of more than five million and storages of nearly 300 TB. The proposed system completes the production task smoothly and efficiently.During the routine support for many large-scale production tasks, each part of the system performed consistently with the system design proposed in this study, demonstrating that the study can help build a stable and efficient quantitative remote sensing production system on cloud-native infrastructures

palabra clave

production system;quantitative product;algorithm integration;container;cloud computing;system architecture;workflow

References

  1. 1.
    Bai Z G. 2013. Technical characteristics of the GaoFen-1 satellite. Aerospace China, (8): 5-9
  2. 2.
    Bernstein D. 2014. Containers and cloud: from LXC to docker to kubernetes. IEEE Cloud Computing, 1(3): 81-84
  3. 3.
    Burns B, Grant B, Oppenheimer D, Brewer E and Wilkes J. 2016. Borg, omega, and kubernetes: lessons learned from three container-management systems over a decade. Queue, 14(1): 70-93
  4. 4.
    Deng X D, Liu P H, Liu X P, Wang R Y, Zhang Y Y, He J and Yao Y. 2019. Geospatial big data: new paradigm of remote sensing applications. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(10): 3841-3851
  5. 5.
    Drusch M, Del Bello U, Carlier S, Colin O, Fernandez V, Gascon F, Hoersch B, Isola C, Laberinti P, Martimort P, Meygret A, Spoto F, Sy O, Marchese F and Bargellini P. 2012. Sentinel-2: ESA’s optical high-resolution mission for GMES operational services. Remote sensing of Environment, 120: 25-36
  6. 6.
    Esfandiari M, Ramapriyan H, Behnke J and Sofinowski E. 2007. Earth observing system (EOS) data and information system (EOSDIS)—evolution update and future//2007 IEEE International Geoscience and Remote Sensing Symposium. Barcelona: IEEE: 4005-4008
  7. 7.
    Fan J Q, Yan J N, Ma Y and Wang L Z. 2018. Big data integration in remote sensing across a distributed metadata-based spatial infrastructure. Remote Sensing, 10(1): 7
  8. 8.
    Hu T, Renzullo L J, Cao B, Van Dijk A I J M, Du Y M, Li H, Cheng J, Xu Z H, Zhou J and Liu Q H. 2019. Directional variation in surface emissivity inferred from the MYD21 product and its influence on estimated surface upwelling longwave radiation. Remote Sensing of Environment, 228: 45-60
  9. 9.
    Hu T, Renzullo L J, Van Dijk A I J M, He J, Tian S Y, Xu Z H, Zhou J, Liu T J and Liu Q H. 2020. Monitoring agricultural drought in Australia using MTSAT-2 land surface temperature retrievals. Remote Sensing of Environment, 236: 111419
  10. 10.
    Irons J R, Dwyer J L and Barsi J A. 2012. The next Landsat satellite: the Landsat data continuity mission. Remote Sensing of Environment, 122: 11-21
  11. 11.
    Kang W C, Xiang Y M, Wang F, Wan L and You H J. 2018. Flood detection in Gaofen-3 SAR images via fully convolutional networks. Sensors, 18(9): 2915
  12. 12.
    Li H Y and Tang P. 2020. Dps-MuSyQ: a distributed parallel processing system for multi-source data synergized quantitative remote sensing products producing. IEEE Access, 8: 79510-79520
  13. 13.
    Li H Y, Zhang Z and Tang P. 2020. A web-based remote sensing data processing and production system with the unified integration of multi-disciplinary data and models. IEEE Access, 8: 162961-162972
  14. 14.
    Liu Q H, Wen J G, Zhou X, Zhao J, Li Z Y, Li X, Ma M G, Wang W Z, Liao X H, Liu S M, Fan W J, Xiao Q, Zhong B, Li J, Xin X Z, Li L, Jia L, Gao Z H, Jin J D, Liang S, Xin J, Liao C J and Wu Y R. 2023. Technique system of remote sensing product generation and validation of GF common products. National Remote Sensing Bulletin， 27(3): 544-562
  15. 15.
    Liu Q H, Zhong B, Tang P, Zhang H H, Li H Y, Wu S L, Xin X Z, Li J, Jia L, Shan X J, Zhang Z, Wen J G, Du Y M, Li L, Yang A X, Li H, Hu G C, Zhao J, Zhang H L, Yu S S, Dou B C and Wu J J. 2018. Remote sensing data products oriented quantitative computing system—the GSC best practice data computing environment 2018. Journal of Global Change Data and Discovery, 2(3): 271-278
  16. 16.
    Lü X F, Cheng C Q, Gong J Y and Guan L. 2011. Review of data storage and management technologies for massive remote sensing data. Science China Technological Sciences, 54(12): 3220-3232
  17. 17.
    Ma Y, Wu H P, Wang L Z, Huang B, Ranjan R, Zomaya A and Jie W. 2015. Remote sensing big data computing: challenges and opportunities. Future Generation Computer Systems, 51: 47-60
  18. 18.
    Merkel D. 2014. Docker: lightweight Linux containers for consistent development and deployment. Linux Journal, 2014(239): 2
  19. 19.
    Nie J, Deng L, Hao X L, Liu M and He Y. 2018. Application of GF-4 satellite in drought remote sensing monitoring: a case study of Southeastern Inner Mongolia. Journal of Remote Sensing, 22(3): 400-407
  20. 20.
    Pan T. 2015. Technical characteristics of the GaoFen-2 satellite. Aerospace China, (1): 3-9
  21. 21.
    Roy D P, Wulder M A, Loveland T R, Woodcock C E, Allen R G, Anderson M C, Helder D, Irons J R, Johnson D M, Kennedy R, Scambos T A, Schaaf C B, Schott J R, Sheng Y, Vermote E F, Belward A S, Bindschadler R, Cohen W B, Gao F, Hipple J D, Hostert P, Huntington J, Justice C O, Kilic A, Kovalskyy V, Lee Z P, Lymburner L, Masek J G, McCorkel J, Shuai Y, Trezza R, Vogelmann J, Wynne R H and Zhu Z. 2014. Landsat-8: science and product vision for terrestrial global change research. Remote sensing of Environment, 145: 154-172
  22. 22.
    Shao Z F, Fu H Y, Li D R, Altan O and Cheng T. 2019. Remote sensing monitoring of multi-scale watersheds impermeability for urban hydrological evaluation. Remote Sensing of Environment, 232: 111338
  23. 23.
    Wang L Z, Ma Y, Yan J N, Chang V and Zomaya A Y. 2018. pipsCloud: high performance cloud computing for remote sensing big data management and processing. Future Generation Computer Systems, 78: 353-368
  24. 24.
    Wang Q, Wu C Q, Li Q and Li J S. 2010. Chinese HJ-1A/B satellites and data characteristics. Science China Earth Sciences, 53(1): 51-57
  25. 25.
    Yan J N, Ma Y, Wang L Z, Choo K K R and Jie W. 2018. A cloud-based remote sensing data production system. Future Generation Computer Systems, 86: 1154-1166
  26. 26.
    Yang J, Dong C H, Lu N M, Yang Z D, Shi J M, Zhang P, Liu Y J and Cai B. 2009. FY-3A: the new generation polar-orbiting meteorological satellite of china. Acta Meteorologica Sinica, 67(4): 501-509
  27. 27.
    Zhang Y J, Zheng M T, Xiong J X, Lu Y H and Xiong X D. 2014. On-Orbit geometric calibration of ZY-3 three-line array imagery with multistrip data sets. IEEE Transactions on Geoscience and Remote Sensing, 52(1): 224-234
  28. 28.
    Zhang Z, Tang P, Li H Y and Feng Z. 2016. Refined domain model for multisource data synergized quantitative remote sensing production system. Journal of Remote Sensing, 20(2): 184-196
  29. 29.
    Zhong B, Yang A X, Liu Q H, Wu S L, Shan X J, Mu X H, Hu L F and Wu J J. 2021. Analysis ready data of the Chinese GaoFen satellite data. Remote Sensing, 13(9): 1709
  30. 30.
    Zhou J M, Zhang X, Liu Z P and Li Z. 2021. Extraction and analysis of mountain glacier movement from GF-1 satellite data. National Remote Sensing Bulletin, 25(2): 530-538

Leer el texto completo

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website