高分遥感共性产品生产系统关键设计
GF quantitative remote sensing production system： Core design
- 2023年27卷第3期 页码：651-664
纸质出版日期： 2023-03-07
DOI： 10.11834/jrs.20210420
扫 描 看 全 文
浏览全部资源
扫码关注微信
摘要
以众多遥感卫星的发射与行业应用系统为代表，遥感应用技术在近年间取得了快速发展，国产高分系列卫星等平台所提供的数据越来越丰富，以遥感共性产品作为信息源的应用系统也已经覆盖了多个行业。然而这两者之间的桥梁，即由国产卫星数据生产标准化系列遥感共性产品的能力，仍显得相对薄弱，极大限制了国产遥感数据的使用率与影响力。这一能力的改善需要从产品体系、产品算法与生产系统等多方面共同推进。本文从高分遥感共性产品生产系统的角度，针对系统所面对的来自数据集成、算法集成、生产编排与云计算等方面的诸多挑战，提出了一套完整的系统关键设计。系统在容器化集成运行的基础上，涵盖了集群、数据、算法、权限、参数、工作流、生产等一系列关键环节的关键技术。系统研发与生产实践表明，本系统可以流畅的集成各类共性产品算法，实现稳定高效的业务化生产平台，助力于国产卫星产品服务能力的有效提高。
Abstract
A fast development in remote sensing science and technologies has been witnessed in recent years with the launch of various remote sensing satellites and the establishment of numerous industrial application systems. Satellite series
like GaoFen
has pushed the richness of data to a new level. Moreover
quantitative product-driven application systems have become increasingly influential in many disciplines. By contrast
the bridge between data and application
namely
the production capability of quantitative products
seems weak
greatly limiting the usage and influence of GaoFen data. The improvement of production capability comes from multiple aspects
including product hierarchy
algorithm model
and production system. In this study
we propose
from the production system perspective
an integrated system design that responds to the four main challenges of the system: uniform data access
heterogeneous algorithm integration
layered workflow orchestration
and cloud infrastructure adaptation.
The system is based on algorithm containerization
where executable algorithms and all their dependencies are encapsulated. Thus
it can run uniformly and consistently on different infrastructures without worrying about the complexity caused by deployment. This scenario helps the system to manage diverse remote sensing algorithms uniformly. We employ the Kubernetes container orchestration platform to automate the execution
scaling
and management of containerized algorithms. A containerized cluster consists of multiple master nodes
many computing nodes
and multiple data centers. Multiple algorithm repositories are constructed to support the system and cope with the high computing and data throughput density of remote sensing algorithms. Each algorithm repository is further divided into several subrepositories to improve load balancing. User-defined role-based access control for algorithms is set up to protect the intellectual properties of algorithm owners. A recommended algorithm image architecture is introduced to standardize algorithm encapsulation. A set of nine properties are abstracted to describe uniformly any data entity parameter of an algorithm. This approach ensures that suitable input data can be found for user-uploaded algorithms to run in the system. For data visualization and quantitative computing scenarios
a multiscenario data organization strategy is proposed to avoid excessive data operations
such as projection transform or subdivision. The business logic of the system
from user order creation to product calculation
is detailed for clear implementation. The production sometimes involves workflow batches. We propose a stratified workflow aggregation strategy to optimize workflow execution.
The system has been used for large-scale production of various GF quantitative remote sensing products
including surface reflectance product
normalized difference vegetation index product
leaf area index product
and surface albedo product. These products fully cover China’s area for eight successive years from 2013 to 2020
with quantities of more than five million and storages of nearly 300 TB. The proposed system completes the production task smoothly and efficiently.
During the routine support for many large-scale production tasks
each part of the system performed consistently with the system design proposed in this study
demonstrating that the study can help build a stable and efficient quantitative remote sensing production system on cloud-native infrastructures
关键词
生产系统共性产品算法集成容器云计算系统架构工作流
Keywords
production systemquantitative productalgorithm integrationcontainercloud computingsystem architectureworkflow
references
Bai Z G. 2013. Technical characteristics of the GaoFen-1 satellite. Aerospace China, (8): 5-9
白照广. 2013. 高分一号卫星的技术特点. 中国航天, (8): 5-9
Bernstein D. 2014. Containers and cloud: from LXC to docker to kubernetes. IEEE Cloud Computing, 1(3): 81-84 [DOI: 10.1109/MCC.2014.51http://dx.doi.org/10.1109/MCC.2014.51]
Burns B, Grant B, Oppenheimer D, Brewer E and Wilkes J. 2016. Borg, omega, and kubernetes: lessons learned from three container-management systems over a decade. Queue, 14(1): 70-93 [DOI: 10.1145/2898442.2898444http://dx.doi.org/10.1145/2898442.2898444]
Deng X D, Liu P H, Liu X P, Wang R Y, Zhang Y Y, He J and Yao Y. 2019. Geospatial big data: new paradigm of remote sensing applications. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(10): 3841-3851 [DOI: 10.1109/JSTARS.2019.2944952http://dx.doi.org/10.1109/JSTARS.2019.2944952]
Drusch M, Del Bello U, Carlier S, Colin O, Fernandez V, Gascon F, Hoersch B, Isola C, Laberinti P, Martimort P, Meygret A, Spoto F, Sy O, Marchese F and Bargellini P. 2012. Sentinel-2: ESA’s optical high-resolution mission for GMES operational services. Remote sensing of Environment, 120: 25-36 [DOI: 10.1016/j.rse.2011.11.026http://dx.doi.org/10.1016/j.rse.2011.11.026]
Esfandiari M, Ramapriyan H, Behnke J and Sofinowski E. 2007. Earth observing system (EOS) data and information system (EOSDIS)—evolution update and future//2007 IEEE International Geoscience and Remote Sensing Symposium. Barcelona: IEEE: 4005-4008 [DOI: 10.1109/IGARSS.2007.4423727http://dx.doi.org/10.1109/IGARSS.2007.4423727]
Fan J Q, Yan J N, Ma Y and Wang L Z. 2018. Big data integration in remote sensing across a distributed metadata-based spatial infrastructure. Remote Sensing, 10(1): 7 [DOI: 10.3390/rs10010007http://dx.doi.org/10.3390/rs10010007]
Hu T, Renzullo L J, Cao B, Van Dijk A I J M, Du Y M, Li H, Cheng J, Xu Z H, Zhou J and Liu Q H. 2019. Directional variation in surface emissivity inferred from the MYD21 product and its influence on estimated surface upwelling longwave radiation. Remote Sensing of Environment, 228: 45-60 [DOI: 10.1016/j.rse.2019.04.012http://dx.doi.org/10.1016/j.rse.2019.04.012]
Hu T, Renzullo L J, Van Dijk A I J M, He J, Tian S Y, Xu Z H, Zhou J, Liu T J and Liu Q H. 2020. Monitoring agricultural drought in Australia using MTSAT-2 land surface temperature retrievals. Remote Sensing of Environment, 236: 111419 [DOI: 10.1016/j.rse.2019.111419http://dx.doi.org/10.1016/j.rse.2019.111419]
Irons J R, Dwyer J L and Barsi J A. 2012. The next Landsat satellite: the Landsat data continuity mission. Remote Sensing of Environment, 122: 11-21 [DOI: 10.1016/j.rse.2011.08.026http://dx.doi.org/10.1016/j.rse.2011.08.026]
Kang W C, Xiang Y M, Wang F, Wan L and You H J. 2018. Flood detection in Gaofen-3 SAR images via fully convolutional networks. Sensors, 18(9): 2915 [DOI: 10.3390/s18092915http://dx.doi.org/10.3390/s18092915]
Li H Y and Tang P. 2020. Dps-MuSyQ: a distributed parallel processing system for multi-source data synergized quantitative remote sensing products producing. IEEE Access, 8: 79510-79520 [DOI: 10.1109/ACCESS.2020.2989138http://dx.doi.org/10.1109/ACCESS.2020.2989138]
Li H Y, Zhang Z and Tang P. 2020. A web-based remote sensing data processing and production system with the unified integration of multi-disciplinary data and models. IEEE Access, 8: 162961-162972 [DOI: 10.1109/ACCESS.2020.3021791http://dx.doi.org/10.1109/ACCESS.2020.3021791]
Liu Q H, Wen J G, Zhou X, Zhao J, Li Z Y, Li X, Ma M G, Wang W Z, Liao X H, Liu S M, Fan W J, Xiao Q, Zhong B, Li J, Xin X Z, Li L, Jia L, Gao Z H, Jin J D, Liang S, Xin J, Liao C J and Wu Y R. 2023. Technique system of remote sensing product generation and validation of GF common products. National Remote Sensing Bulletin， 27(3): 544-562
柳钦火, 闻建光, 周翔, 赵坚, 李增元, 李新, 马明国, 王维真, 廖小罕, 刘绍民, 范闻捷, 肖青, 仲波, 李静, 辛晓洲, 李丽, 贾立, 高志海, 金家栋, 梁师, 邢进, 廖楚江, 吴一戎. 2023. 高分遥感共性产品生成和真实性检验技术体系. 遥感学报, 27(3): 544-562 [DOI： 10.11834/jrs.20235022http://dx.doi.org/10.11834/jrs.20235022]
Liu Q H, Zhong B, Tang P, Zhang H H, Li H Y, Wu S L, Xin X Z, Li J, Jia L, Shan X J, Zhang Z, Wen J G, Du Y M, Li L, Yang A X, Li H, Hu G C, Zhao J, Zhang H L, Yu S S, Dou B C and Wu J J. 2018. Remote sensing data products oriented quantitative computing system—the GSC best practice data computing environment 2018. Journal of Global Change Data and Discovery, 2(3): 271-278
柳钦火, 仲波, 唐娉, 张宏海, 李宏益, 吴善龙, 辛晓洲, 李静, 贾立, 单小军, 张正, 闻建光, 杜永明, 李丽, 杨爱霞, 历华, 胡光成, 赵静, 张海龙, 余珊珊, 窦宝成, 吴俊君. 2018. 多源协同定量遥感产品生产系统——2018年中国地理学会地理大数据计算环境“优秀实用案例”. 全球变化数据学报, 2(3): 271-278 [DOI: 10.3974/geodp.2018.03.04http://dx.doi.org/10.3974/geodp.2018.03.04]
Lü X F, Cheng C Q, Gong J Y and Guan L. 2011. Review of data storage and management technologies for massive remote sensing data. Science China Technological Sciences, 54(12): 3220-3232 [DOI: 10.1007/s11431-011-4549-zhttp://dx.doi.org/10.1007/s11431-011-4549-z]
Ma Y, Wu H P, Wang L Z, Huang B, Ranjan R, Zomaya A and Jie W. 2015. Remote sensing big data computing: challenges and opportunities. Future Generation Computer Systems, 51: 47-60 [DOI: 10.1016/j.future.2014.10.029http://dx.doi.org/10.1016/j.future.2014.10.029]
Merkel D. 2014. Docker: lightweight Linux containers for consistent development and deployment. Linux Journal, 2014(239): 2
Nie J, Deng L, Hao X L, Liu M and He Y. 2018. Application of GF-4 satellite in drought remote sensing monitoring: a case study of Southeastern Inner Mongolia. Journal of Remote Sensing, 22(3): 400-407
聂娟, 邓磊, 郝向磊, 刘明, 贺英. 2018. 高分四号卫星在干旱遥感监测中的应用. 遥感学报, 22(3): 400-407 [DOI: 10.11834/jrs.20187067http://dx.doi.org/10.11834/jrs.20187067]
Pan T. 2015. Technical characteristics of the GaoFen-2 satellite. Aerospace China, (1): 3-9
潘腾. 2015. 高分二号卫星的技术特点. 中国航天, (1): 3-9
Roy D P, Wulder M A, Loveland T R, Woodcock C E, Allen R G, Anderson M C, Helder D, Irons J R, Johnson D M, Kennedy R, Scambos T A, Schaaf C B, Schott J R, Sheng Y, Vermote E F, Belward A S, Bindschadler R, Cohen W B, Gao F, Hipple J D, Hostert P, Huntington J, Justice C O, Kilic A, Kovalskyy V, Lee Z P, Lymburner L, Masek J G, McCorkel J, Shuai Y, Trezza R, Vogelmann J, Wynne R H and Zhu Z. 2014. Landsat-8: science and product vision for terrestrial global change research. Remote sensing of Environment, 145: 154-172 [DOI: 10.1016/j.rse.2014.02.001http://dx.doi.org/10.1016/j.rse.2014.02.001]
Shao Z F, Fu H Y, Li D R, Altan O and Cheng T. 2019. Remote sensing monitoring of multi-scale watersheds impermeability for urban hydrological evaluation. Remote Sensing of Environment, 232: 111338 [DOI: 10.1016/j.rse.2019.111338http://dx.doi.org/10.1016/j.rse.2019.111338]
Wang L Z, Ma Y, Yan J N, Chang V and Zomaya A Y. 2018. pipsCloud: high performance cloud computing for remote sensing big data management and processing. Future Generation Computer Systems, 78: 353-368 [DOI: 10.1016/j.future.2016.06.009http://dx.doi.org/10.1016/j.future.2016.06.009]
Wang Q, Wu C Q, Li Q and Li J S. 2010. Chinese HJ-1A/B satellites and data characteristics. Science China Earth Sciences, 53(1): 51-57 [DOI: 10.1007/s11430-010-4139-0http://dx.doi.org/10.1007/s11430-010-4139-0]
Yan J N, Ma Y, Wang L Z, Choo K K R and Jie W. 2018. A cloud-based remote sensing data production system. Future Generation Computer Systems, 86: 1154-1166 [DOI: 10.1016/j.future.2017.02.044http://dx.doi.org/10.1016/j.future.2017.02.044]
Yang J, Dong C H, Lu N M, Yang Z D, Shi J M, Zhang P, Liu Y J and Cai B. 2009. FY-3A: the new generation polar-orbiting meteorological satellite of china. Acta Meteorologica Sinica, 67(4): 501-509
杨军, 董超华, 卢乃锰, 杨忠东, 施进明, 张鹏, 刘玉洁, 蔡斌. 2009. 中国新一代极轨气象卫星——风云三号. 气象学报, 67(4): 501-509 [DOI: 10.11676/qxxb2009.050http://dx.doi.org/10.11676/qxxb2009.050]
Zhang Y J, Zheng M T, Xiong J X, Lu Y H and Xiong X D. 2014. On-Orbit geometric calibration of ZY-3 three-line array imagery with multistrip data sets. IEEE Transactions on Geoscience and Remote Sensing, 52(1): 224-234 [DOI: 10.1109/TGRS.2013.2237781http://dx.doi.org/10.1109/TGRS.2013.2237781]
Zhang Z, Tang P, Li H Y and Feng Z. 2016. Refined domain model for multisource data synergized quantitative remote sensing production system. Journal of Remote Sensing, 20(2): 184-196
张正, 唐娉, 李宏益, 冯峥. 2016. 多源数据协同定量遥感产品生产系统的领域模型. 遥感学报, 20(2): 184-196 [DOI: 10.11834/jrs.20164293http://dx.doi.org/10.11834/jrs.20164293]
Zhong B, Yang A X, Liu Q H, Wu S L, Shan X J, Mu X H, Hu L F and Wu J J. 2021. Analysis ready data of the Chinese GaoFen satellite data. Remote Sensing, 13(9): 1709 [DOI: 10.3390/rs13091709http://dx.doi.org/10.3390/rs13091709]
Zhou J M, Zhang X, Liu Z P and Li Z. 2021. Extraction and analysis of mountain glacier movement from GF-1 satellite data. National Remote Sensing Bulletin, 25(2): 530-538
周建民, 张鑫, 刘志平, 李震. 2021. 高分一号山地冰川运动速度提取与分析. 遥感学报, 25(2): 530-538 [DOI: 10.11834/jrs.20219080http://dx.doi.org/10.11834/jrs.20219080]
文章被引用时，请邮件提醒。
提交
相关作者
暂无数据
相关机构
中国资源卫星应用中心
清华大学 地球系统科学系 教育部地球系统模拟重点实验室
清华大学 中国城市研究院
东亚迁徙鸟类与栖息地生态学野外观测研究站
国家测绘产品质量检验测试中心
选择翻译语言
- 地址：北京市海淀区北四环西路19号 《遥感学报》编辑部 邮编：100190
- 联系电话：010-58887052 Email：nrsb@aircas.ac.cn
- 技术支持由北京北大方正电子有限公司提供 京ICP备20021838号-9
京公网安备11010802024621 - 本系统建议在Chrome、 IE9+ 以上版本浏览器阅读本站内容，360浏览器请切换至极速模式
- Cookies帮助我们提供服务并提供个性化体验。使用本网站，即表示您同意我们使用Cookies
0
回到顶部