Real-time dense point cloud generation and digital model construction of surface environment based on UAV platform
- Pages: 1-17(2023)
DOI： 10.11834/jrs.20232597
Quote
扫 描 看 全 文
- About Journal
- Journal Online
- Special Source
摘要
实时构建地表环境高逼真数字模型一直是遥感航测领域的研究热点，也是实现地理环境的虚拟映射并进而形成数字孪生地理环境的关键基础。针对当前地表环境三维数字模型构建中存在的速度慢、时效性低和大场景应用受限等问题，本文在综述现有国内外研究的基础上，结合团队研究基础，提出一个基于无人机平台的地表环境实时稠密点云生成与数字模型构建方法。研究通用实时定位与地图构建框架，打通了从数据获取到三维点云生成、数字模型重建、结果分析等技术链条，设计动态内存管理模块和联合GNSS优化的PI-SLAM，突破了在线获取数据与位姿解算、实时稠密点云生成、实时数字表面模型（DSM）和数字正射影像图（DOM）构建等关键技术。经实验验证，方法在稠密点云生成和数字模型构建精度与现有离线建模算法相当的同时，速度提升了30-50倍，达到在线采集数据并实时建模的程度。应用案例表明，本方法可应用于灾害预警救援、应急管理、作战模拟等高时效地形三维重建中，同时能够为数字孪生地理环境数字底座构建提供技术支撑。
Abstract
The development of high-fidelity 3D digital models of the land surface environment in real-time has become essential in many fields, including urban planning, agricultural surveying and mapping, disaster management, and military applications. Building a high-fidelity digital model of the land surface environment in real time is the key foundation for realizing the virtual mapping of the geographic environment and then forming a digital twin geographic environment.However, current methods for constructing these models suffer from several challenges such as slow speed, low timeliness, and limited application in large scenarios.To overcome these challenges, this paper proposes a new real-time dense point cloud generation and digital model construction method based on unmanned aerial vehicle (UAV) platforms. The proposed algorithm significantly improves the speed and accuracy of land surface model construction compared to existing algorithms, it enables the degree of online data collection and real-time modeling. The algorithm is based on a general simultaneous localization and mapping framework, the technical chain from data acquisition, data processing, feature extraction and matching to 3D point cloud generation, digital model reconstruction and result analysis is unobstructed. At the meantime, the algorithm breaks through the key technologies in scene reconstruction such as online data acquisition and pose problem solving, real-time dense point cloud generation, digital surface model (DSM) and digital orthophoto map (DOM) construction.According to the experimental results of the dataset containing different land surface environments such as cities, farmland, mountains, and deserts, the algorithm proposed in this paper can process high-quality land surface environment dense point clouds and digital models while being 30-50 times faster than existing algorithms such as Pix4DMapper. On average, it can process a high-resolution image in less than one second, while the interval for general aerial survey drones is 1-2 seconds. Furthermore, the application of the algorithm proposed in this paper to emergency rescue work of mountain flood disasters can assist in real-time reconstruction of inaccessible areas, surveying the area and location of washed-out and sediment areas, and support emergency rescue work.In conclusion, the proposed real-time dense point cloud generation and digital model construction method based on UAV platforms represents a significant advancement in the field of geographic information systems. It has the potential to revolutionize various geographic fields, including disaster warning and management, emergency response, military applications, and agricultural and urban planning. Additionally, due to the breakthrough of the algorithm, the core problem of real-time mapping from reality to virtual in the construction of digital twins in geographic environments has been solved. Therefore, it can empower digital twins and provide a digital foundation for digital twin applications in geographic environments. Based on the real-time mapping of three-dimensional information of the ground environment, parallel simulation and decision-making can be carried out. Therefore, it is also expected to achieve functions such as emergency warning, scheme evaluation and decision optimization based on the dynamic monitoring of target information in the scene, further improving the automation and intelligence level of the applied system.
关键词
实时重建稠密点云数字孪生地表环境数字模型
Keywords
Real-time reconstructionDense Point cloudDigital twinTerrain environmentDigital model
references
Arandjelović R and Zisserman. 2012. A ,"Three things everyone should know to improve object retrieval," 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, 2911-2918 [DOI: 10.1109/CVPR.2012.6248018http://dx.doi.org/10.1109/CVPR.2012.6248018]
Barnes C, Shechtman E, Finkelstein A, & Goldman DB. (2009). PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph:28-24
Barroso Laguna A, Mikolajczyk K. 2022.Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters Revisited. IEEE Trans Pattern Anal Mach Intell.[DOI: 10.1109/TPAMI.2022.3145820http://dx.doi.org/10.1109/TPAMI.2022.3145820]
Bay H, Tuytelaars T, Van Gool L. 2006. SURF: Speeded Up Robust Features. Computer Vision–ECCV 2006. Lecture Notes in Computer Science, vol 3951. [DOI:10.1007/11744023_32http://dx.doi.org/10.1007/11744023_32]
Beevers KR, & Huang WH. 2007. Fixed-lag Sampling Strategies for Particle Filtering SLAM. Proceedings 2007 IEEE International Conference on Robotics and Automation, 2433-2438
Bloesch, M, Czarnowski,J, Clark,R Leutenegger,S and Davison,A,J. 2018. CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition:2560-2568[DOI:10.1109/CVPR.2018.00271http://dx.doi.org/10.1109/CVPR.2018.00271]
Boykov Y, Veksler O, Zabih R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on pattern analysis and machine intelligence, 2001, 23(11): 1222-1239
Bruno Bodin, Harry Wagstaff, Sajad Saeedi, Luigi Nardi, Emanuele Vespa, John H Mayer, Andy Nisbet, Mikel Luján, Calonder M, Lepetit V, Strecha C, et al. 2010. Brief: Binary robust independent elementary features. European conference on computer vision. Springer, Berlin, Heidelberg, 2010: 778-792
Bu S, Zhao Y, Wan G, & Liu Z. 2016. Map2DFusion: Real-time incremental UAV image mosaicing based on monocular SLAM. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4564-4571
Cernea D. 2023. OpenMVS: Open Multiple View Stereovision. [2023-03-28]https://github.com/cdcseacave/openMVS/https://github.com/cdcseacave/openMVS/
C. Forster, M. Pizzoli and D. Scaramuzza. 2014. "SVO: Fast semi-direct monocular visual odometry," 2014 IEEE International Conference on Robotics and Automation (ICRA): 15-22[DOI: 10.1109/ICRA.2014.6906584http://dx.doi.org/10.1109/ICRA.2014.6906584]
Chang D, Bozic A, Zhang T, Yan Q, Chen Y, Süsstrunk S, & Nießner M. 2022. RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering. European Conference on Computer Vision
CHEN Bao，WANG Pinhe，DONG Qiulei. 2022. Fast 3D Reconstruction of Satellite Images via global affine model. National Remote Sensing Bulletin，1-14
陈豹，王品贺，董秋雷.利用全局仿射模型进行卫星图像快速三维重建.遥感学报:1-14 [DOI： 10.11834/jrs.20222039http://dx.doi.org/10.11834/jrs.20222039]
Chen L, Zhao Y, Xu S, Bu S, Han P, & Wan G. 2020. DenseFusion: Large-Scale Online Dense Pointcloud and DSM Mapping for UAVs. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4766-4773
Cheng HM, Xu DL, Wang ST. 2021. The role of geophysical prospecting technology in the emergency handling of landslide disasters: A case study of the Shaziba landslide in Tunbao Township, Enshi Prefecture, Hubei Province. Journal of Engineering Geophysics.
程怀蒙, 徐玳笠, and 王思桐.(2021)"滑坡灾害应急处置中物探技术的作用——以湖北恩施州屯堡乡沙子坝滑坡为例." 工程地球物理学报
DeTone D, Malisiewicz T, Rabinovich. 2018. A. Superpoint: Self-supervised interest point detection and description. Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018: 224-236
Furukawa Y, Ponce J. Accurate, dense, and robust multiview stereopsis. 2009. IEEE transactions on pattern analysis and machine intelligence, 2009, 32(8): 1362-1376
Galliani S, Lasinger K, Schindler K. 2015. Massively parallel multiview stereopsis by surface normal diffusion. Proceedings of the IEEE International Conference on Computer Vision. 2015: 873-881
Goesele M, Snavely N, Curless B, et al. 2007. Multi-view stereo for community photo collections. 2007 IEEE 11th International Conference on Computer Vision. IEEE, 2007: 1-8
Guo JT, Hong HB, Zhong KK, Liu XJ. & Guo Y. 2020. Production control method of aerospace manufacturing workshop based on digital twin. Chinese Mechanical Engineering.2020,31(07):808-814
郭具涛,洪海波,钟珂珂,刘骁佳,郭宇.基于数字孪生的航天制造车间生产管控方法.中国机械工程,2020,31(07):808-814
Wang HP, Liu Y, Hu QY, Wang B, Chen JG, Dong Z, Guo YL, Wang WP. & Yang BS. 2023.RoReg: Pairwise Point Cloud Registration with Oriented Descriptors and Local Rotations. IEEE Transactions on Pattern Analysis and Machine Intelligence[DOI: 10.1109/TPAMI.2023.3244951http://dx.doi.org/10.1109/TPAMI.2023.3244951]
Han P, Ma C, Chen J, Chen L, Bu S, Xu S, Zhao Y, Zhang C, & Hagino T. 2022. Fast Tree Detection and Counting on UAVs for Sequential Aerial Images with Generating Orthophoto Mosaicing. Remote. Sens., 14, 4113
Hinzmann T, Schönberger J L, Pollefeys M. 2018. Mapping on the fly: real-time 3d dense reconstruction, digital surface map and incremental orthomosaic generation for unmanned aerial vehicles. Field and Service Robotics. Springer, 2018: 383-396.
Jiang N, Cui Z, Tan P. 2013. A global linear method for camera pose registration. Proceedings of the IEEE International Conference on Computer Vision. 2013: 481-488
Juan Andrés Cardoso Arango, & Mounir Louhaichi. 2019. Protocol for data collection and processing from UAVs imageryusingOpenDroneMap[2023-01-02].https://www.researchgate.net/publication/339747134_Protocol_for_data_collection_and_processing_from_UAVs_imagery_using_OpenDroneMaphttps://www.researchgate.net/publication/339747134_Protocol_for_data_collection_and_processing_from_UAVs_imagery_using_OpenDroneMap
Kazhdan M, Hoppe H.2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (ToG), 32(3): 1-13
Kern A.M, Bobbe Y, Khedar and U. Bestmann.2020."OpenREALM: Real-time Mapping for Unmanned Aerial Vehicles," 2020 International Conference on Unmanned Aircraft Systems (ICUAS), Athens, Greece. 902-911[DOI: 10.1109/ICUAS48674.2020.9213960http://dx.doi.org/10.1109/ICUAS48674.2020.9213960]
Klein G, Murray D.2007. Parallel tracking and mapping for small AR workspaces. 2007 6th IEEE and ACM international symposium on mixed and augmented reality. IEEE, 2007: 225-234
Küng, O, Strecha, C, Beyeler, A, Zufferey, J, Floreano, D, Fua, P.V, & Gervaix, F. 2012. The Accuracy of Automatic Photogrammetric Techniques on Ultra-light UAV Imagery
Lhuillier M, Quan L. 2005. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE transactions on pattern analysis and machine intelligence, 2005, 27(3): 418-433.
Li DR.2021. Smart cities based on digital twins.Internet world.2021(07):12
李德仁.基于数字孪生的智慧城市.互联网天地,2021(07):12
Li H, Tao F, Wang HQ, Song WY, Zhang ZF, Fan BB, Wu CL, Li YP, Li LL, Wen XY,Zhang XS, Luo GF.2019. Integrated development framework and key technologies for complex product design and manufacturing based on digital twin.Computer integrated manufacturing systems
李浩,陶飞,王昊琪,宋文燕,张在房,樊蓓蓓,武春龙,李玉鹏,李琳利,文笑雨,张新生,罗国富.基于数字孪生的复杂产品设计制造一体化开发框架与关键技术.计算机集成制造系统,25(06):1320-1336[DOI:10.13196/j.cims.2019.06.002http://dx.doi.org/10.13196/j.cims.2019.06.002]
Liu CJ, Lv J, Ren ML, Chen S, Zhang XL, Song WL, Zhang DW. 2022. Research and practice of intelligent flood control system in the Huaihe River Basin of digital twin. China flood prevention and drought control.2022,32(01):47-53
刘昌军,吕娟,任明磊,陈胜,张晓蕾,宋文龙,张大伟.数字孪生淮河流域智慧防洪体系研究与实践.中国防汛抗旱,2022,32(01):47-53[DOI:10.16867/j.issn.1673-9264.2021375http://dx.doi.org/10.16867/j.issn.1673-9264.2021375]
Lorensen W E , Cline H E .1987. Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics, 1987:163-169
Lowe D G. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2): 91-110
Lowe G. 2004. Sift-the scale invariant feature transform. Int. J., 2004, 2:91-110
Tancik M, Casser V, Yan XC, Pradhan., Midenhall., Srinivasan P.P, Barron T.J., & Kretzschmar H.2022.Block-NeRF: Scalable Large Scene Neural View Synthesis, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 8238-8248[DOI: 10.1109/CVPR52688.2022.00807http://dx.doi.org/10.1109/CVPR52688.2022.00807]
Mildenhall B, Srinivasan P P, Tancik M, Barron T.J., Ramamoorthi R. & Ng R.2020. Nerf: Representing scenes as neural radiance fields for view synthesis. European Conference on Computer Vision. Springer, Cham, 2020: 405-421
Muja M, Lowe D G. 2009. Fast approximate nearest neighbors with automatic algorithm configuration. In VISAPP International Conference on Computer Vision Theory and Applications. 2009: 331-340
Mur-Artal R, Montiel JM, & Tardós JD. 2015. ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics, 31, 1147-1163
Nealen A, Igarashi T, Sorkine-Hornung O, & Alexa M. 2006. Laplacian mesh optimization. Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia.. 2006: 381-389
Rosten E, Drummond T. (2006) Machine learning for high-speed corner detection. European conference on computer vision. Springer, Berlin, Heidelberg, 2006: 430-443.
Rublee E, Rabaud V , Konolige K , & Bradski G. 2011. ORB: an efficient alternative to SIFT or SURF. IEEE International Conference on Computer Vision,pp. 2564-2571 [DOI: 10.1109/ICCV.2011.6126544http://dx.doi.org/10.1109/ICCV.2011.6126544]
Sarlin P E, DeTone D, Malisiewicz T, & Rabinovich A.2020. Superglue: Learning feature matching with graph neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 4938-4947
Schonberger J L, Frahm J M. 2016.Structure-from-Motion Revisited. IEEE Conference on Computer Vision & Pattern Recognition. IEEE, 2016:4104-4113
Snavely N, Seitz S M, Szeliski R. 2008. Modeling the world from internet photo collections. International journal of computer vision, 2008, 80(2): 189-210
Sorkine O, Cohen-Or D.2004. Least-squares meshes. Proceedings Shape Modeling Applications, 2004. IEEE, 2004: 191-199.
Strasdat H, Montiel J M M, Davison A J. 2012. Visual SLAM: why filter?. Image and Vision Computing, 2012, 30(2): 65-77.
T. Bailey, J. Nieto, J. Guivant, M. Stevens and E. Nebot, 2006. Consistency of the EKF-SLAM Algorithm, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems :3562-3568 [DOI: 10.1109/IROS.2006.281644http://dx.doi.org/10.1109/IROS.2006.281644]
Tao F, Liu WR, Liu JH., Liu XJ., Liu Q., Qu T., Hu TL., Zhang ZN., Xiang F., Xu WJ., Wang JQ., Zhang YF., Liu ZY., Li H., Cheng JF., Qi QL., Zhang M., Zhang H., Sui FY., He LR., Yi WM. & Cheng H. 2018. Digital twinning and its application. Computer integrated manufacturing system, 2018, 24(1):18
陶飞, 刘蔚然, 刘检华, 刘晓军， 刘强， 屈挺， 胡天亮， 张执南， 向峰， 徐文君， 王军强， 张映峰， 刘振宇， 李浩， 程江峰， 戚庆林， 张萌， 张贺， 隋芳源， 何立荣， 易旺民 & 程珲.数字孪生及其应用探索. 计算机集成制造系统, 2018, 24(1):18
Tola E, Strecha C, Fua P. 2012. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Machine Vision and Applications, 2012, 23(5): 903-920.
Turki, H, Ramanan, D, & Satyanarayanan, M. 2022. Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR):12912-12921
Waechter M, Moehrle N, Goesele M.2014. Let there be color! Large-scale texturing of 3D reconstructions. European conference on computer vision. Springer, Cham, 2014: 836-850.
Wang W, ZhaoY, Han P, Zhao P, & Bu S. (2019). TerrainFusion: Real-time Digital Surface Model Reconstruction based on Monocular SLAM. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 7895-7902
Wei, Y, Liu, S, Rao, Y, Zhao, W, Lu, J, & Zhou, J. 2021. NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 5590-5599.
Wu C. 2011. VisualSFM: A visual structure from motion system.
Wu C. 2007. SiftGPU : A GPU Implementation of Scale Invariant Feature Transform (SIFT).
Xia RL, Li T, Yu W, Li YR. 2021. Basin digital twin theory and its practice in flood control of the Yellow River.China Water Conservancy.2021(20):11-13.
夏润亮,李涛,余伟,李英睿.流域数字孪生理论及其在黄河防汛中的实践[J].中国水利,2021(20):11-13
Yao Y, Luo Z, Li SW, Fang T. & Quan L.2018. Mvsnet: Depth inference for unstructured multi-view stereo. Proceedings of the European Conference on Computer Vision (ECCV). 2018: 767-783
Yang S T，Wang P F，Wang J，Lou H Z and Gong T L. 2021. River flow estimation method based on UAV aerial photogrammetry. National Remote Sensing Bulletin， 25（6）：1284-1293
杨胜天，王鹏飞，王娟，娄和震，巩同梁.2021.结合无人机航空摄影测量的河道流量估算.遥感学报，25（6）： 1284-1293[DOI： 10.11834/jrs.20209082http://dx.doi.org/10.11834/jrs.20209082]
Yu Z, Gao S. 2020. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 1949-1958.
Zhu, Z, Peng, S, Larsson, V, Xu, W, Bao, H, Cui, Z, Oswald, M.R, & Pollefeys, M. 2021. NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12776-12786
Zhang J X，Liu F and Wang J. 2021. Review of the light-weighted and small UAV system for aerial photography and remote sensing. National Remote Sensing Bulletin， 25（3）：708-724
张继贤，刘飞，王坚.2021.轻小型无人机测绘遥感系统研究进展.遥感学报，25（3）：708-724 [DOI： 10.11834/jrs.20210052http://dx.doi.org/10.11834/jrs.20210052]
Zhao, Y, Zhao, P, Xu, S, Chen, L, Han, P, Bu, S, & Jiang, H. 2021. Svar: A Tiny C++ Header Brings Unified Interface for Multiple programming Languages. ArXiv, abs/2108.08464
Zhou T, Brown M, Snavely N..& Lowe G.D. 2017. Unsupervised learning of depth and ego-motion from video. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 1851-1858
Zhu Q, Zhu J, Huang HP, Wang W, &Zhang LG. 2020. Real-world 3D spatial information platform and digital twin Sichuan-Tibet railway. High-speed rail technology.2020,11(02):46-53
朱庆,朱军,黄华平,王玮,张利国.实景三维空间信息平台与数字孪生川藏铁路[J].高速铁路技术,2020,11(02):46-53
Alert me when the article has been cited
提交
相关作者
暂无数据
相关机构
State Key Laboratory of Remote Sensing Science, Aerospace Information Research Institute, Chinese Academy of Sciences
College of Forestry, Beijing Forestry University
University of Chinese Academy of Sciences
Faculty of Geographical Science, Beijing Normal University
Faculty of Geosciences and Environmental Engineering, Southwest Jiaotong University
- Postal code：100190
- Tel：010-58887052 Email：nrsb@aircas.ac.cn
- Technical support is provided by Beijing Founder electronics co., LTD 京ICP备20021838号-9
京公网安备11010802024621 - It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
- Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
0
Bulk Citation
Export to
TOP