Anterior|Siguiente

Real-time dense point cloud generation and digital model construction of surface environment based on UAV platform

  • role： First author第一作者
  • Affiliation：

    Northwestern Polytechnical University, Xi'an 710072, China

  • Email：huboni@mail.nwpu.edu.cn
  • Introduction：胡博妮，研究方向为深度学习、复杂环境下的无人机定位与感知。E-mail： huboni@mail.nwpu.edu.cn
HU Boni1,  
  • Affiliation：

    Northwestern Polytechnical University, Xi'an 710072, China

CHEN Lin1,  
  • Affiliation：

    Army Armored Force Academy, Beijing 100072, China

XU Bingli2,  
  • role： Corresponding author通信作者
  • Affiliation：

    Northwestern Polytechnical University, Xi'an 710072, China

  • Email：bushuhui@nwpu.edu.cn
  • Introduction：布树辉，研究方向为实时建图与智能感知。E-mail： bushuhui@nwpu.edu.cn
BU Shuhui1*,  
  • Affiliation：

    Northwestern Polytechnical University, Xi'an 710072, China

HAN Pengcheng1,  
  • Affiliation：

    Northwestern Polytechnical University, Xi'an 710072, China

LI Kun1,  
  • Affiliation：

    Northwestern Polytechnical University, Xi'an 710072, China

XIA Zhenyu1,  
  • Affiliation：

    Northwestern Polytechnical University, Xi'an 710072, China

LI Ni1,  
  • Affiliation：

    University of Information Engineering, Zhengzhou 450000, China

LI Ke3,  
  • Affiliation：

    University of Information Engineering, Zhengzhou 450000, China

CAO Xuefeng3,  
  • Affiliation：

    University of Aerospace Engineering, Beijing 101400, China

WAN Gang4

resumen

The development of high-fidelity 3D digital model of the land surface environment in real time has become essential in many fields, including urban planning, agricultural surveying and mapping, disaster management, and military applications. Building a high-fidelity digital model of the land surface environment in real time is the key foundation for realizing the virtual mapping of the geographic environment and then forming a digital twin geographic environment. However, current methods for constructing these models suffer from several challenges such as slow speed, low timeliness, and limited application in large scenarios. To overcome these challenges, this paper proposes a new real-time dense point cloud generation and digital model construction method based on unmanned aerial vehicle (UAV) platforms. The proposed algorithm remarkably improves the speed and accuracy of land surface model construction compared with existing algorithms and improves the degree of online data collection and real-time modeling. The algorithm is based on a general simultaneous localization and mapping framework. The technical chain from data acquisition, data processing, feature extraction and matching to 3D point cloud generation, digital model reconstruction, and result analysis is unobstructed. The algorithm breaks through the key technologies in scene reconstruction such as online data acquisition and pose problem solving, real-time dense point cloud generation, digital surface model, and digital orthophoto map construction.According to the experimental results of the dataset containing different land surface environments such as cities, farmland, mountains, and deserts, the algorithm proposed in this paper can process high-quality land surface environment dense point clouds and digital models while being 30—50 times faster than existing algorithms such as Pix4DMapper. On average, it can process a high-resolution image in less than 1 second, whereas the interval for general aerial survey drones is 1—2 seconds. Furthermore, the application of the algorithm proposed in this paper to emergency rescue work of mountain flood disasters can assist in the real-time reconstruction of inaccessible areas, survey the area and location of washed-out and sediment areas, and support emergency rescue work.The proposed real-time dense point cloud generation and digital model construction method based on UAV platforms represents a considerable advancement in the field of geographic information systems. It can revolutionize various geographic fields, including disaster warning and management, emergency response, military applications, and agricultural and urban planning. Additionally, due to the breakthrough of the algorithm, it solves the core problem of real-time mapping from reality to virtual in the construction of digital twins in geographic environments. Therefore, it can empower digital twins and provide a digital foundation for digital twin applications in geographic environments. Based on the real-time mapping of 3D information of the ground environment, parallel simulation and decision making can be conducted. Therefore, it is also expected to achieve functions such as emergency warning, scheme evaluation, and decision optimization based on the dynamic monitoring of target information in the scene, further improving the automation and intelligence level of the applied system.

palabra clave

remote sensing;Real-time reconstruction;Dense Point cloud;digital twin;Terrain environment;Digital model

References

  1. 1.
    Arandjelović R and Zisserman A. 2012. Three things everyone should know to improve object retrieval//2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence: IEEE: 2911-2918
  2. 2.
    Bailey T, Nieto J, Guivant J, Stevens M and Nebot E. 2006. Consistency of the EKF-SLAM algorithm//2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. Beijing: IEEE: 3562-3568
  3. 3.
    Barnes C, Shechtman E, Finkelstein A and Goldman D B. 2009. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 28(3): 24
  4. 4.
    Barroso-Laguna A and Mikolajczyk K. 2023. Key.Net: keypoint detection by handcrafted and learned CNN filters revisited. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1): 698-711
  5. 5.
    Bay H, Tuytelaars T and Van Gool L. 2006. SURF: speeded up robust features//9th European Conference on Computer Vision. Graz: Springer: 404-417
  6. 6.
    Beevers K R and Huang W H. 2007. Fixed-lag sampling strategies for particle filtering SLAM//Proceedings 2007 IEEE International Conference on Robotics and Automation. Rome: IEEE: 2433-2438
  7. 7.
    Bloesch M, Czarnowski J, Clark R, Leutenegger S and Davison A J. 2018. CodeSLAM - learning a compact, optimisable representation for dense visual SLAM//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE: 2560-2568
  8. 8.
    Boykov Y, Veksler O and Zabih R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11): 1222-1239
  9. 9.
    Bu S H, Zhao Y, Wan G and Liu Z B. 2016. Map2DFusion: real-time incremental UAV image mosaicing based on monocular SLAM//2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Daejeon: IEEE: 4564-4571
  10. 10.
    Calonder M, Lepetit V, Strecha C and Fua P. 2010. BRIEF: binary robust independent elementary features//11th European Conference on Computer Vision. Heraklion: Springer: 778-792
  11. 11.
    Chang D, Božič A, Zhang T, Yan Q S, Chen Y C, Süsstrunk S and Nießner M. 2022. RC-MVSNet: unsupervised multi-view stereo with neural rendering//17th European Conference on Computer Vision. Tel Aviv: Springer: 665-680
  12. 12.
    Chen L, Zhao Y, Xu S B, Bu S H, Han P C and Wan G. 2020. DenseFusion: large-scale online dense pointcloud and DSM mapping for UAVs//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas: IEEE: 4766-4773
  13. 13.
    Cheng H M, Xu D L and Wang S T. 2021. Application of geophysical prospecting in emergency responseto landslide disaster--A case study of Shaziba Landslide in Enshi, Huabei. Chinese Journal of Engineering Geophysics, 18(2): 273-281
  14. 14.
    DeTone D, Malisiewicz T and Rabinovich A. 2018. SuperPoint: self-supervised interest point detection and description//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE: 337-349
  15. 15.
    Forster C, Pizzoli M and Scaramuzza D. 2014. SVO: fast semi-direct monocular visual odometry//2014 IEEE International Conference on Robotics and Automation (ICRA). Hong Kong, China: IEEE: 15-22
  16. 16.
    Furukawa Y and Ponce J. 2010. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8): 1362-1376
  17. 17.
    Galliani S, Lasinger K and Schindler K. 2015. Massively parallel multiview stereopsis by surface normal diffusion//Proceedings of the 2015 IEEE International Conference on Computer Vision. Santiago: IEEE: 873-881
  18. 18.
    Goesele M, Snavely N, Curless B, Hoppe H and Seitz S M. 2007. Multi-view stereo for community photo collections//2007 IEEE 11th International Conference on Computer Vision. Rio de Janeiro: IEEE: 1-8
  19. 19.
    Guo J T, Hong H B, Zhong K K, Liu X J and Guo Y. 2020. Production management and control method of aerospace manufacturing workshops based on digital twin. China Mechanical Engineering, 31(7): 808-814
  20. 20.
    Han P C, Ma C B, Chen J, Chen L, Bu S H, Xu S B, Zhao Y, Zhang C H and Hagino T. 2022. Fast tree detection and counting on UAVs for sequential aerial images with generating orthophoto mosaicing. Remote Sensing, 14(16): 4113
  21. 21.
    Hinzmann T, Schönberger J L, Pollefeys M and Siegwart R. 2018. Mapping on the fly: real-time 3D dense reconstruction, digital surface map and incremental orthomosaic generation for unmanned aerial vehicles//Hutter M and Siegwart R, eds. Field and Service Robotics. Cham: Springer: 383-396
  22. 22.
    Jiang N J, Cui Z P and Tan P. 2013. A global linear method for camera pose registration//Proceedings of the 2013 IEEE International Conference on Computer Vision. Sydney: IEEE: 481-488
  23. 23.
    Kazhdan M and Hoppe H. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics, 32(3): 29
  24. 24.
    Kern A, Bobbe M, Khedar Y and Bestmann U. 2020. OpenREALM: real-time mapping for unmanned aerial vehicles//2020 International Conference on Unmanned Aircraft Systems (ICUAS). Athens: IEEE: 902-911
  25. 25.
    Klein G and Murray D. 2007. Parallel tracking and mapping for small AR workspaces//2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Nara: IEEE: 225-234
  26. 26.
    Lhuillier M and Quan L. 2005. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3): 418-433
  27. 27.
    Li D R. 2021. Smart cities based on digital twins. Internet World, (7): 12
  28. 28.
    Li H, Tao F, Wang H Q, Song W Y, Zhang Z F, Fan B B, Wu C L, Li Y P, Li L L, Wen X Y, Zhang X S and Luo G F. 2019. Integration framework and key technologies of complex product design-manufacturing based on digital twin. Computer Integrated Manufacturing Systems, 25(6): 1320-1336
  29. 29.
    Liu C J, Lü J, Ren M L, Chen S, Zhang X L, Song W L and Zhang D W. 2022. Research and application of digital twin intelligent flood prevention system in Huaihe River Basin. China Flood and Drought Management, 32(1): 47-53
  30. 30.
    Lorensen W E and Cline H E. 1987. Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics, 21(4): 163-169
  31. 31.
    Lowe D G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2): 91-110
  32. 32.
    Mildenhall B, Srinivasan P P, Tancik M, Barron J T, Ramamoorthi R and Ng R. 2020. NeRF: representing scenes as neural radiance fields for view synthesis//16th European Conference on Computer Vision. Glasgow: Springer: 405-421
  33. 33.
    Muja M and Lowe D G. 2009. Fast approximate nearest neighbors with automatic algorithm configuration//Proceedings of the Fourth International Conference on Computer Vision Theory and Applications. Lisboa: SciTePress: 331-340
  34. 34.
    Mur-Artal R, Montiel J M M and Tardós J D. 2015. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 31(5): 1147-1163
  35. 35.
    Nealen A, Igarashi T, Sorkine O and Alexa M. 2006. Laplacian mesh optimization//Proceedings of the 4th international Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia. Kuala Lumpur: ACM: 381-389
  36. 36.
    Pan X C, Jiang T, Yu A Z, Wang X and Zhang Y. 2019. Geo-positioning of remote sensing images with reference image. Journal of Remote Sensing (in Chinese), 23(4): 673–684 DOI: .
  37. 37.
    Rosten E and Drummond T. 2006 Machine learning for high-speed corner detection//9th European Conference on Computer Vision. Graz: Springer: 430-443
  38. 38.
    Rublee E, Rabaud V, Konolige K and Bradski G. 2011. ORB: an efficient alternative to SIFT or SURF//2011 International Conference on Computer Vision. Barcelona: IEEE: 2564-2571
  39. 39.
    Sarlin P E, DeTone D, Malisiewicz T and Rabinovich A. 2020. SuperGlue: learning feature matching with graph neural networks//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE: 4937-4946
  40. 40.
    Schönberger J L and Frahm J M. 2016. Structure-from-motion revisited//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 4104-4113
  41. 41.
    Snavely N, Seitz S M and Szeliski R. 2008. Modeling the world from internet photo collections. International Journal of Computer Vision, 80(2): 189-210
  42. 42.
    Sorkine O and Cohen-Or D. 2004. Least-squares meshes//Proceedings Shape Modeling Applications, 2004. Genova: IEEE: 191-199
  43. 43.
    Strasdat H, Montiel J M M and Davison A J. 2012. Visual SLAM: why filter?. Image and Vision Computing, 30(2): 65-77
  44. 44.
    Tancik M, Casser V, Yan X C, Pradhan S, Midenhall B P, Srinivasan P, Barron J T and Kretzschmar H. 2022. Block-NeRF: scalable large scene neural view synthesis//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE: 8238-8248
  45. 45.
    Tao F, Liu W R, Liu J H, Liu X J, Liu Q, Qu T, Hu T L, Zhang Z N, Xiang F, Xu W J, Wang J Q, Zhang Y F, Liu Z Y, Li H, Cheng J F, Qi Q L, Zhang M, Zhang H, Sui F Y, He L R, Yi W M and Cheng H. 2018. Digital twin and its potential application exploration. Computer Integrated Manufacturing Systems, 24(1): 1-18
  46. 46.
    Tola E, Strecha C and Fua P. 2012. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Machine Vision and Applications, 23(5): 903-920
  47. 47.
    Turki H, Ramanan D and Satyanarayanan M. 2022. Mega-NeRF: scalable construction of large-scale NeRFs for virtual fly-throughs//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE: 12912-12921
  48. 48.
    Waechter M, Moehrle N and Goesele M. 2014. Let there be color! Large-scale texturing of 3D reconstructions//13th European Conference on Computer Vision. Zurich: Springer: 836-850
  49. 49.
    Wang H P, Liu Y, Hu Q Y, Wang B, Chen J G, Dong Z, Guo Y L, Wang W P and Yang B S. 2023. RoReg: pairwise point cloud registration with oriented descriptors and local rotations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8): 10376-10393
  50. 50.
    Wang W, Zhao Y, Han P C, Zhao P C and Bu S H. 2019. TerrainFusion: real-time digital surface model reconstruction based on monocular SLAM//2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Macau, China: IEEE: 7895-7902
  51. 51.
    Wei Y, Liu S H, Rao Y M, Zhao W, Lu J W and Zhou J. 2021. NerfingMVS: guided optimization of neural radiance fields for indoor multi-view stereo//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal: IEEE: 5590-5599
  52. 52.
    Wu C. 2011. VisualSFM: a visual structure from motion system. [2023-03-13]..
  53. 53.
    Xia R L, Li T, Yu W and Li Y R. 2021. Digital twin theory of watershed and its application in flood control of the Yellow River. China Water Conservancy, (20): 11-13
  54. 54.
    Yang S T, Wang P F, Wang J, Lou H Z and Gong T L. 2021. River flow estimation method based on UAV aerial photogrammetry. National Remote Sensing Bulletin, 25(6): 1284-1293
  55. 55.
    Yao Y, Luo Z X, Li S W, Fang T and Quan L. 2018. MVSNet: depth inference for unstructured multi-view stereo//Proceedings of the 15th European Conference on Computer Vision. Munich: Springer: 785-801
  56. 56.
    Yu Z and Gao S H. 2020. Fast-MVSNet: sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE: 1946-1955
  57. 57.
    Zhang J X, Liu F and Wang J. 2021. Review of the light-weighted and small UAV system for aerial photography and remote sensing. National Remote Sensing Bulletin, 25(3): 708-724
  58. 58.
    Zhao Y, Zhao P C, Xu S B, Chen L, Han P C, Bu S H and Jiang H K. 2021. Svar: a tiny C++ header brings unified interface for multiple programming languages. arXiv:2108.08464
  59. 59.
    Zhou T H, Brown M, Snavely N and Lowe D G. 2017. Unsupervised learning of depth and ego-motion from video//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE: 6612-6619
  60. 60.
    Zhu Q, Zhu J, Huang H P, Wang W and Zhang L G. 2020. Real 3D spatial information platform and digital twin Sichuan-Tibet railway. High Speed Railway Technology, 11(2): 46-53
  61. 61.
    Zhu Z H, Peng S Y, Larsson V, Xu W W, Bao H J, Cui Z P, Oswald M R and Pollefeys M. 2021. NICE-SLAM: neural implicit scalable encoding for SLAM//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE: 12776-12786

Leer el texto completo

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website