سابِق|تال

Recognition method of the main object of three-dimensional photogrammetric modeling of cultural relics

  • role： First author第一作者
  • Affiliation：

    State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

    College of History and Philosophy, Tarim University, Alaer 843300, China

  • Email：niu_wenyuan@whu.edu.cn
  • Introduction：牛文渊，1987年生，男，博士研究生，研究方向为摄影测量、文化遗产数字化。E-mail：niu_wenyuan@whu.edu.cn
NIU Wenyuan12,  
  • role： Corresponding author通信作者
  • Affiliation：

    State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

    Institute of Yangtze River Civilization Archaeology Research, Wuhan University, Wuhan 430072, China

    Laboratory of Associated Digital Yungang Society, Datong 037034, China

  • Email：huangxf@whu.edu.cn
  • Introduction：黄先锋，1978年生，男，教授，研究方向为摄影测量、计算机视觉、文化遗产数字化。E-mail：huangxf@whu.edu.cn
HUANG Xianfeng134*,  
  • Affiliation：

    State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

JIN Jie1,  
  • Affiliation：

    State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

MAO Zhu1,  
  • Affiliation：

    State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China

GONG Yiping1,  
  • Affiliation：

    Wuhan Daspatial Technology Co., LTD, Wuhan 430223, China

XU Jianmin5,  
  • Affiliation：

    Wuhan Daspatial Technology Co., LTD, Wuhan 430223, China

ZHAO Junhong5

ملخص

Photogrammetry technology helps us reconstruct three dimensional models of cultural relics just by taking photos. However, the background where the cultural relics are located also participates in modeling simultaneously, which wastes storage space and computing resources. Meanwhile, the independence and aesthetics of the three dimensional models are destroyed. Additionally, pure models of cultural relics are obtained by manually deleting the background in three dimensional scenes, which is time consuming and cannot satisfy the practical needs of the flourishing development of digital cultural heritage.This research aims to obtain the three dimensional pure cultural relic models by deleting the redundant background of the photogrammetric model on the basis of object recognition without manual interaction.This paper proposed a method to delete the background of the three dimensional photogrammetric model of cultural relics by objects recognition. First, we recognized the foreground of the cultural relic image by using the deep learning network Mask R-CNN and One Cut, respectively. Second, we extracted the masks of cultural relics by combining the results of Mask R-CNN and One Cut. Last, we applied the masks of cultural relics to delete the background of three dimensional cultural relic models on the basis of the mapping relationships between images and three dimensional models. Moreover, we used the multi-view constraints to optimize the three dimensional recognition accuracy. Additionally, we improved the One Cut method by automatically setting the initial value. In the processing of three dimensional projecting to two dimensional, regarding the cases where triangles overlap, we applied the depth information to distinguish the triangles of foreground and background in three dimensional models.To evaluate proposed method, two cultural relics were selected for the experiments, including Buddha statues in the Beilin Museum in Shaanxi and Mayan masks in the Mexican Museum. We took photos of them and obtained three dimensional models via GET3D (get3d. cn). Our method performs effectively for the Buddha model and the Mayan masks model. Apparently, most of the background of the models is eliminated, and the main bodies of the models are completely preserved. Compared with the artificially labeled ground truth, it can be found that 1) our method preserved three dimensional models complete with a satisfactory recall of 99.23% and 99.20% for the Buddha model and the Mayan masks model, respectively; 2) the algorithm erased the triangles of background with a simplification rate of 85.34% and 86.44% for the Buddha model and the Mayan masks model, respectively; 3) with the advantage of the multi-view constraints, the recognition accuracy of the three dimensional model is higher than two dimensional image.The method proposed in this paper can automatically delete the background of the three dimensional photogrammetric model without manual intervention and preserve the integrity of the object well. The experimental results demonstrate the proposed method is feasible and effective. However, when applied to large three dimensional models, our method is limited to efficiency, given that we distinguished the overlapped triangles successively. Moreover, our pipeline provides a reference for recognizing three dimensional objects in various three dimensional scenes.

مفهوم

remote sensing;Cultural Relics Digitization;deep learning;One Cut;Three-Dimensional Saliency Detection;Main Object Recognition

References

  1. 1.
    Aldoma A, Tombari F, Di Stefano L and Vincze M. 2012. A global hypotheses verification method for 3D object recognition//Lazebnik A, Fitzgibbon S, Perona P, Sato Y and Schmidt C, eds. Computer Vision - ECCV 2012. Florence, Italy: Springer: 511-524
  2. 2.
    Borji A. 2015. What is a salient object? A dataset and a baseline model for salient object detection. IEEE Transactions on Image Processing, 24(2): 742-756
  3. 3.
    Boykov Y, Veksler O and Zabih R. 2001a. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11): 1222-1239
  4. 4.
    Boykov Y Y and Jolly M P. 2001b. Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images//Proceedings Eighth IEEE International Conference on Computer Vision. Vancouver, BC, Canada: IEEE: 105-112
  5. 5.
    Carvalho L E and von Wangenheim A. 2019. 3D object recognition and classification: a systematic literature review. Pattern Analysis and Applications, 22(4): 1243-1292 DOI: ]
  6. 6.
    Chen H T. 2010. Preattentive co-saliency detection//2010 IEEE International Conference on Image Processing. Hong Kong, China: IEEE: 1117-1120
  7. 7.
    Ch’ng E, Cai S D, Zhang T E and Leow F T. 2019. Crowdsourcing 3D cultural heritage: best practice for mass photogrammetry. Journal of Cultural Heritage Management and Sustainable Development, 9(1): 24-42
  8. 8.
    Cong R M, Lei J J, Fu H Z, Cheng M M, Lin W S and Huang Q M. 2019. Review of visual saliency detection with comprehensive information. IEEE Transactions on Circuits and Systems for Video Technology, 29(10): 2941-2959
  9. 9.
    Fu K R, Gu I Y H, Yun Y X, Gong C and Yang J. 2014. Graph construction for salient object detection in videos//2014 22nd International Conference on Pattern Recognition. Stockholm, Sweden: IEEE: 2371-2376
  10. 10.
    Gong Y P, Zhang F, Jia X Y, Huang X F, Li D R and Mao Z. 2021.“Deep Neural Networks for Quantitative Damage Evaluation ofBuilding Losses Using Aerial Oblique Images: Case Study on theGreat Wall (China).” Remote Sensing 13(7): 19
  11. 11.
    Guo Y L, Bennamoun M, Sohel F, Lu M and Wan J W. 2014. 3D object recognition in cluttered scenes with local surface features: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11): 2270-2287
  12. 12.
    Guo Y L, Wang H Y, Hu Q Y, Liu H, Liu L and Bennamoun M. 2020. Deep learning for 3D point clouds: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence
  13. 13.
    Han J W, Zhang D W, Cheng G, Liu N and Xu D. 2018. Advanced deep-learning techniques for salient and category-specific object detection: a survey. IEEE Signal Processing Magazine, 35(1): 84-100
  14. 14.
    He K M, Gkioxari G, Dollár P and Girshick R. 2017. Mask R-CNN//2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 2980-2988
  15. 15.
    Jakab M, Benesova W and Racev M. 2015. 3D object recognition based on local descriptors//Proceedings Volume 9406, Intelligent Robots and Computer Vision XXXII: Algorithms and Techniques. San Francisco, California, United States: SPIE: 94060L
  16. 16.
    Li D P, Wang H Y, Liu N, Wang X M and Xu J. 2020. 3D object recognition and pose estimation from point cloud using stably observed point pair feature. IEEE Access, 8: 44335-44345
  17. 17.
    Li H L, Meng F M and Ngan K N. 2013. Co-salient object detection from multiple images. IEEE Transactions on Multimedia, 15(8): 1896-1909
  18. 18.
    Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P and Zitnick C L. 2014. Microsoft COCO: common objects in context//Fleet D, Pajdla T, Schiele B and Tuytelaars T, eds. Computer Vision - ECCV 2014. Zurich, Switzerland: Springer: 740-555
  19. 19.
    Loaiza C, Daniel A, María M M and Gabriel M B. 2020. “Virtual Museums. Captured Reality and 3d Modeling.” Journal of Cultural Heritage 45: 234-9
  20. 20.
    Luo Y, Yuan J S and Lu J W. 2016. Finding spatio-temporal salient paths for video objects discovery. Journal of Visual Communication and Image Representation, 38: 45-54
  21. 21.
    Peng H W, Li B, Xiong W H, Hu W M and Ji R R. 2014. RGBD salient object detection: a benchmark and algorithms//Fleet D, Pajdla T, Schiele B and Tuytelaars T, eds. Computer Vision - ECCV 2014. Zurich, Switzerland: Springer: 92-109
  22. 22.
    Qu L Q, He S F, Zhang J W, Tian J D, Tang Y D and Yang Q X. 2017. RGBD salient object detection via deep fusion. IEEE Transactions on Image Processing, 26(5): 2274-2285
  23. 23.
    Rother C, Kolmogorov V and Blake A. 2004. “GrabCut”: interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics, 23(3): 309-314
  24. 24.
    Shen Z Q, Ma X and Li Y B. 2018. A hybrid 3D descriptor with global structural frames and local signatures of histograms. IEEE Access, 6: 39261-39272
  25. 25.
    Singh R D, Mittal A and Bhatia R K. 2019. 3D convolutional neural network for object recognition: a review. Multimedia Tools and Applications, 78(12): 15951-15995
  26. 26.
    Sun X, Yang B S and Li Q Q. 2011. “Structural Segmentation Method for 3d Building Models Based on Voxel Analysis.” Acta Geodeticaet Cartographica Sinica 40 (5):582-6
  27. 27.
    Tang M, Gorelick L, Veksler O and Boykov Y. 2013. GrabCut in one cut//2013 IEEE International Conference on Computer Vision. Sydney, NSW, Australia: IEEE: 1769-1776
  28. 28.
    Ullah I, Jian M W, Hussain S, Guo J, Yu H, Wang X and Yin Y L. 2020. A brief survey of visual saliency detection. Multimedia Tools and Applications, 79(45): 34605-34645
  29. 29.
    Vetrivel A., M. Gerke, N. Kerle, and G. Vosselman. 2015. Segmentation of Uav-Based Images Incorporating 3d Point Cloud Information. Paper presented at the Joint ISPRS Conference on Photogrammetric Image Analysis (PIA) and High Resolution Earth Imaging for Geospatial Information (HRIGI), Technische UnivMunchen, Munich, GERMANY, Mar 25-27
  30. 30.
    Wen W W, Wen G J, Hui B W and Qiu S H. 2018. 3D object recognition based on improved point cloud descriptors//Proceedings Volume 10806, Tenth International Conference on Digital Image Processing (ICDIP 2018. Shanghai, China: SPIE: 108060O
  31. 31.
    Yan Y M, Gao F J, Deng S P, and Su N. 2017. “A Hierarchical Building Segmentation in Digital Surface Models for 3d Reconstruction.”Sensors 17 (2) [DOI: 10.3390/s17020222]
  32. 32.
    Yang C, Zhang F., Gao Y L, Mao Z, Li L, and Huang X F. 2021. “Moving Car Recognition and Removal for 3d Urban Modelling Using Oblique Images.” Remote Sensing 13 (17): 19
  33. 33.
    Zhao Z Q, Zheng P, Xu S T and Wu X D. 2019. Object detection with deep learning: a review. IEEE Transactions on Neural Networks and Learning Systems, 30(11): 3212-3232

قراءة النص الكامل

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website