سابِق|تال

Remote sensing knowledge graph construction and its application in typical scenarios

  • role： First author第一作者
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

  • Email：zhangyj@whu.edu.cn
  • Introduction：张永军，研究方向为摄影测量与遥感、计算机视觉等。E-mail： zhangyj@whu.edu.cn
ZHANG Yongjun,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

WANG Fei,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

LI Yansheng,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

OUYANG Song,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

WEI Dong,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

LIU Xiaojian,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

KONG Deyu,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

CHEN Ruixian,  
  • Affiliation：

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

ZHANG Bin

ملخص

Compared with the current powerful acquisition capabilities of remote sensing data, its intelligent processing and knowledge service capabilities are relatively lagging. The contradiction between the accumulation of massive multisource remote sensing data and the limited information island is becoming increasingly prominent. Therefore, there is an urgent need for effective remote sensing domain knowledge modeling technology to assist in mining the useful information of remote sensing big data and form knowledge service capabilities. A Knowledge Graph (KG) describes the concepts and their relationships in the physical world in symbolic form. It has strong knowledge modeling and reasoning capabilities and has been successfully applied in search engines, e-commerce, social network analysis and other fields. Inspired by the general KGs, this paper conceives of establishing a remote sensing domain KG for the first time, which can provide support for knowledge modeling and knowledge services in the remote sensing field.First, this paper reviews the development history of general KGs. Second, it discusses the technologies of constructing remote sensing KGs. Compared with general KGs, remote sensing KGs are oriented to the field of remote sensing geosciences. They have significant disciplinary characteristics and spatiotemporal graph characteristics in terms of graph nodes, graph relationships and graph reasoning. Specific performances are as follows: (1) Images are an important part of remote sensing, which play an irreplaceable role and are ignored by general KGs. (2) Remote sensing knowledge is oriented to spatial entities. In addition to semantic relationships, the description of entity relationships also requires spatial and temporal relationships. (3) Traditional logical reasoning and natural language processing learning reasoning cannot effectively deal with image entities and spatial relationships. To solve the above problems, this paper draws on the construction scheme of the general KG and related domain KG and proposes the basic construction process of the remote sensing KG.Third, it introduces typical geoscience application cases driven by remote sensing KGs, which include three cases: (1) Marine oil spill monitoring. Marine oil spill KG is used for oil pollution identification, cause reasoning, and spill risk assessment, etc. (2) Land cover classification. Coupling remote sensing KG reasoning and deep learning for land cover classification. Numerous experiments have proven that KG can improve the classification results. (3) Evaluation of the carrying capacity of resources and the environment and suitability of land and space development. Ontology can not only express the knowledge system of evaluation in a standardized manner but also infer the evaluation results based on the constructed knowledge. Finally, it analyzes the application status and future research directions of remote sensing KGs. This paper points out four feasible and important research directions: (1) Exploring the theories and methods of creating multimodal remote sensing KGs; (2) Cooperative update and alignment fusion of remote sensing KGs; (3) Intelligent remote sensing image classification based on remote sensing KG representation learning; and (4) Scientific decision support analysis assisted by remote sensing KGs.Generally, the research of remote sensing KGs is conducive to better summarizing the conceptual knowledge of remote sensing, managing the new knowledge contained in remote sensing big data, and providing flexible and convenient remote sensing knowledge query and service capabilities to users in multiple fields, and it will help comprehensively improve the application capabilities of massive multisource remote sensing observation results and will play an important role in the study of global remote sensing land cover classification, climate change, international humanitarian assistance, and so on.

مفهوم

remote sensing knowledge graph;knowledge graph application;artificial intelligence;knowledge service;representation learning;domain knowledge modeling

References

  1. 1.
    Aksoy S. 2006. Modeling of remote sensing image content using attributed relational graphs//Proceedings of the Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Hong Kong, China: Springer: 475-483
  2. 2.
    Alirezaie M, Längkvist M, Sioutis M and Loutfi A. 2019. Semantic referee: a neural-symbolic framework for enhancing geospatial semantic segmentation. Semantic Web, 10(5): 863-880
  3. 3.
    Biederman I. 1987. Recognition-by-components: a theory of human image understanding. Psychological Review, 94(2): 115-147
  4. 4.
    Bordes A, Usunier N, Garcia-Durán A and Weson J. 2013. Translating embeddings for modeling multi-relational data//Proceedings of the 26th International Conference on Neural Information Processing Systems. Lake Tahoe Nevada: Curran Associates Inc.: 2787-2795
  5. 5.
    Bruna J, Zaremba W, Szlam A and LeCun Y. 2014. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203
  6. 6.
    Cao X, Chen X H, Zhang W W, Liao A P, Chen L J, Chen Z G and Chen J. 2016. Global cultivated land mapping at 30 m spatial resolution. Science China Earth Sciences, 59(12): 2275-2284
  7. 7.
    Chen J, Liu W Z, Wu H, Li Z L, Zhao Y and Zhang L. 2019. Basic issues and research agenda of geospatial knowledge service. Geomatics and Information Science of Wuhan University, 44(1): 38-47
  8. 47.
    [DOI ]
  9. 9.
    Chen J, Xiang L G and Gong J Y. 2013. Virtual globe-based integration and sharing service method of GeoSpatial information. Science China Earth Sciences, 56(10): 1780-1790
  10. 10.
    De Cao N, Aziz W and Titov I. 2019. Question answering by reasoning across documents with graph convolutional networks//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: ACL: 2306-2317
  11. 11.
    Devlin J, Chang M W, Lee K and Toutanova K. 2019. BERT: pre-training of deep bidirectional transformers for language understanding//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: ACL: 4171-4186
  12. 12.
    Ding M, Zhou C, Chen Q B, Yang H X and Tang J. 2019. Cognitive graph for multi-hop reading comprehension at scale//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: ACL
  13. 13.
    Dong J W, Kuang W H and Liu J Y. 2017. Continuous land cover change monitoring in the remote sensing big data era. Science China Earth Sciences, 60(12): 2223-2224
  14. 14.
    Ehrlinger L and Wöß W. 2016. Towards a definition of knowledge graphs//Proceedings of the Posters and Demos Track of the 12th International Conference on Semantic Systems. Leipzig, Germany: CEUR-WS.org
  15. 15.
    Fan J, Yu W Z, Wu W and Shen Y. 2017. Knowledge-guided fitting method for sparse time series remote sensing data. Journal of Remote Sensing, 21(5): 749-756
  16. 16.
    Färber M, Bartscherer F, Menne C and Rettinger A. 2018. Linked data quality of DBpedia, freebase, OpenCyc, wikidata, and YAGO. Semantic Web, 9(1): 77-129
  17. 17.
    Gruber T R. 1995. Toward principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies, 43(5/6): 907-928
  18. 18.
    Gu H Y, Li H T, Yan L, Han Y S, Yu F, Yang Y and Liu Z J. 2018. A geographic object-based image analysis methodology based on geo-ontology. Geomatics and Information Science of Wuhan University, 43(1): 31-36
  19. 19.
    Hamilton W L, Ying R and Leskovec J. 2017. Inductive representation learning on large graphs//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc.: 1025-1035
  20. 20.
    He L, Shao B, Xiao Y H, Li Y T, Liu T Y, Chen E H and Xia H H. 2018. Neurally-guided semantic navigation in knowledge graph. IEEE Transactions on Big Data, 8(3): 607-615
  21. 21.
    Horrocks I, Patel-Schneider P F and van Harmelen F. 2003. From SHIQ and RDF to OWL: the making of a web ontology language. Journal of Web Semantics, 1(1): 7-26
  22. 22.
    Hou Z W, Zhu Y Q, Gao X, Pan P, Luo K and Wang D X. 2015. Time-ontology and its application in geodata retrieval. Journal of Geo-Information Science, 17(4): 379-390
  23. 23.
    Hu T Y, Li X C, Gong P, Yu W C and Huang X C. 2020. Evaluating the effect of plain afforestation project and future spatial suitability in Beijing. Science China Earth Sciences, 63(10): 1587-1598
  24. 24.
    Huang J L. 2020. Research and Application of Text Classification Algorithm in Patent Field Based on Knowledge Graph. Changchun: Jilin University
  25. 25.
    Huang M F. 2016. Remote Sensing Detection Mechanism and Information Extraction Method of Water Petroleum Pollution. Beijing: Science Press
  26. 26.
    Jiang B C, Wan G, Xu J, Li F and Wen H Q. 2018. Geographic knowledge graph building extracted from multi-sourced heterogeneous data. Acta Geodaetica et Cartographica Sinica, 47(8): 1051-1061
  27. 27.
    Kalfoglou Y and Schorlemmer M. 2003. Ontology mapping: the state of the art. The knowledge Engineering Review, 18(1): 1-31
  28. 28.
    Karpatne A, Atluri G, Faghmous J H, Steinbach M, Banerjee A, Ganguly A, Shekhar S, Samatova N and Kumar V. 2017. Theory-guided data science: a new paradigm for scientific discovery from data. IEEE Transactions on Knowledge and Data Engineering, 29(10): 2318-2331
  29. 29.
    Kipf T N and Welling M. 2017. Semi-supervised classification with graph convolutional networks//Proceedings of the 5th International Conference on Learning Representations. Toulon: OpenReview.net
  30. 30.
    Krishna R, Zhu Y K, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li L J, Shamma D A, Bernstein M S and Li F F. 2017. Visual genome: connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1): 32-73
  31. 31.
    Kroetsch M and Weikum G. 2016. Special issue on knowledge graphs. Journal of Web Semantics, 37(38): 53-54
  32. 32.
    Lehmann J, Isele R, Jakob M, Jentzsch A, Kontokostas D, Mendes P N, Hellmann S, Morsey M, van Kleef P, Auer S and Bizer C. 2015. DBpedia-A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web, 6(2): 167-195
  33. 33.
    Li D R. 2016. Towards geo-spatial information science in big data era. Acta Geodaetica et Cartographica Sinica, 45(4): 379-384
  34. 34.
    Li Y S, Kong D Y, Zhang Y J, Ji Z and Xiao R. 2020. Zero-shot remote sensing image scene classification based on robust cross-domain mapping and gradual refinement of semantic space. Acta Geodaetica et Cartographica Sinica, 49(12): 1564-1574
  35. 35.
    Li Y S, Ouyang S and Zhang Y J. 2022. Combining deep learning and ontology reasoning for remote sensing image semantic segmentation . Knowledge-Based Systems. 243, 108469.
  36. 36.
    Li Y S, Zhu Z H, Yu J G and Zhang Y J. 2021. Learning deep cross-modal embedding networks for zero-shot remote sensing image scene classification. IEEE Transactions on Geoscience and Remote Sensing, 59(12): 10590-10603
  37. 37.
    Lin B Y, Chen X Y, Chen J M and Ren X. 2019. KagNet: knowledge-aware graph networks for commonsense reasoning//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong, China: ACL: 2829-2839
  38. 38.
    Lin H and You L. 2015. A tentative study on knowledge engineering for virtual geographic environments. Journal of Geo-Information Science, 17(12): 1423-1430
  39. 39.
    Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P and Zitnick C L. 2014. Microsoft COCO: common objects in context//Proceedings of the 13th European Conference on Computer Vision. Zurich: Springer: 740-755
  40. 40.
    Liu J N, Liu H Y, Chen X H, Guo X, Guo W Y, Zhu X M, Zhao Q B and Li J. 2022. Terrorism event model by knowledge graph. Geomatics and Information Science of Wuhan University, 47(2): 313-322
  41. 41.
    Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y and Berg A C. 2016. SSD: single shot MultiBox detector//Proceedings of the 14th European Conference on Computer Vision. Amsterdam: Springer: 21-37
  42. 42.
    Lu F, Yu L and Qiu P Y. 2017. On geographic knowledge graph. Journal of Geo-Information Science, 19(6): 723-734
  43. 43.
    Lü G N. 2011. Geographic analysis-oriented Virtual Geographic Environment: framework, structure and functions. Science China Earth Sciences, 54(5): 733-743
  44. 44.
    Mikolov T, Sutskever I, Chen K, Corrado G and Dean J. 2013. Distributed representations of words and phrases and their compositionality//Proceedings of the 26th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc.: 3111-3119
  45. 45.
    Paulheim H. 2017. Knowledge graph refinement: a survey of approaches and evaluation methods. Semantic Web, 8(3): 489-508
  46. 46.
    Pujara J, Miao H, Getoor L and Cohen W. 2013. Knowledge graph identification//Proceedings of the 12th International Semantic Web Conference. Sydney: Springer: 542-557
  47. 47.
    Redmon J, Divvala S, Girshick R and Farhadi A. 2016. You only look once: unified, real-time object detection//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE: 779-788
  48. 48.
    Reichstein M, Camps-Valls G, Stevens B, Jung M, Denzler J, Carvalhais N and Prabhat. 2019. Deep learning and process understanding for data-driven earth system science. Nature, 566(7743): 195-204
  49. 49.
    Ren S Q, He K M, Girshick R and Sun J. 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 1137-1149
  50. 50.
    Ronneberger O, Fischer P and Brox T. 2015. U-Net: convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer: 234-241
  51. 51.
    Schuster M and Paliwal K K. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11): 2673-2681
  52. 52.
    Singhal A. 2012. Introducing the Knowledge Graph: Things, Not Strings. Official Google Blog
  53. 53.
    Spaulding M L. 2017. State of the art review and future directions in oil spill modeling. Marine Pollution Bulletin, 115(1/2): 7-19
  54. 54.
    Suchanek F M, Kasneci G and Weikum G. 2007. Yago: a core of semantic knowledge//Proceedings of the 16th international conference on World Wide Web. Banff: ACM: 697-706
  55. 55.
    Vashishth S, Jain P and Talukdar P. 2018. CESI: canonicalizing open knowledge bases using embeddings and side information//Proceedings of the 2018 World Wide Web Conference. Lyon: ACM
  56. 56.
    Wang D X, Zhu Y Q, Pan P, Luo K and Hou Z W. 2016. Construction of geodata spatial ontology and its application in data retrieval. Journal of Geo-Information Science, 18(4): 443-452
  57. 57.
    Wang M, Qi G, Wang H F and Zheng Q. 2020. Richpedia: a comprehensive multi-modal knowledge graph//Joint International Semantic Technology Conference. Cham：Springer: 130-145
  58. 58.
    Wang Q, Mao Z D, Wang B and Guo L. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on knowledge & Data Engineering, 29(12): 2724-2743
  59. 59.
    Xie R, Luo Z W, Wang Y C and Chen W. 2017. Key techniques for establishing domain specific large scale knowledge graph of remote sensing satellite. Radio Engineering, 47(4): 1-6
  60. 60.
    Xu B, Xu Y, Liang J Q, Xie C H, Liang B, Cui W Y and Xiao Y H. 2017. CN-DBpedia: a never-ending Chinese knowledge extraction system//Proceedings of the 30th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Arras, France: Springer: 428-438
  61. 61.
    Yang Y J, Xu B, Hu J W, Tong M H, Zhang P and Zheng L. 2018. Accurate and efficient method for constructing domain knowledge graph. Journal of Software, 29(10): 2931-2947
  62. 62.
    Zaremba W, Sutskever I and Vinyals O. 2014. Recurrent neural network regularization. arXiv preprint arXiv: 1409.2329
  63. 63.
    Zhang F J, Liu X, Tang J, Dong Y X, Yao P R, Zhang J, Gu X T, Wang Y, Shao B, Li R and Wang K S. 2019a. OAG: toward linking large-scale heterogeneous entity graphs//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Anchorage: ACM
  64. 64.
    Zhang H M, Wang M C, Chen X Y, Wang F Y, Yang G D and Gao S. 2022. Remote sensing change detection based on deep belief networks optimized by domain knowledge. Geomatics and Information Science of Wuhan University, 47(5): 762-768, 788
  65. 65.
    Zhang N Y, Deng S M, Sun Z L, Wang G Y, Chen X, Zhang W and Chen H J. 2019b. Long-tail relation extraction via knowledge graph embeddings and graph convolution networks//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Minneapolis: ACL: 3016-3025
  66. 66.
    Zhang X Y, Zhang C J, Wu M G and Lü G N. 2020. Spatiotemporal features based geographical knowledge graph construction. SCIENTIA SINICA: Informationis, 50(7): 1019-1032
  67. 67.
    Zhang Y H, Qi P and Manning C D. 2018. Graph convolution over pruned dependency trees improves relation extraction//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: ACL: 2205-2215
  68. 68.
    Zhao J, Liu K, He S Z and Chen Y B. 2018. Knowledeg Graph. Beijing: Higher Education Press: 20-29
  69. 69.
    Zhou X R, Shao Z F and Liu J. 2012. Geographic ontology driven hierarchical semantic of remote sensing image//Proceedings of 2012 International Conference on Computer Vision in Remote Sensing. Xiamen, China: IEEE

قراءة النص الكامل

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website