Précédent|Prochain

Street view imagery： Methods and applications based on artificial intelligence

  • role： First author第一作者
  • Affiliation：

    Institute of Remote Sensing and Geographical Information Systems, School of Earth and Space Sciences, Peking University, Beijing 100871, China

  • Email：zhangfan@link.cuhk.edu.hk
  • Introduction：张帆，1990年生,男,助理研究员,研究方向为地理信息科学与地理大数据理论方法。E-mail:zhangfan@link.cuhk.edu.hk
ZHANG Fan,  
  • role： Corresponding author通信作者
  • Affiliation：

    Institute of Remote Sensing and Geographical Information Systems, School of Earth and Space Sciences, Peking University, Beijing 100871, China

  • Email：liuyu@urban.pku.edu.cn
  • Introduction：刘瑜，1971年生,男,教授, 研究方向为地理信息科学与地理大数据理论方法。E-mail: liuyu@urban.pku.edu.cn
LIU Yu*

résumé

Street view imagery is a promising and growing big geo-data that provides current and historical images in more than 200 countries for urban physical environment representation and audit. Such data not only describes the visual details of the urban physical environment but also contains information about urban functions, socioeconomics, and human dynamics. Street view imagery has the potential to complement new and traditional data, such as remote sensing imagery and social sensing data.However, traditional digital image processing techniques for street view imagery handling are limited. Extracting rich semantic information from street view imagery efficiently has always been a challenging issue. Until recently, the development of artificial intelligence has led to numerous breakthroughs in image processing and machine learning. Indeed, the last few years have witnessed the fast development of deep learning and computer vision techniques, which facilitate the understanding of scene semantics from street view imagery and the quantitative representation of the urban physical and built environments. Many new applications, novel methods, and thoughts regarding street view imagery have emerged, covering research fields, such as geography, urban planning, urban design, urban economics, public health, environmental psychology, and energy. This trend has provided new perspectives for big geo-data-driven urban environment analysis, human–land relationship study, and spatial data mining and knowledge discovery.To summarize this research trend, this paper reviews the recent works on urban physical environment analysis using street view imagery. The key supporting techniques for street view imagery analytics are discussed in terms of two dimensions: deep learning and computer vision. Deep learning has been applied recently to various computer vision tasks, such as image classification, image segmentation, and object detection. The success of deep learning techniques is attributed to their ability to learn rich high-level image representations as opposed to the hand-designed low-level features used in other image understanding methods.Additionally, this paper summarizes the street view imagery applications in three aspects: place representation, sense of place, and place semantics reasoning. “Place representation” includes works that extract visual elements that constitute the urban physical environment; “sense of place” refers to works that use street view imagery to understand how people respond to their surrounding environments regarding perceptions and emotions; and “place semantics reasoning” refers to studies that attempt to infer and estimate invisible factors, socio-economics, demographics, and human dynamics from street view imagery.More importantly, the issues of this research field, such as the spatio-temporal uniformity of street view image data, and the lack of solid workflow of data analytics, are highlighted.Finally, the development prospects of street view imagery are discussed. Crowdsourcing platforms and the field of the autonomous vehicle will increase the number of street view image sources. More issues, including how physical environments are involved and whether a universal law exists regarding the distribution of physical elements in space, are expected to be explored in future works.

mots-clés

street view imagery;place semantics;urban physical environment;deep learning;computer vision

References

  1. 1.
    Alvarez Leon L F and Quinn S. 2019. The value of crowdsourced street-level imagery: examining the shifting property regimes of OpenStreetCam and Mapillary. GeoJournal, 84(2): 395-414
  2. 2.
    Arietta S M, Efros A A, Ramamoorthi R and Agrawala M. 2014. City forensics: using visual elements to predict non-visual city attributes. IEEE Transactions on Visualization and Computer Graphics, 20(12): 2624-2633
  3. 3.
    Cao R, Zhu J S, Tu W, Li Q Q, Cao J Z, Liu B Z, Zhang Q and Qiu G P. 2018. Integrating aerial and street view images for urban land use classification. Remote Sensing, 10(10): 1553
  4. 4.
    Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S and Schiele B. 2016. The cityscapes dataset for semantic urban scene understanding//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE: 3213-3223
  5. 5.
    Dalal N and Triggs B. 2005. Histograms of oriented gradients for human detection//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, CA, USA: IEEE: 886-893
  6. 6.
    Doersch C, Singh S, Gupta A, Sivic J and Efros A A. 2012. What makes Paris look like Paris? ACM Transactions on Graphics, 31(4): 101
  7. 7.
    Dubey A, Naik N, Parikh D, Raskar R and Hidalgo C A. 2016. Deep learning the city: quantifying urban perception at a global scale//Proceedings of the 14th European Conference on Computer Vision – ECCV 2016. Amsterdam, The Netherlands: Springer: 196-212
  8. 8.
    Everingham M, Van Gool L, Williams C K I, Winn J and Zisserman A. 2010. The PASCAL Visual Object Classes (VOC) challenge. International Journal of Computer Vision, 88(2): 303-338
  9. 9.
    Fu X, Jia T X, Zhang X Q, Li S L and Zhang Y L. 2019. Do street-level scene perceptions affect housing prices in Chinese megacities? An analysis using open access datasets and deep learning. PLoS One, 14(5): e0217505
  10. 10.
    Gebru T, Krause J, Wang Y L, Chen D Y, Deng J, Aiden E L and Li F F. 2017. Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. Proceedings of the National Academy of Sciences of the United States of America, 114(50): 13108-13113
  11. 11.
    Gong F Y, Zeng Z C, Zhang F, Li X J, Ng E and Norford L K. 2018. Mapping sky, tree, and building view factors of street canyons in a high-density urban environment. Building and Environment, 134: 155-167
  12. 12.
    Gong P. 2019. Towards more extensive and deeper application of remote sensing. Journal of Remote Sensing, 23(4): 567-569
  13. 13.
    Hao X H and Long Y. 2017. Street greenery: a new indicator for evaluating walkability. Shanghai Urban Planning Review, (1): 32-36, 49
  14. 14.
    He K M, Gkioxari G, Dollár P and Girshick R. 2017a. Mask R-CNN//Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE: 2980-2988
  15. 15.
    He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE: 770-778
  16. 16.
    He L, Páez A and Liu D S. 2017b. Built environment and violent crime: an environmental audit approach using Google Street View. Computers, Environment and Urban Systems, 66: 83-95
  17. 17.
    Helbich M, Yao Y, Liu Y, Zhang J B, Liu P H and Wang R Y. 2019. Using deep learning to examine street view green and blue spaces and their associations with geriatric depression in Beijing, China. Environment International, 126: 107-117
  18. 18.
    Hinton G E and Salakhutdinov R R. 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786): 504-507
  19. 19.
    Hu C B, Zhang F, Gong F Y, Ratti C and Li X. 2020. Classification and mapping of urban canyon geometry using Google Street View images and deep multitask learning. Building and Environment, 167: 106424
  20. 20.
    Huang G, Liu Z, Van Der Maaten L and Weinberger K Q. 2017. Densely connected convolutional networks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE: 2261-2269
  21. 21.
    Huang J X , Obracht-Prondzynska H, Kamrowska-Zaluska D, Sun Y M and Li L S .2021.The image of the City on social media: A comparative study using “Big Data” and “Small Data” methods in the Tri-City Region in Poland. Landscape and Urban Planning 206:103977.
  22. 22.
    Ibrahim M R, Haworth J and Cheng T. 2019. URBAN-i: from urban scenes to mapping slums, transport modes, and pedestrians in cities using deep learning and computer vision. Environment and Planning B: Urban Analytics and City Science
  23. 23.
    Ilic L, Sawada M and Zarzelli A. 2019. Deep mapping gentrification in a large Canadian city using deep learning and Google Street View. PLoS One, 14(3): e0212814
  24. 24.
    Kang H Y, Wang J, Wang Y, Angsuesser S and Fei T. 2017. Mapping the sensitivity of the public emotion to the movement of stock market value: a case study of Manhattan//Proceedings of the International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Wuhan: [s.n.]
  25. 25.
    Kang Y H, Zhang F, Gao S, Lin H and Liu Y. 2020. A review of urban physical environment sensing using streat view imagery in public health studies. Annals of GIS, 26(3): 261-275
  26. 26.
    Kang Y H, Jia Q Y, Gao S, Zeng X H, Wang Y Y, Angsuesser S, Liu Y, Ye X Y and Fei T. 2019. Extracting human emotions at different places based on facial expressions and spatial clustering analysis. Transactions in GIS, 23(3): 450-480
  27. 27.
    Khosla A, An B, Lim J J and Torralba A. 2014. Looking beyond the visible scene//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA: IEEE: 3710-3717
  28. 28.
    Krizhevsky A, Sutskever I and Hinton G E. 2012. ImageNet classification with deep convolutional neural networks//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada: ACM: 1097-1105
  29. 29.
    Law S, Paige B and Russell C. 2018. Take a look around: using street view and satellite images to estimate house prices. arXiv:1807.07155
  30. 30.
    Law S, Seresinhe C I, Shen Y and Gutierrez-Roig M. 2020. Street-Frontage-Net: urban image classification using deep convolutional neural networks. International Journal of Geographical Information Science, 34(4): 681-707
  31. 31.
    LeCun Y, Bengio Y and Hinton G. 2015. Deep learning. Nature, 521(7553): 436-444
  32. 32.
    Li X J, Cai B Y, Qiu W S, Zhao J H and Ratti C. 2019. A novel method for predicting and mapping the occurrence of sun glare using Google Street View. Transportation Research Part C: Emerging Technologies, 106: 132-144
  33. 33.
    Li X J and Ratti C. 2019. Mapping the spatio-temporal distribution of solar radiation within street canyons of Boston using Google Street View panoramas and building height model. Landscape and Urban Planning, 191: 103387
  34. 34.
    Li X J, Ratti C and Seiferling I. 2018. Quantifying the shade provision of street trees in urban landscape: a case study in Boston, USA, using Google Street View. Landscape and Urban Planning, 169: 81-91
  35. 35.
    Li X J, Zhang C R and Li W D. 2017. Building block level urban land-use information retrieval based on Google Street View images. GIScience and Remote Sensing, 54(6): 819-835
  36. 36.
    Li X J, Zhang C R, Li W D, Ricard R, Meng Q Y and Zhang W X. 2015. Assessing street-level urban greenery using Google Street View and a modified green view index. Urban Forestry and Urban Greening, 14(3): 675-685
  37. 37.
    Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P and Zitnick C L. 2014. Microsoft COCO: common objects in context//Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer: 740-755
  38. 38.
    Liu L, Silva E A, Wu C Y and Wang H. 2017. A machine learning-based method for the large-scale evaluation of the qualities of the urban environment. Computers, Environment and Urban Systems, 65: 113-125
  39. 39.
    Liu L, Zhou B L, Zhao J H and Ryan B D. 2016a. C-IMAGE: city cognitive mapping through geo-tagged photos. GeoJournal, 81(6): 817-861
  40. 40.
    Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y and Berg A C. 2016b. SSD: single shot MultiBox detector//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer: 21-37
  41. 41.
    Liu Y, Liu X, Gao S, Gong L, Kang C G, Zhi Y, Chi G H and Shi L. 2015. Social sensing: a new approach to understanding our socioeconomic environments. Annals of the Association of American Geographers, 105(3): 512-530
  42. 42.
    Liu Z Y, Yang A Q, Gao M Y, Jiang H, Kang Y H, Zhang F and Fei T. 2019. Towards feasibility of photovoltaic road for urban traffic-solar energy estimation using street view image. Journal of Cleaner Production, 228: 303-318
  43. 43.
    Long Y and Liu L. 2017. How green are the streets? An analysis for central areas of Chinese cities using Tencent Street View. PLoS One, 12(2): e0171110
  44. 44.
    Long Y and Ye Y. 2019. Measuring human-scale urban form and its performance. Landscape and Urban Planning, 191: 103612
  45. 45.
    Long Y and Zhou Y. 2017. Pictorial urbanism: a new approach for human scale urban morphology study. Planners, 33(2): 54-60
  46. 46.
    Lu Y, Yang Y Y, Sun G B and Gou Z H. 2019. Associations between overhead-view and eye-level urban greenness and cycling behaviors. Cities, 88: 10-18
  47. 47.
    Ma R X, Wang W, Zhang F, Shim K and Ratti C. 2019. Typeface reveals spatial economical patterns. Scientific Reports, 9(1): 15946
  48. 48.
    Ma X Y, Ma C Y, Wu C, Xi Y L, Yang R F, Peng N Y Z, Zhang C and Ren F.2021. Measuring human perceptions of streetscapes to better inform urban renewal: A perspective of scene semantic parsing. Cities 110:103086.
  49. 49.
    Naik N, Kominers S D, Raskar R, Glaeser E L and Hidalgo C A. 2017. Computer vision uncovers predictors of physical urban change. Proceedings of the National Academy of Sciences of the United States of America, 114(29): 7571-7576
  50. 50.
    Naik N, Philipoom J, Raskar R and Hidalgo C. 2014. Streetscore -- predicting the perceived safety of one million streetscapes//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Columbus, OH, USA: IEEE: 793-799
  51. 51.
    Nair V and Hinton G E. 2010. Rectified linear units improve restricted Boltzmann machines//Proceedings of the 27th International Conference on International Conference on Machine Learning. Madison, WI, United States: ACM: 807-814
  52. 52.
    Oliva A and Torralba A. 2001. Modeling the shape of the scene: a holistic representation of the spatial envelope. International Journal of Computer Vision, 42(3): 145-175
  53. 53.
    Patterson G and Hays J. 2012. Sun attribute database: discovering, annotating, and recognizing scene attributes//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, USA: IEEE: 2751-2758
  54. 54.
    Pred A. 1984. Place as historically contingent process: structuration and the time-geography of becoming places. Annals of the Association of American Geographers, 74(2): 279-297
  55. 55.
    Quercia D, O’Hare N K and Cramer H. 2014. Aesthetic capital: what makes london look beautiful, quiet, and happy?//Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing. Maryland, Baltimore, USA: ACM: 945-955
  56. 56.
    Ren S Q, He K M, Girshick R and Sun J. 2015. Faster R-CNN: towards real-time object detection with region proposal networks//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge, MA, USA: ACM: 91-99
  57. 57.
    Roda C, Charreire H, Feuillet T, Mackenbach J D, Compernolle S, Glonti K, Ben Rebah M, Bárdos H, Rutter H, McKee M, De Bourdeaudhuij I, Brug J, Lakerveld J B J and Oppert J M. 2016. Mismatch between perceived and objectively measured environmental obesogenic features in European neighbourhoods. Obesity Reviews, 17(S1): 31-41
  58. 58.
    Rumelhart D E, Hinton G E and Williams R J. 1986. Learning representations by back-propagating errors. Nature, 323(6088): 533-536
  59. 59.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z H, Karpathy A, Khosla A, Bernstein M, Berg A C and Li F F. 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3): 211-252
  60. 60.
    Salesses P, Schechtner K and Hidalgo C A. 2013. The collaborative image of the city: mapping the inequality of urban perception. PLoS One, 8(7): e68400
  61. 61.
    Seiferling I, Naik N, Ratti C and Proulx R. 2017. Green streets - Quantifying and mapping urban trees with street-level imagery and computer vision. Landscape and Urban Planning, 165: 93-101
  62. 62.
    Seresinhe C I, Preis T and Moat H S. 2017. Using deep learning to quantify the beauty of outdoor places. Royal Society Open Science, 4(7): 170170
  63. 63.
    Simonyan K and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
  64. 64.
    Srivastava S, Muñoz J E V, Lobry S and Tuia D. 2020. Fine-grained landuse characterization using ground-based pictures: a deep learning solution based on globally available data. International Journal of Geographical Information Science, 34(6): 1117-1136
  65. 65.
    Su S L, Zhou H, Xu M Y, Ru H, Wang W and Weng M. 2019. Auditing street walkability and associated social inequalities for planning implications. Journal of Transport Geography, 74: 62-76
  66. 66.
    Suel E, Polak J W, Bennett J E and Ezzati M. 2019. Measuring social, environmental and health inequalities using deep learning and street imagery. Scientific Reports, 9(1): 6229
  67. 67.
    Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V and Rabinovich A. 2015. Going deeper with convolutions//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA: IEEE: 1-9
  68. 68.
    Tang J X and Long Y. 2019. Measuring visual quality of street space and its temporal variation: methodology and its application in the Hutong area in Beijing. Landscape and Urban Planning, 191: 103436
  69. 69.
    Wang R, Chen H S, Liu Y, Lu Y and Yao Y. 2019a. Neighborhood social reciprocity and mental health among older adults in China: the mediating effects of physical activity, social interaction, and volunteering. BMC Public Health, 19(1): 1036
  70. 70.
    Wang R Y, Helbich M, Yao Y, Zhang J B, Liu P H, Yuan Y and Liu Y. 2019b. Urban greenery and mental wellbeing in adults: cross-sectional mediation analyses on multiple pathways across different greenery measures. Environmental Research, 176: 108535
  71. 71.
    Wang R Y, Liu Y, Lu Y, Zhang J B, Liu P H, Yao Y and Grekousis G. 2019c. Perceptions of built environment and health outcomes for older Chinese in Beijing: a big data approach with street view images and deep learning technique. Computers, Environment and Urban Systems, 78: 101386
  72. 72.
    Wang W S, Yang S, He Z Y, Wang M J, Zhang J L and Zhang W S. 2018. Urban perception of commercial activeness from satellite images and streetscapes//Proceedings of Companion of the Web Conference 2018. ACM Press, Lyon, France: ACM: 647-654
  73. 73.
    Wang Z T, Liang Q H, Duarte F, Zhang F, Charron L, Johnsen L, Cai B and Ratti C. 2019d. Quantifying legibility of indoor spaces using Deep Convolutional Neural Networks: case studies in train stations. Building and Environment, 160: 106099
  74. 74.
    Wilson J Q and Kelling G L. 1982. Broken windows. The Atlantic Monthly, 249: 29-38
  75. 75.
    Ye Y, Zeng W, Shen Q M, Zhang X H and Lu Y. 2019. The visual quality of streets: a human-centred continuous measurement based on machine learning algorithms and street view images. Environment and Planning B: Urban Analytics and City Science, 46(8): 1439-1457
  76. 76.
    Yin L and Wang Z X. 2016. Measuring visual enclosure for street walkability: using machine learning algorithms and Google Street View imagery. Applied Geography, 76: 147-153
  77. 77.
    Yoshimura Y, Cai B, Wang Z T and Ratti C. 2019. Deep learning architect: classification for architectural design through the eye of artificial intelligence//Geertman S, Zhan Q M, Allan A and Pettit C, eds. Computational Urban Planning and Management for Smart Cities. Cham: Springer: 249-265
  78. 78.
    Zhang F, Duarte F, Ma R X, Milioris D, Lin H and Ratti C. 2016. Indoor space recognition using deep convolutional neural network: a case study at MIT campus. arXiv:1610.02414
  79. 79.
    Zhang F, Fan Z Y, Kang Y H, Hu Y J and Carlo Ratti.202. “Perception bias”: Deciphering a mismatch between urban crime and perception of safety. Landscape and Urban Planning 207:104003.
  80. 80.
    Zhang F, Hu M Y, Che W T, Lin H and Fang C Y. 2018a. Framework for virtual cognitive experiment in virtual geographic environments. ISPRS International Journal of Geo-Information, 7(1): 36
  81. 81.
    Zhang F, Wu L, Zhu D and Liu Y. 2019a. Social sensing from street-level imagery: a case study in learning spatio-temporal urban mobility patterns. ISPRS Journal of Photogrammetry and Remote Sensing, 153: 48-58
  82. 82.
    Zhang F, Zhang D, Liu Y and Lin H. 2018b. Representing place locales using scene elements. Computers, Environment and Urban Systems, 71: 153-164
  83. 83.
    Zhang F, Zhou B L, Liu L, Liu Y, Fung H H, Lin H and Ratti C. 2018c. Measuring human perceptions of a large-scale urban region using machine learning. Landscape and Urban Planning, 180: 148-160
  84. 84.
    Zhang F, Zhou B L, Ratti C and Liu Y. 2019b. Discovering place-informative scenes and objects using social media photos. Royal Society Open Science, 6(3): 181375
  85. 85.
    Zhang F, Zu J Y , Hu M Y, Zhu D, Kang Y H, Gao S, Zhang Y and Huang Z .2020. Uncovering inconspicuous places using social media check-ins and street view images. Computers, Environment and Urban Systems 81:101478.
  86. 86.
    Zhang L Y, Pei T, Chen Y J, Song C and Liu X Q. 2019. A review of urban environmental assessment based on street view images. Journal of Geo-information Science, 21(1): 46-58
  87. 87.
    Zhao H S, Shi J P, Qi X J, Wang X G and Jia J Y. 2017. Pyramid scene parsing network//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE: 6230-6239
  88. 88.
    Zhou B L, Lapedriza A, Khosla A, Oliva A and Torralba A. 2018. Places: a 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 1452-1464
  89. 89.
    Zhou B L, Lapedriza A, Xiao J X, Torralba A and Oliva A. 2014. Learning deep features for scene recognition using places database//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada: ACM: 487-495
  90. 90.
    Zhou B L, Zhao H, Puig X, Xiao T T, Fidler S, Barriuso A and Torralba A. 2019. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision, 127(3): 302-321
  91. 91.
    Zhu D, Zhang F, Wang S Y, Wang Y L, Cheng X M, Huang Z and Liu Y. 2020. Understanding place characteristics in geographic contexts through graph convolutional neural networks. Annals of the American Association of Geographers, 110(2): 408-420

Lire l'article complet

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website