Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset
Authors: 
Xinlei Liu, Kevin Wu, Minchu Kulkarni, Michael Saugstad, Peyton Anton Rapo, Jeremy Freiburger, Maryam Hosseini, 
Chu Li, 
Jon E. FroehlichAuthors Info & Claims
Xinlei Liu, Kevin Wu, Minchu Kulkarni, Michael Saugstad, Peyton Anton Rapo, Jeremy Freiburger, Maryam Hosseini, Chu Li, Jon E. FroehlichAuthors Info & ClaimsArticle No.: 103, Pages 1 - 12
Abstract
We examine the feasibility of using deep learning to infer 33 classes of sidewalk accessibility conditions in pre-cropped streetscape images, including bumpy, brick/cobblestone, cracks, height difference (uplifts), narrow, uneven/slanted, pole, and sign. We present two experiments: first, a comparison between two state-of-the-art computer vision models, Meta’s DINOv2 and OpenAI’s CLIP-ViT, on a cleaned dataset of ∼ 24k images; second, an examination of a larger but noisier crowdsourced dataset (∼ 87k images) on the best performing model from Experiment 1. Though preliminary, Experiment 1 shows that certain sidewalk conditions can be identified with high precision and recall, such as missing tactile warnings on curb ramps and grass grown on sidewalks, while Experiment 2 demonstrates that larger but noisier training data can have a detrimental effect on performance. We contribute an open dataset and classification benchmarks to advance this important area.
References
[1]
Marc A. Adams, Christine B. Phillips, Akshar Patel, and Ariane Middel. 2022. Training Computers to See the Built Environment Related to Physical Activity: Detection of Microscale Walkability Features Using Computer Vision. International Journal of Environmental Research and Public Health 19, 8 (2022). https://doi.org/10.3390/ijerph19084548
[2]
Breck, Polyzotis, Roy, Whang, and Zinkevich. 2019. Data Validation for Machine Learning. In Proceedings of Machine Learning and Systems. https://arxiv.org/pdf/2203.02155
[3]
Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Noack, Hendrik Patzlaff, Felix Naumann, and Hazar Harmouch. 2022. The Effects of Data Quality on Machine Learning Performance. arxiv:2207.14529
[4]
Haihua Chen, Jiangping Chen, and Junhua Ding. 2021. Data Evaluation and Enhancement for Quality Improvement of Machine Learning. IEEE Transactions on Reliability 70, 2 (2021), 831–847. https://doi.org/10.1109/TR.2021.3070863
[5]
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2022. Reproducible scaling laws for contrastive language-image learning. arxiv:2212.07143
[6]
Pierre Dognin, Igor Melnyk, Youssef Mroueh, Inkit Padhi, Mattia Rigotti, Jarret Ross, Yair Schiff, Richard A. Young, and Brian Belgodere. 2022. Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge. Journal of Artificial Intelligence Research 73 (Jan. 2022), 437–459. https://doi.org/10.1613/jair.1.13113
[7]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arxiv:2010.11929
[8]
Michael Duan, Shosuke Kiami, Logan Milandin, Johnson Kuang, Michael Saugstad, Maryam Hosseini, and Jon E. Froehlich. 2022. Scaling Crowd+AI Sidewalk Accessibility Assessments: Initial Experiments Examining Label Quality and Cross-city Training on Performance. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 82, 5 pages. https://doi.org/10.1145/3517428.3550381
[9]
Jon E. Froehlich, Anke M. Brock, Anat Caspi, João Guerreiro, Kotaro Hara, Reuben Kirkham, Johannes Schöning, and Benjamin Tannert. 2019. Grand challenges in accessible maps. Interactions 26, 2 (feb 2019), 78–81. https://doi.org/10.1145/3301657
[10]
Jon E. Froehlich, Yochai Eisenberg, Maryam Hosseini, Fabio Miranda, Marc Adams, Anat Caspi, Holger Dieterich, Heather Feldner, Aldo Gonzalez, Claudina De Gyves, Joy Hammel, Reuben Kirkham, Melanie Kneisel, Delphine LabbÉ, Steve J. Mooney, Victor Pineda, ClÁUdia PinhÃO, Ana RodrÍGuez, Manaswi Saha, Michael Saugstad, Judy Shanley, Ather Sharif, Qing Shen, Claudio Silva, Maarten Sukel, Eric K. Tokuda, Sebastian Felix Zappe, and Anna Zivarts. 2022. The Future of Urban Accessibility for People with Disabilities: Data Collection, Analytics, Policy, and Tools. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 102, 8 pages. https://doi.org/10.1145/3517428.3550402
[11]
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions From Blind People. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[12]
Kotaro Hara, Jin Sun, Robert Moore, David Jacobs, and Jon Froehlich. 2014. Tohme: detecting curb ramps in google street view using crowdsourcing, computer vision, and machine learning. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 189–204. https://doi.org/10.1145/2642918.2647403
[13]
Maryam Hosseini, Fabio Miranda, Jianzhe Lin, and Claudio T. Silva. 2022. CitySurfaces: City-scale semantic segmentation of sidewalk materials. Sustainable Cities and Society 79 (2022), 103630. https://doi.org/10.1016/j.scs.2021.103630
[14]
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023. Ultralytics YOLO. https://github.com/ultralytics/ultralytics
[15]
Bon Woo Koo, Subhrajit Guhathakurta, Nisha Botchwey, and Aaron Hipp. 2023. Can good microscale pedestrian streetscapes enhance the benefits of macroscale accessible urban form? An automated audit approach using Google street view images. Landscape and Urban Planning 237 (2023), 104816. https://doi.org/10.1016/j.landurbplan.2023.104816
[16]
Chu Li, Zhihan Zhang, Michael Saugstad, Esteban Safranchik, Chaitanyashareef Kulkarni, Xiaoyu Huang, Shwetak Patel, Vikram Iyer, Tim Althoff, and Jon E. Froehlich. 2024. LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 643, 21 pages. https://doi.org/10.1145/3613904.3642089
[17]
Christopher D Manning. 2008. Introduction to information retrieval. Syngress Publishing,.
[18]
Daniela Massiceti, Samreen Anjum, and Danna Gurari. 2022. VizWiz grand challenge workshop at CVPR 2022. SIGACCESS Access. Comput.133, Article 1 (aug 2022), 1 pages. https://doi.org/10.1145/3560232.3560233
[19]
U.S. Department of Transportation. 2024. Safe Streets and Roads for All Grant Program. https://www.transportation.gov/grants/SS4A Accessed June 20, 2024.
[20]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. 2024. DINOv2: Learning Robust Visual Features without Supervision. arxiv:2304.07193
[21]
F. Prior, J. Almeida, P. Kathiravelu, T. Kurc, K. Smith, T.J. Fitzgerald, and J. Saltz. 2020. Open access image repositories: high-quality data to enable machine learning research. Clinical Radiology 75, 1 (2020), 7–12. https://doi.org/10.1016/j.crad.2019.04.002
[22]
Manaswi Saha, Michael Saugstad, Hanuma Teja Maddali, Aileen Zeng, Ryan Holland, Steven Bower, Aditya Dash, Sage Chen, Anthony Li, Kotaro Hara, and Jon Froehlich. 2019. Project Sidewalk: A Web-based Crowdsourcing Tool for Collecting Sidewalk Accessibility Data At Scale. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300292
[23]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. LAION-5B: An open large-scale dataset for training next generation image-text models. arxiv:2210.08402
[24]
Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information processing & management 45, 4 (2009), 427–437.
[25]
United Nations Human Settlements Programme (UN-Habitat). 2016. The New Urban Agenda. https://habitat3.org/the-new-urban-agenda/ Accessed June 20, 2024.
[26]
Galen Weld, Esther Jang, Anthony Li, Aileen Zeng, Kurtis Heimerl, and Jon E. Froehlich. 2019. Deep Learning for Automatically Detecting Sidewalk Accessibility Problems Using Streetscape Imagery. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 196–209. https://doi.org/10.1145/3308561.3353798
Index Terms
- Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset
Recommendations
The Future of Urban Accessibility: The Role of AI
ASSETS '24: Proceedings of the 26th International ACM SIGACCESS Conference on Computers and AccessibilityWe have entered a new era of computing—one where AI permeates every aspect of society from education to healthcare. In this workshop, we examine the emerging role of AI in the design of equitable and accessible cities, transportation systems, and ...
Deep Learning for Automatically Detecting Sidewalk Accessibility Problems Using Streetscape Imagery
ASSETS '19: Proceedings of the 21st International ACM SIGACCESS Conference on Computers and AccessibilityRecent work has applied machine learning methods to automatically find and/or assess pedestrian infrastructure in online map imagery (e.g., satellite photos, streetscape panoramas). While promising, these methods have been limited by two interrelated ...
A dataset for the recognition of obstacles on blind sidewalk
AbstractRecently, the technology of assisting the navigation of visually impaired persons with computer vision has been greatly developed. A number of scholars have conducted related research, including indoor and outdoor object detection for blind ...
Comments
Information & Contributors
Information
Published In
October 2024
1475 pages
ISBN:9798400706776
DOI:10.1145/3663548
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2024
Check for updates
Author Tags
Qualifiers
- Poster
- Research
- Refereed limited
Funding Sources
- Pacific Northwest Transportation Consortium
- NSF (National Science Foundation)
Conference
ASSETS '24
Sponsor:
ASSETS '24: The 26th International ACM SIGACCESS Conference on Computers and Accessibility
October 27 - 30, 2024
NL, St. John's, Canada
Acceptance Rates
Overall Acceptance Rate 436 of 1,556 submissions, 28%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 58Total Downloads
- Downloads (Last 12 months)58
- Downloads (Last 6 weeks)5
Reflects downloads up to 28 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format