skip to main content
10.1109/ASE51524.2021.9678625acmconferencesArticle/Chapter ViewAbstractPublication PagesaseConference Proceedingsconference-collections
research-article

Automated repair of size-based inaccessibility issues in mobile applications

Published: 24 June 2022 Publication History

Abstract

An increasing number of people are dependent on mobile devices to access data and complete essential tasks. For people with disabilities, mobile apps that violate accessibility guidelines can prevent them from carrying out these activities. Size-Based Inaccessibility is one of the top accessibility issues in mobile applications. These issues make apps difficult to use, especially for older people and people with motor disabilities. Existing accessibility related techniques are limited in terms of helping developers to resolve these issues. In this paper, we present our novel automated approach for repairing Size-Based Inaccessibility issues in mobile applications. Our empirical evaluation showed that our approach was able to successfully resolve 99% of the reported Size-Based Inaccessibility issues and received a high approval rating in a user study of the appearance of the repaired user interfaces.

References

[1]
"WHO-Disability and health." [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/disability-and-health
[2]
J. T. Morris, M. W. Sweatman, and M. L. Jones, "Smartphone use and activities by people with disabilities: 2015--2016 survey," Journal on Technology Persons with Disabilities, vol. 5, p. 50--66. [Online]. Available: http://hdl.handle.net/10211.3/190202
[3]
J. Lazar, "The potential role of U.S. consumer protection laws in improving digital accessibility for people with disabilities," U. Pa. JL & Soc. Change, vol. 22, p. 185, 2019.
[4]
"Level Access: What Accessibility Standards Apply to Mobile Applications?" https://www.levelaccess.com/what-accessibility-standards-apply-to-mobile-phone-applications/, updated: 2021-04-21.
[5]
S. Yan and P. G. Ramachandran, "The Current Status of Accessibility in Mobile Apps," ACM Transactions on Accessible Computing, vol. 12, no. 1, pp. 1--31, Feb. 2019. [Online]. Available: http://dl.acm.org/citation.cfm?doid=3312747.3300176
[6]
C. Vendome, D. Solano, S. Liñán, and M. Linares-Vásquez, "Can Everyone use my app? An Empirical Study on Accessibility in Android Apps," in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), Sep. 2019, pp. 41--52, iSSN: 2576--3148.
[7]
A. Alshayban, I. Ahmed, and S. Malek, "Accessibility Issues in Android Apps: State of Affairs, Sentiments, and Ways Forward," 2020, p. 12.
[8]
A. S. Ross, X. Zhang, J. Fogarty, and J. O. Wobbrock, "An Epidemiology-inspired Large-scale Analysis of Android App Accessibility," ACM Transactions on Accessible Computing, vol. 13, no. 1, pp. 4:1--4:36, Apr. 2020. [Online].
[9]
M. E. Mott, R.-D. Vatavu, S. K. Kane, and J. O. Wobbrock, "Smart Touch: Improving Touch Accuracy for People with Motor Impairments with Template Matching," in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ser. CHI '16. New York, NY, USA: Association for Computing Machinery, May 2016, pp. 1934--1946. [Online].
[10]
S. N. Duff, C. B. Irwin, J. L. Skye, M. E. Sesto, and D. A. Wiegmann, "The Effect of Disability and Approach on Touch Screen Performance during a Number Entry Task," Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 54, no. 6, pp. 566--570, Sep. 2010, publisher: SAGE Publications Inc. [Online].
[11]
X. Valencia, J. E. Pérez, M. Arrue, J. Abascal, C. Duarte, and L. Moreno, "Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets," Interacting with Computers, vol. 29, no. 6, pp. 794--812, Nov. 2017. [Online].
[12]
T. Guerreiro, H. Nicolau, J. Jorge, and D. Gonçalves, "Towards accessible touch interfaces," in Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility, ser. ASSETS '10. New York, NY, USA: Association for Computing Machinery, Oct. 2010, pp. 19--26. [Online].
[13]
L. Nurgalieva, J. J. J. Laconich, M. Baez, F. Casati, and M. Marchese, "A Systematic Literature Review of Research-Derived Touchscreen Design Guidelines for Older Adults," IEEE Access, vol. 7, pp. 22 035--22 058, 2019, conference Name: IEEE Access.
[14]
"Material Desing Accessibility." [Online]. Available: https://material.io/design/usability/accessibility.htmllayout-and-typography
[15]
"W3 Target Size." [Online]. Available: https://www.w3.org/WAI/WCAG21/Understanding/target-size
[16]
"BBC Mobile Accessibility Guidlone." [Online]. Available: https://www.bbc.co.uk/accessibility/forproducts/guides/mobile/
[17]
X. Zhang, T. Tran, Y. Sun, I. Culhane, S. Jain, J. Fogarty, and J. Mankoff, "Interactiles: 3d printed tactile interfaces to enhance mobile touchscreen accessibility," in Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ser. ASSETS '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 131--142. [Online].
[18]
B. Taylor, A. Dey, D. Siewiorek, and A. Smailagic, "Customizable 3d printed tactile maps as interactive overlays," in Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, ser. ASSETS '16. New York, NY, USA: Association for Computing Machinery, 2016, p. 71--79. [Online].
[19]
L. He, Z. Wan, L. Findlater, and J. E. Froehlich, "Tactile: A preliminary toolchain for creating accessible graphics with 3d-printed overlays and auditory annotations," in Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, ser. ASSETS '17. New York, NY, USA: Association for Computing Machinery, 2017, p. 397--398. [Online].
[20]
S. K. Kane, M. R. Morris, and J. O. Wobbrock, "Touchplates: Low-cost tactile overlays for visually impaired touch screen users," in Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, ser. ASSETS '13. New York, NY, USA: Association for Computing Machinery, 2013. [Online].
[21]
S. K. Kane, M. R. Morris, A. Z. Perkins, D. Wigdor, R. E. Ladner, and J. O. Wobbrock, "Access overlays: Improving non-visual access to large touch screens for blind users," in Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ser. UIST '11. New York, NY, USA: Association for Computing Machinery, 2011, p. 273--282. [Online].
[22]
S. K. Kane, J. P. Bigham, and J. O. Wobbrock, "Slide rule: Making mobile touch screens accessible to blind people using multi-touch interaction techniques," in Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility, ser. Assets '08. New York, NY, USA: Association for Computing Machinery, 2008, p. 73--80. [Online].
[23]
S. Azenkot, C. L. Bennett, and R. E. Ladner, "Digitaps: Eyes-free number entry on touchscreens with minimal audio feedback," in Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, ser. UIST '13. New York, NY, USA: Association for Computing Machinery, 2013, p. 85--90. [Online].
[24]
X. Zhang, A. S. Ross, A. Caspi, J. Fogarty, and J. O. Wobbrock, "Interaction proxies for runtime repair and enhancement of mobile application accessibility," in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ser. CHI '17. New York, NY, USA: Association for Computing Machinery, 2017, p. 6024--6037. [Online].
[25]
Y. Zhong, A. Weber, C. Burkhardt, P. Weaver, and J. P. Bigham, "Enhancing Android accessibility for users with hand tremor by reducing fine pointing and steady tapping," in Proceedings of the 12th International Web for All Conference. Florence Italy: ACM, May 2015, pp. 1--10. [Online].
[26]
"UI Automator." [Online]. Available: https://developer.android.com/training/testing/ui-automator
[27]
"Layout Inspector." [Online]. Available: https://developer.android.com/studio/debug/layout-inspector
[28]
"Google Accessibility for Android." [Online]. Available: https://developer.android.com/guide/topics/ui/accessibility
[29]
"Android Accessibility Help: Get started with Accessibility Scanner," https://support.google.com/accessibility/android/answer/6376570?hl=en, updated: 2021-04-21.
[30]
"google/Accessibility-Test-Framework-for-Android," Dec. 2020, original-date: 2015-09-12T00:49:01Z. [Online]. Available: https://github.com/google/Accessibility-Test-Framework-for-Android
[31]
"IBM Mobile Accessibility Checker," May 2020, original-date: 2017-11-06T14:35:17Z. [Online]. Available: https://github.com/IBMa/MAC
[32]
A. Sanoja and S. Gançarski, "Block-o-Matic: A web page segmentation framework," in 2014 International Conference on Multimedia Computing and Systems (ICMCS), Apr. 2014, pp. 595--600.
[33]
S. Mahajan and W. G. J. Halfond, "Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques," in 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), Apr. 2015, pp. 1--10, iSSN: 2159--4848.
[34]
S. Mahajan, N. Abolhassani, P. McMinn, and W. G. J. Halfond, "Automated repair of mobile friendly problems in web pages," in Proceedings of 40th International Conference on Software Engineering. Gothenburg Sweden: ACM, May 2018, pp. 140--150. [Online].
[35]
M. Ester, H.-P. Kriegel, and X. Xu, "A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise," p. 6.
[36]
"Apktool - A tool for reverse engineering 3rd party, closed, binary Android apps." [Online]. Available: https://ibotpeaches.github.io/Apktool/
[37]
"Google Developers: Android Debug Bridge (adb)," https://developer.android.com/studio/command-line/adb, updated: 2021-02-18.
[38]
"SALEM Project Web Site," Aug. 2021. [Online]. Available: https://sites.google.com/usc.edu/salem/
[39]
Google, "Android Apps on Google Play." [Online]. Available: https://play.google.com/store/apps?hl=engl=US
[40]
"Screen Sizes," https://screensiz.es/nexus-6p, updated: 2021-04-19.
[41]
"Amazon Mechanical Turk." [Online]. Available: https://www.mturk.com/
[42]
S. Mahajan, A. Alameer, P. McMinn, and W. G. J. Halfond, "Automated repair of internationalization presentation failures in web pages using style similarity clustering and search-based techniques," in 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), 2018, pp. 215--226.
[43]
A. Alameer, P. Chiou, and W. G. Halfond, "Efficiently repairing internationalization presentation failures by solving layout constraints," in Proceedings of the IEEE international conference on software testing, verification, and validation (ICST), Apr. 2019, tex.acceptancerate: 28% (31/110) tex.pubtype: Conference.
[44]
M. Linares-Vásquez, G. Bavota, C. Bernal-Cárdenas, M. D. Penta, R. Oliveto, and D. Poshyvanyk, "Multi-Objective Optimization of Energy Consumption of GUIs in Android Apps," ACM Transactions on Software Engineering and Methodology, vol. 27, no. 3, pp. 14:1--14:47, Sep. 2018. [Online].
[45]
S. Hao, B. Liu, S. Nath, W. G. Halfond, and R. Govindan, "Puma: Programmable ui-automation for large scale dynamic analysis of mobile apps," in Proceedings of the ACM International Conference on Mobile Systems, Applications, and Services (MobiSys), June 2014.
[46]
N. Salehnamadi, A. Alshayban, J.-W. Lin, I. Ahmed, S. Branham, and S. Malek, "Latte: Use-Case and Assistive-Service Driven Automated Accessibility Testing Framework for Android," in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery, May 2021, no. 274, pp. 1--11. [Online].
[47]
P. T. Chiou, A. S. Alotaibi, and W. G. J. Halfond, "Detecting and localizing keyboard accessibility failures in web applications," in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2021. New York, NY, USA: Association for Computing Machinery, Aug. 2021, pp. 855--867. [Online].
[48]
M. Bajammal and A. Mesbah, "Semantic Web Accessibility Testing via Hierarchical Visual Analysis," in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), May 2021, pp. 1610--1621, iSSN: 1558--1225.
[49]
P. Panchekha, A. T. Geller, M. D. Ernst, Z. Tatlock, and S. Kamil, "Verifying that web pages have accessible layout," in Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, ser. PLDI 2018. New York, NY, USA: Association for Computing Machinery, Jun. 2018, pp. 1--14. [Online].
[50]
P. Panchekha, M. D. Ernst, Z. Tatlock, and S. Kamil, "Modular verification of web page layout," Proceedings of the ACM on Programming Languages, vol. 3, no. OOPSLA, pp. 151:1--151:26, Oct. 2019. [Online].
[51]
J. Wu, G. Reyes, S. C. White, X. Zhang, and J. P. Bigham, "When can accessibility help?: an exploration of accessibility feature recommendation on mobile devices," in Proceedings of the 18th International Web for All Conference. Ljubljana Slovenia: ACM, Apr. 2021, pp. 1--12. [Online].
[52]
X. Zhang, L. de Greef, A. Swearngin, S. White, K. Murray, L. Yu, Q. Shan, J. Nichols, J. Wu, C. Fleizach, A. Everitt, and J. P. Bigham, "Screen recognition: Creating accessibility metadata for mobile applications from pixels," 2021.
[53]
A. S. Ross, X. Zhang, J. Fogarty, and J. O. Wobbrock, "Examining image-based button labeling for accessibility in android apps through large-scale analysis," in Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, ser. ASSETS '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 119--130. [Online].
[54]
"Google: Our all -new TalkBack screen reader," https://blog.google/products/android/all-new-talkback/, updated: 2021-04-21.
[55]
X. Zhang, A. S. Ross, and J. Fogarty, "Robust annotation of mobile application interfaces in methods for accessibility repair and enhancement," in Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ser. UIST '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 609--621. [Online].
[56]
J. Chen, C. Chen, Z. Xing, X. Xu, L. Zhu, G. Li, and J. Wang, "Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning," Mar. 2020. [Online]. Available: https://arxiv.org/abs/2003.00380v2
[57]
F. Mehralian, N. Salehnamadi, and S. Malek, "Data-driven accessibility repair revisited: on the effectiveness of generating labels for icons in Android apps," in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2021. New York, NY, USA: Association for Computing Machinery, Aug. 2021, pp. 107--118. [Online].
[58]
E. Brady and J. P. Bigham, "Crowdsourcing accessibility: Human-powered access technologies," Foundations and Trends® in Human-Computer Interaction, vol. 8, no. 4, pp. 273--372, 2015. [Online].
[59]
"Android Accessibility Help: Magnification," https://support.google.com/accessibility/android/answer/6006949, updated: 2021-04-21.
[60]
Z. Liu, C. Chen, J. Wang, Y. Huang, J. Hu, and Q. Wang, "Owl eyes: Spotting ui display issues via visual understanding," in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '20. New York, NY, USA: Association for Computing Machinery, 2020, p. 398--409. [Online].
[61]
A. Alameer, P. T. Chiou, and W. G. J. Halfond, "Efficiently repairing internationalization presentation failures by solving layout constraints," in 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), 2019, pp. 172--182.
[62]
R. K. Saha, Y. Lyu, H. Yoshida, and M. R. Prasad, "Elixir: Effective object-oriented program repair," in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), Oct. 2017, pp. 648--659.
[63]
S. H. Tan, Z. Dong, X. Gao, and A. Roychoudhury, "Repairing crashes in Android apps," in Proceedings of the 40th International Conference on Software Engineering, ser. ICSE '18. New York, NY, USA: Association for Computing Machinery, May 2018, pp. 187--198. [Online].
[64]
R. S. Shariffdeen, S. H. Tan, M. Gao, and A. Roychoudhury, "Automated Patch Transplantation," ACM Transactions on Software Engineering and Methodology, vol. 30, no. 1, pp. 6:1--6:36, Dec. 2021. [Online].
[65]
F. Long and M. Rinard, "Automatic patch generation by learning correct code," in Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, ser. POPL '16. New York, NY, USA: Association for Computing Machinery, Jan. 2016, pp. 298--312. [Online].

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ASE '21: Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering
November 2021
1446 pages
ISBN:9781665403375

Sponsors

In-Cooperation

  • IEEE CS

Publisher

IEEE Press

Publication History

Published: 24 June 2022

Check for updates

Qualifiers

  • Research-article

Conference

ASE '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media