research-article

Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs in UI Development

Authors:

Chunyang ChenAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems, Volume 12, Issue 4

Article No.: 36, Pages 1 - 26

https://doi.org/10.1145/3531065

Published: 04 November 2022 Publication History

Abstract

Approximately 50% of development resources are devoted to user interface (UI) development tasks [9]. Occupying a large proportion of development resources, developing icons can be a time-consuming task, because developers need to consider not only effective implementation methods but also easy-to-understand descriptions. In this article, we present Auto-Icon+, an approach for automatically generating readable and efficient code for icons from design artifacts. According to our interviews to understand the gap between designers (icons are assembled from multiple components) and developers (icons as single images), we apply a heuristic clustering algorithm to compose the components into an icon image. We then propose an approach based on a deep learning model and computer vision methods to convert the composed icon image to fonts with descriptive labels, thereby reducing the laborious manual effort for developers and facilitating UI development. We quantitatively evaluate the quality of our method in the real-world UI development environment and demonstrate that our method offers developers accurate, efficient, readable, and usable code for icon designs, in terms of saving 65.2% implementing time.

References

[1]

Rakesh Agrawal and Ramakrishnan Srikant. 1994. Fast algorithms for mining association rules. In Proceedings of the 20th International Conference on Very Large Data Bases (VLDB’94), Vol. 1215. 487–499.

[2]

Airbnb. 2021. Sketching Interfaces: Generating Code from Low Fidelity Wireframes. Retrieved May 6, 2022 from https://airbnb.design/sketching-interfaces/.

[3]

Alibaba. 2020. Iconfont+. Retrieved September 28, 2020 from https://www.iconfont.cn/.

[4]

Alibaba. 2020. Imgcook. Retrieved September 29, 2020 from https://www.imgcook.com/.

[5]

Android. 2019. Navigate a Systrace Report. Retrieved September 1, 2020 from https://developer.android.com/studio/profile/sys trace/navigate-report.

[6]

Gary Ang and Ee Peng Lim. 2021. Learning network-based multi-modal mobile user interface embeddings. In Proceedings of the 26th International Conference on Intelligent User Interfaces. 366–376.

Digital Library

[7]

Apple. 2021. Xcode Overview—Apple Developer. Retrieved May 6, 2022 from https://developer.apple.com/xcode/.

[8]

Robert D. Austin. 2001. The effects of time pressure on quality in software development: An agency model. Information Systems Research 12, 2 (2001), 195–207.

Digital Library

[9]

Michel Beaudouin-Lafon. 2004. Designing interaction, not interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces. 15–22.

Digital Library

[10]

Tony Beltramelli. 2018. Pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 1–6.

Digital Library

[11]

Alison Black. 2017. Icons as carriers of information. In Information Design: Research and Practice. Routledge, 315–329.

[12]

Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021. VINS: Visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[13]

Margaret Butler. 2010. Android: Changing the mobile landscape. IEEE Pervasive Computing 10, 1 (2010), 4–7.

Digital Library

[14]

Antonin Carette, Mehdi Adel Ait Younes, Geoffrey Hecht, Naouel Moha, and Romain Rouvoy. 2017. Investigating the energy impact of Android smells. In Proceedings of the 2017 IEEE 24th International Conference on Software Analysis, Evolution, and Reengineering (SANER’17). IEEE, Los Alamitos, CA, 115–126.

[15]

Chunyang Chen, Sidong Feng, Zhengyang Liu, Zhenchang Xing, and Shengdong Zhao. 2020. From lost to found: Discover missing UI design semantics through recovering missing tags. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–22.

Digital Library

[16]

Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery DC: Design search and knowledge discovery through auto-created GUI component gallery. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–22.

Digital Library

[17]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI design image to GUI skeleton: A neural machine translator to bootstrap mobile GUI implementation. In Proceedings of the 40th International Conference on Software Engineering. 665–676.

Digital Library

[18]

Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhut, Guoqiang Li, and Jinshui Wang. 2020. Unblind your apps: Predicting natural-language labels for mobile GUI components by deep learning. In Proceedings of the 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE’20). IEEE, Los Alamitos, CA, 322–334.

Digital Library

[19]

Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, and Liming Zhu. 2020. Object detection for graphical user interface: Old fashioned or deep learning or a combination?arXiv preprint arXiv:2008.05132 (2020).

[20]

Qiuyuan Chen, Chunyang Chen, Safwat Hassan, Zhengchang Xing, Xin Xia, and Ahmed E. Hassan. 2021. How should I improve the UI of my app? A study of user reviews of popular apps in the Google Play. ACM Transactions on Software Engineering and Methodology 30, 3 (2021), 1–38.

Digital Library

[21]

Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019. StoryDroid: Automated generation of storyboard for Android apps. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). IEEE, Los Alamitos, CA, 596–607.

Digital Library

[22]

Sen Chen, Lingling Fan, Chunyang Chen, Minhui Xue, Yang Liu, and Lihua Xu. 2019. GUI-squatting attack: Automated generation of Android phishing apps. IEEE Transactions on Dependable and Secure Computing 18, 6 (2019), 2551–2568.

Digital Library

[23]

Sen Chen, Minhui Xue, Lingling Fan, Shuang Hao, Lihua Xu, Haojin Zhu, and Bo Li. 2018. Automated poisoning attacks and defenses in malware detection systems: An adversarial machine learning approach. Computers & Security 73 (2018), 326–344.

[24]

Yizong Cheng. 1995. Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence 17, 8 (1995), 790–799.

Digital Library

[25]

Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning 20, 3 (1995), 273–297.

[26]

Tobias Dehling, Fangjian Gao, Stephan Schneider, and Ali Sunyaev. 2015. Exploring the far side of mobile health: Information security and privacy of mobile health apps on iOS and Android. JMIR mHealth and uHealth 3, 1 (2015), e8.

[27]

Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological) 39, 1 (1977), 1–22.

[28]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 248–255.

[29]

Android Developers. 2020. Inspect GPU Rendering Speed and Overdraw. Retrieved September 28, 2020 from https://developer.android.com/topic/performance/ rendering/inspect-gpu-rendering.

[30]

Android Developers. 2022. Slow rendering. Retrieved June 16, 2022 from https://developer.android.com/topic/performance/vitals/render.

[31]

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD’96), Vol. 96. 226–231.

[32]

Michael P. Fay and Michael A. Proschan. 2010. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics Surveys 4 (2010), 1.

[33]

Ruitao Feng, Sen Chen, Xiaofei Xie, Lei Ma, Guozhu Meng, Yang Liu, and Shang-Wei Lin. 2019. MobiDroid: A performance-sensitive malware detection system on mobile platform. In Proceedings of the 2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS’19). IEEE, Los Alamitos, CA, 61–70.

[34]

Sidong Feng and Chunyang Chen. 2021. GIFdroid: Automated replay of visual bug reports for Android apps. arXiv preprint arXiv:2112.04128 (2021).

[35]

Sidong Feng, Chunyang Chen, and Zhenchang Xing. 2022. Gallery DC: Auto-created GUI component gallery for design search and knowledge discovery. arXiv preprint arXiv:2204.06700 (2022).

[36]

Sidong Feng, Suyu Ma, Jinzhong Yu, Chunyang Chen, Tingting Zhou, and Yankun Zhen. 2021. Auto-Icon: An automated code generation tool for icon designs assisting in UI development. In Proceedings of the 26th International Conference on Intelligent User Interfaces. 59–69.

Digital Library

[37]

FluidUI. 2021. FluidUI.com—Create Web and Mobile Prototypes in Minutes. Retrieved May 6, 2022 from https://www.fluidui.com/.

[38]

Fontello. 2020. Icon Fonts Generator. Retrieved September 7, 2020 from http://fontello.com/.

[39]

Fonticons. 2020. Font Awesome. Retrieved September 28, 2020 from https://fontawesome.com/.

[40]

Bin Fu, Jialiu Lin, Lei Li, Christos Faloutsos, Jason Hong, and Norman Sadeh. 2013. Why people hate your app: Making sense of user feedback in a mobile app store. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1276–1284.

Digital Library

[41]

Yi Gao, Yang Luo, Daqing Chen, Haocheng Huang, Wei Dong, Mingyuan Xia, Xue Liu, and Jiajun Bu. 2017. Every pixel counts: Fine-grained UI rendering analysis for mobile applications. In Proceedings of the IEEE INFOCOM Conference on Computer Communications (INFOCOM’17). IEEE, Los Alamitos, CA, 1–9.

[42]

María Gómez, Romain Rouvoy, Bram Adams, and Lionel Seinturier. 2016. Mining test repositories for automatic detection of UI performance regressions in Android apps. In Proceedings of the 13th International Conference on Mining Software Repositories. 13–24.

Digital Library

[43]

Google. 2020. Material Icons. Retrieved September 9, 2020 from https://material.io/resources/icons/.

[44]

Chris Harrison, Gary Hsieh, Karl D. D. Willis, Jodi Forlizzi, and Scott E. Hudson. 2011. Kineticons: Using iconographic motion in graphical user interface design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1999–2008.

Digital Library

[45]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.

[46]

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).

[47]

Gang Huang, Mengwei Xu, Felix Xiaozhu Lin, Yunxin Liu, Yun Ma, Saumay Pushp, and Xuanzhe Liu. 2017. ShuffleDog: Characterizing and adapting user-perceived latency of Android apps. IEEE Transactions on Mobile Computing 16, 10 (2017), 2913–2926.

Digital Library

[48]

IBM. 2020. UI Icons: Design Language. Retrieved September 9, 2020 from https://www.ibm.com/design/language/iconography/ ui-icons/library.

[49]

IcoMoon. 2020. Icon Font and SVG Icon Sets. Retrieved September 7, 2020 from https://icomoon.io/.

[50]

Muhammad Nazrul Islam. 2015. Exploring the intuitiveness of iconic, textual and icon with texts signs for designing user-intuitive web interfaces. In Proceedings of the 2015 18th International Conference on Computer and Information Technology (ICCIT’15). IEEE, Los Alamitos, CA, 450–455.

[51]

Bridgett A. King and Norman E. Youngblood. 2016. E-government in Alabama: An analysis of county voting and election website content, usability, accessibility, and mobile readiness. Government Information Quarterly 33, 4 (2016), 715–726.

[52]

Charalambos Koutsourelakis and Konstantinos Chorianopoulos. 2010. Icons in mobile phones: Comprehensibility differences between older and younger users. Information Design Journal 18, 1 (2010), 22–35.

[53]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097–1105.

Digital Library

[54]

Rob Larsen. 2018. Mastering SVG: Ace Web Animations, Visualizations, and Vector Graphics with HTML, CSS, and JavaScript. Packt Publishing Ltd.

[55]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998), 2278–2324.

[56]

Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020. GUIComp: A GUI design assistant with real-time, multi-faceted feedback. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[57]

Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10 (1966), 707–710.

[58]

Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A. Myers. 2021. Screen2Vec: Semantic embedding of GUI screens and GUI components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.

Digital Library

[59]

Wenjie Li, Yanyan Jiang, Chang Xu, Yepang Liu, Xiaoxing Ma, and Jian Lü. 2019. Characterizing and detecting inefficient image displaying issues in Android apps. In Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering (SANER’19). IEEE, Los Alamitos, CA, 355–365.

[60]

Aristidis Likas, Nikos Vlassis, and Jakob J. Verbeek. 2003. The global k-means clustering algorithm. Pattern Recognition 36, 2 (2003), 451–461.

[61]

Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv preprint arXiv:1312.4400 (2013).

[62]

Mario Linares-Vasquez, Christopher Vendome, Qi Luo, and Denys Poshyvanyk. 2015. How developers detect and fix performance bottlenecks in Android apps. In Proceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME’15). IEEE, Los Alamitos, CA, 352–361.

Digital Library

[63]

Yepang Liu, Chang Xu, and Shing-Chi Cheung. 2014. Characterizing and detecting performance bugs for smartphone applications. In Proceedings of the 36th International Conference on Software Engineering. 1013–1024.

Digital Library

[64]

Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl eyes: Spotting UI display issues via visual understanding. In Proceedings of the 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE’20). IEEE, Los Alamitos, CA, 398–409.

Digital Library

[65]

Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Guided bug crush: Assist manual GUI testing of Android apps via hint moves. arXiv preprint arXiv:2201.12085 (2022).

[66]

David G. Lowe. 1999. Object recognition from local scale-invariant features. In Proceedings of the 7th IEEE International Conference on Computer Vision (ICCV’99), Vol. 99. 1150–1157.

[67]

Muhammad Noman Malik, Huma Hayat Khan, and Fazli Subhan. 2017. Sustainable design of mobile icons: Investigating effect on mentally retarded user’s. Journal of Medical Imaging and Health Informatics 7, 6 (2017), 1419–1428.

[68]

Mika V. Mäntylä, Kai Petersen, Timo O. A. Lehtinen, and Casper Lassenius. 2014. Time pressure: A controlled experiment of test case development and requirements review. In Proceedings of the 36th International Conference on Software Engineering. 83–94.

Digital Library

[69]

Jan Meskens, Kris Luyten, and Karin Coninx. 2009. Plug-and-design: Embracing mobile devices as part of the design environment. In Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 149–154.

Digital Library

[70]

Jan Meskens, Kris Luyten, and Karin Coninx. 2009. Shortening user interface design iterations through realtime visualisation of design actions on the target device. In Proceedings of the 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’09). IEEE, Los Alamitos, CA, 132–135.

Digital Library

[71]

Microsoft. 2021. Sketch2Code. Retrieved May 6, 2022 from https://sketch2code.azurewebsites.net/.

[72]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. 3111–3119.

Digital Library

[73]

Mockup. 2021. Mockup.io—Present and Manage iOS Mockups. Retrieved May 6, 2022 from https://mockup.io/about/.

[74]

Higinio Mora, Virgilio Gilart-Iglesias, Raquel Pérez-del Hoyo, and María Dolores Andújar-Montoya. 2017. A comprehensive system for monitoring urban accessibility in smart cities. Sensors 17, 8 (2017), 1834.

[75]

Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated reporting of GUI design violations for mobile apps. In Proceedings of the 40th International Conference on Software Engineering. 165–175.

Digital Library

[76]

Kevin Patrick Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Transactions on Software Engineering 46, 2 (2018), 196–221.

[77]

Fionn Murtagh and Pierre Legendre. 2014. Ward’s hierarchical agglomerative clustering method: Which algorithms implement Ward’s criterion?Journal of Classification 31, 3 (2014), 274–295.

Digital Library

[78]

Marc Najork and Janet L. Wiener. 2001. Breadth-first crawling yields high-quality pages. In Proceedings of the 10th International Conference on World Wide Web. 114–118.

Digital Library

[79]

Theresa Neil. 2014. Mobile Design Pattern Gallery: UI Patterns for Smartphone Apps. O’Reilly Media Inc.

[80]

Javad Nejati and Aruna Balasubramanian. 2016. An in-depth study of mobile browser performance. In Proceedings of the 25th International Conference on World Wide Web. 1305–1315.

Digital Library

[81]

Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse engineering mobile application user interfaces with REMAUI (T). In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE’15). IEEE, Los Alamitos, CA, 248–259.

Digital Library

[82]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311–318.

Digital Library

[83]

Photoshop. 2021. Photoshop Apps—Desktop, Mobile, and Tablet. Retrieved May 6, 2022 from https://www.photoshop.com/en.

[84]

Proto. 2021. Proto.io Prototyping Tool—Prototyping for All. Retrieved May 6, 2022 from https://proto.io/.

[85]

J. Ross Quinlan. 1983. Learning efficient classification procedures and their application to chess end games. In Machine Learning. Springer, 463–482.

[86]

Sanae Rosen, Bo Han, Shuai Hao, Z. Morley Mao, and Feng Qian. 2017. Push or request: An investigation of HTTP/2 server push for improving mobile performance. In Proceedings of the 26th International Conference on World Wide Web. 459–468.

Digital Library

[87]

Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O. Wobbrock. 2018. Examining image-based button labeling for accessibility in Android apps through large-scale analysis. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. 119–130.

Digital Library

[88]

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4510–4520.

[89]

Thomas W. Sederberg and Rida T. Farouki. 1992. Approximation by interval Bézier curves. IEEE Computer Graphics and Applications5 (1992), 87–88.

Digital Library

[90]

Peter Selinger. 2003. Potrace: A polygon-based tracing algorithm. Potrace. Retrieved May 6, 2022 from http://potrace.sourceforge.net/potrace.

[91]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[92]

Sketch. 2021. Sketch Developer—File Format. Retrieved May 6, 2022 from https://developer.sketch.com/file-format/.

[93]

Sketch. 2021. Sketch—The Digital Design Toolkit. Retrieved May 6, 2022 from https://www.sketch.com.

[94]

Android Studio. 2021. Android Studio Home Page. Retrieved May 6, 2022 from https://developer.android.com/studio.

[95]

Shamik Sural, Gang Qian, and Sakti Pramanik. 2002. Segmentation and histogram generation using the HSV color space for image retrieval. In Proceedings of the International Conference on Image Processing, Vol. 2. IEEE, Los Alamitos, CA, II.

[96]

Kashyap Todi, Luis A. Leiva, Daniel Buschek, Pin Tian, and Antti Oulasvirta. 2021. Conversations with GUIs. In Proceedings of the 2021 Designing Interactive Systems Conference. 1447–1457.

Digital Library

[97]

Fahui Wang. 2012. Measurement, optimization, and impact of health care accessibility: A methodological review. Annals of the Association of American Geographers 102, 5 (2012), 1104–1112.

[98]

Xu Wang, Chunyang Chen, and Zhenchang Xing. 2019. Domain-specific machine translation with recurrent neural network for software localization. Empirical Software Engineering 24, 6 (2019), 3514–3545.

[99]

Xiang-Yang Wang, Jun-Feng Wu, and Hong-Ying Yang. 2010. Robust image retrieval based on color histogram of local feature regions. Multimedia Tools and Applications 49, 2 (2010), 323–345.

Digital Library

[100]

Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.

Digital Library

[101]

Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P. Bigham. 2021. Screen parsing: Towards reverse engineering of UI models from screenshots. In Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology. 470–483.

Digital Library

[102]

Tianyong Wu, Jierui Liu, Zhenbo Xu, Chaorong Guo, Yanli Zhang, Jun Yan, and Jian Zhang. 2016. Light-weight, inter-procedural and callback-aware resource leak detection for Android apps. IEEE Transactions on Software Engineering 42, 11 (2016), 1054–1076.

Digital Library

[103]

Xusheng Xiao, Xiaoyin Wang, Zhihao Cao, Hanlin Wang, and Peng Gao. 2019. ICONINTENT: Automatic identification of sensitive UI widgets based on icon classification for Android apps. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). IEEE, Los Alamitos, CA, 257–268.

Digital Library

[104]

Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: A hybrid tool for GUI element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1655–1659.

Digital Library

[105]

Shunguo Yan and P. G. Ramachandran. 2019. The current status of accessibility in mobile apps. ACM Transactions on Accessible Computing 12, 1 (2019), 1–31.

Digital Library

[106]

Bo Yang, Zhenchang Xing, Xin Xia, Chunyang Chen, Deheng Ye, and Shanping Li. 2021. UIS-hunter: Detecting UI design smells in Android apps. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion’21). IEEE, Los Alamitos, CA, 89–92.

Digital Library

[107]

Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, et al. 2021. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.

Digital Library

[108]

Xiaoyi Zhang, Anne Spencer Ross, and James Fogarty. 2018. Robust annotation of mobile application interfaces in methods for accessibility repair and enhancement. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 609–621.

Digital Library

[109]

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, and Guoqiang Li. 2019. ActionNet: Vision-based workflow action recognition from programming screencasts. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). IEEE, Los Alamitos, CA, 350–361.

Digital Library

[110]

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Seenomaly: Vision-based linting of GUI animation effects against design-don’t guidelines. In Proceedings of the 42nd International Conference on Software Engineering (ICSE’20). ACM, New York, NY.

Digital Library

[111]

Hui Zhao, Min Chen, Meikang Qiu, Keke Gai, and Meiqin Liu. 2016. A novel pre-cache schema for high performance Android system. Future Generation Computer Systems 56 (2016), 766–772.

Digital Library

[112]

Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. GUIGAN: Learning to generate GUI designs using generative adversarial networks. In Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE’21). IEEE, Los Alamitos, CA, 748–760.

Digital Library

Cited By

Feng SLu HJiang JXiong THuang LLiang YLi XDeng YAleti AFilkov VRay BZhou M(2024)Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695260(1973-1978)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695260
Feng SMa SWang HKong DChen C(2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642350
Dubiel MBarghouti YKudryavtseva KLeiva L(2024)On-device query intent prediction with lightweight LLMs to support ubiquitous conversationsScientific Reports10.1038/s41598-024-63380-614:1Online publication date: 3-Jun-2024
https://doi.org/10.1038/s41598-024-63380-6
Show More Cited By

Index Terms

Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs in UI Development
1. Human-centered computing
  1. Accessibility
    1. Empirical studies in accessibility

Recommendations

Auto-Icon: An Automated Code Generation Tool for Icon Designs Assisting in UI Development
IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

Approximately 50% of development resources are devoted to UI development tasks [8]. Occupied a large proportion of development resources, developing icons can be a time-consuming task, because developers need to consider not only effective ...
A case study of icon-scenario based animated menu's concept development
MobileHCI '06: Proceedings of the 8th conference on Human-computer interaction with mobile devices and services

This paper reports on the implementation of the Icon-Scenario Based Animated Menu graphical user interface that is adopted across our latest 2G platform. We describe the concept development process of the phone's menu enhanced by the use of a ...
Effects of training and representational characteristics in icon design

Icons are a very important component of graphical user interfaces. However, icon design is still predominantly artistic in nature and as a result icon selection is generally based on usability evaluations after a set of alternative icons are developed. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 12, Issue 4

December 2022

321 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/3561952

Editor:
Shlomo Berkovsky
Macquarie University, Australia

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 November 2022

Online AM: 27 April 2022

Accepted: 11 April 2022

Revised: 11 March 2022

Received: 30 July 2021

Published in TIIS Volume 12, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
648
Total Downloads

Downloads (Last 12 months)224
Downloads (Last 6 weeks)31

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Feng SLu HJiang JXiong THuang LLiang YLi XDeng YAleti AFilkov VRay BZhou M(2024)Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695260(1973-1978)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695260
Feng SMa SWang HKong DChen C(2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642350
Dubiel MBarghouti YKudryavtseva KLeiva L(2024)On-device query intent prediction with lightweight LLMs to support ubiquitous conversationsScientific Reports10.1038/s41598-024-63380-614:1Online publication date: 3-Jun-2024
https://doi.org/10.1038/s41598-024-63380-6
Kumar TGarg VLalar SKumar R(2024)Measuring Impact of Generative AI in Software Development and InnovationIntelligent IT Solutions for Sustainability in Industry 5.0 Paradigm10.1007/978-981-97-1682-1_6(57-67)Online publication date: 5-Jul-2024
https://doi.org/10.1007/978-981-97-1682-1_6
Li LWang RZhan XWang YGao CWang SLiu YJust RFraser G(2023)What You See Is What You Get? It Is Not the Case! Detecting Misleading Icons for Mobile ApplicationsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598076(538-550)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598076
Feng SChen CXing Z(2023)Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial VideosProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606778(1-15)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606778
Feng SXie MXue YChen CGrundy JPollock LPenta M(2023)Read It, Don't Watch It: Captioning Bug Recordings AutomaticallyProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00197(2349-2361)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00197
Feng SXie MChen CGrundy JPollock LPenta M(2023)Efficiency Matters: Speeding Up Automated Testing with GUI Rendering InferenceProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00084(906-918)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00084
Zhang ZDing YHuang C(2023)Automatic Front-end Code Generation from image Via Multi-Head Attention2023 4th International Conference on Computer Engineering and Application (ICCEA)10.1109/ICCEA58433.2023.10135462(869-872)Online publication date: 7-Apr-2023
https://doi.org/10.1109/ICCEA58433.2023.10135462
Raptis GKatsini C(2023)Transforming Ideas into Code: A Novel Approach to Design-to-Development Workflow2023 International Conference on Computer and Applications (ICCA)10.1109/ICCA59364.2023.10401412(1-6)Online publication date: 28-Nov-2023
https://doi.org/10.1109/ICCA59364.2023.10401412

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents