extended-abstract

Enrico: A Dataset for Topic Modeling of Mobile UI Designs

Authors:

Luis A. Leiva,

Asutosh Hota,

Antti OulasvirtaAuthors Info & Claims

MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services

Article No.: 9, Pages 1 - 4

https://doi.org/10.1145/3406324.3410710

Published: 25 February 2021 Publication History

Get Access

Abstract

Topic modeling of user interfaces (UIs), also known as layout design categorization, contributes to a better understanding of the UI functionality. Starting from Rico, a large dataset of mobile UIs, we revised a random sample of 10k UIs and concluded to Enrico (shorthand of Enhanced Rico), a human-supervised high-quality dataset comprising 1460 UIs and 20 design topics. As a validation example, we train a deep learning model for three different UI representations (screenshots, wireframes, and embeddings). The screenshot representation provides the highest discriminative power (95% AUC) and a competitive accuracy of 75% (a random classifier achieves 5% accuracy in the same task). We discuss several applications that can be developed with this new public resource, including e.g. semantic UI captioning and tagging, explainable UI designs, smart tutorials, and improved design search capabilities.

References

[1]

F. Behrang, S. P. Reiss, and A. Orso. 2018. GUIfetch: Supporting App Design and Development Through GUI Search. In Proc. MOBILESoft.

Google Scholar

[2]

G. Berardi, A. Esuli, T. Fagni, and F. Sebastiani. 2015. Multi-Store Metadata-Based Supervised Mobile App Classification. In Proc. SAC.

Google Scholar

[3]

B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, and R. Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proc. UIST.

Google Scholar

[4]

D. Hendrycks and K. Gimpel. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In Proc. ICLR.

Google Scholar

[5]

R. Kumar, A. Satyanarayan, C. Torres, M. Lim, S. Ahmad, S. R. Klemmer, and J. O. Talton. 2013. Webzeitgeist: Design Mining the Web. In Proc. CHI.

Digital Library

Google Scholar

[6]

C. Lee, S. Kim, D. Han, H. Yang, Y.-W. Park, B. C. Kwon, and S. Ko. 2020. GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback. In Proc. CHI.

Google Scholar

[7]

T. F. Liu, M. Craft, J. Situ, E. Yumer, R. Mech, and R. Kumar. 2018. Learning Design Semantics for Mobile Apps. In Proc. UIST.

Google Scholar

[8]

[8] L. McInnes, J. Healy, and J. Melville. 2018. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. ArXiv:1802.03426. (2018).

Google Scholar

[9]

E. Platzer and O. Petrovic. 2011. Learning Mobile App Design From User Review Analysis. iJIM 5, 3 (2011).

Google Scholar

[10]

D. Powers. 2011. Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation. J. Mach. Learn. Technol. 2, 1 (2011).

Google Scholar

[11]

M. T. Ribeiro, S. Singh, and C. Guestrin. 2016. Why should I trust you? Explaining the predictions of any classifier. In Proc. KDD.

Google Scholar

[12]

K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. ICLR.

Google Scholar

[13]

H. Zhu, E. Chen, H. Xiong, H. Cao, and J. Tian. 2014. Mobile App Classification with Enriched Contextual Information. IEEE Trans. Mob. Comput. 13, 7 (2014).

Crossref

Google Scholar

Cited By

View all

Liang PZadeh AMorency L(2024)Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open QuestionsACM Computing Surveys10.1145/365658056:10(1-42)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3656580
Wu JPeng YLi XSwearngin ABigham JNichols J(2024)UIClip: A Data-driven Model for Assessing User Interface DesignProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676408(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676408
Duan PCheng CLi GHartmann BLi Y(2024)UICrit: Enhancing Automated Design Evaluation with a UI Critique DatasetProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676381(1-17)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676381
Show More Cited By

Recommendations

Scenario-based interactive UI design
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Clearly picturing user behavior is one of the key requirements when designing successful interactive software. However, covering all possible user behaviors with one UI is a complex challenge. The Scenario-based Interactive UI Design tool is designed to ...
The University of Alberta user interface management system

In this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...
SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors
IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User Interfaces

User Interface (UI) prototyping is an iterative process where designers initially sketch UIs before transforming them into interactive digital designs. Recent research applies Deep Neural Networks (DNNs) to identify the constituent UI elements of these ...

Comments

Information & Contributors

Information

Published In

MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services

October 2020

248 pages

ISBN:9781450380522

DOI:10.1145/3406324

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 February 2021

Check for updates

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Conference

MobileHCI '20

MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services

October 5 - 8, 2020

Oldenburg, Germany

Acceptance Rates

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

33
Total Citations
View Citations
440
Total Downloads

Downloads (Last 12 months)102
Downloads (Last 6 weeks)8

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Liang PZadeh AMorency L(2024)Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open QuestionsACM Computing Surveys10.1145/365658056:10(1-42)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3656580
Wu JPeng YLi XSwearngin ABigham JNichols J(2024)UIClip: A Data-driven Model for Assessing User Interface DesignProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676408(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676408
Duan PCheng CLi GHartmann BLi Y(2024)UICrit: Enhancing Automated Design Evaluation with a UI Critique DatasetProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676381(1-17)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676381
Jiang YZhou CGarg VOulasvirta A(2024)Graph4GUI: Graph Neural Networks for Representing Graphical User InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642822(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642822
Feng SMa SWang HKong DChen C(2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642350
Huang FLi GLi TLi Y(2024)Automatic Macro Mining from Interaction Traces at ScaleProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642074(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642074
Dubiel MBarghouti YKudryavtseva KLeiva L(2024)On-device query intent prediction with lightweight LLMs to support ubiquitous conversationsScientific Reports10.1038/s41598-024-63380-614:1Online publication date: 3-Jun-2024
https://doi.org/10.1038/s41598-024-63380-6
Peng YHuq FJiang YWu JLi XBigham JPavel A(2024)DreamStruct: Understanding Slides and User Interfaces via Synthetic Data GenerationComputer Vision – ECCV 202410.1007/978-3-031-72691-0_26(466-485)Online publication date: 3-Nov-2024
https://doi.org/10.1007/978-3-031-72691-0_26
Hu HZhan HHuang YLiu DOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Pairwise GUI dataset construction between android phones and tabletsProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668737(59860-59872)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668737
Lehmann FKornecki IBuschek DFeit A(2023)Typing Behavior is About More than Speed: Users' Strategies for Choosing Word Suggestions Despite Slower Typing RatesProceedings of the ACM on Human-Computer Interaction10.1145/36042767:MHCI(1-26)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604276
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

Scenario-based interactive UI design

The University of Alberta user interface management system

SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors

Comments

Information

Published In

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations