skip to main content
10.1145/3173574.3173758acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time

Published: 19 April 2018 Publication History

Abstract

Developing cross-device multi-user interfaces (UIs) is a challenging problem. There are numerous ways in which content and interactivity can be distributed. However, good solutions must consider multiple users, their roles, their preferences and access rights, as well as device capabilities. Manual and rule-based solutions are tedious to create and do not scale to larger problems nor do they adapt to dynamic changes, such as users leaving or joining an activity. In this paper, we cast the problem of UI distribution as an assignment problem and propose to solve it using combinatorial optimization. We present a mixed integer programming formulation which allows real-time applications in dynamically changing collaborative settings. It optimizes the allocation of UI elements based on device capabilities, user roles, preferences, and access rights. We present a proof-of-concept designer-in-the-loop tool, allowing for quick solution exploration. Finally, we compare our approach to traditional paper prototyping in a lab study.

Supplementary Material

suppl.mov (pn2177-file5.mp4)
Supplemental video
suppl.mov (pn2177.mp4)
Supplemental video

References

[1]
Sriram Karthik Badam and Niklas Elmqvist. 2014. PolyChrome: A Cross-Device Framework for Collaborative Web Visualization. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM, New York, NY, USA, 109--118.
[2]
Pei-Yu (Peggy) Chi and Yang Li. 2015. Weave: Scripting Cross-Device Wearable Interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3923--3932.
[3]
Tao Dong, Elizabeth F. Churchill, and Jeffrey Nichols. 2016. Understanding the Challenges of Designing and Developing Multi-Device Experiences. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS '16). ACM, New York, NY, USA, 62--72.
[4]
S. Eilemann, M. Makhinya, and R. Pajarola. 2009. Equalizer: A Scalable Parallel Rendering Framework. IEEE Transactions on Visualization and Computer Graphics 15, 3 (May 2009), 436--452.
[5]
Jacob Eisenstein, Jean Vanderdonckt, and Angel Puerta. 2001. Applying Model-based Techniques to the Development of UIs for Mobile Computers. In Proceedings of the 6th International Conference on Intelligent User Interfaces (IUI '01). ACM, New York, NY, USA, 69--76.
[6]
Niklas Elmqvist. 2011. Distributed User Interfaces: State of the Art. Springer London, London, 1--12.
[7]
Luca Frosini and Fabio Patern` o. 2014. User Interface Distribution in Multi-device and Multi-user Environments with Dynamically Migrating Engines. In Proceedings of the 2014 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '14). ACM, New York, NY, USA, 55--64.
[8]
Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically Generating User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, New York, NY, USA, 93--100.
[9]
Giuseppe Ghiani, Fabio Patern` o, and Carmen Santoro. 2010. On-demand Cross-device Interface Components Migration. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '10). ACM, New York, NY, USA, 299--308.
[10]
Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015. MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3933--3942.
[11]
Jens Grubert, Matthias Kranz, and Aaron Quigley. 2016. Challenges in mobile multi-device ecosystems. mUX: The Journal of Mobile User Experience 5, 1 (26 Aug 2016), 5.
[12]
Inc. Gurobi Optimization. 2016. Gurobi Optimizer Reference Manual. (2016). http://www.gurobi.com
[13]
Peter Hamilton and Daniel J. Wigdor. 2014. Conductor: Enabling and Understanding Cross-device Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2773--2782.
[14]
Richard Han, Veronique Perret, and Mahmoud Naghshineh. 2000. WebSplitter: A Unified XML Framework for Multi-device Collaborative Web Browsing. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). ACM, New York, NY, USA, 221--230.
[15]
Tommi Heikkinen, Jorge Goncalves, Vassilis Kostakos, Ivan Elhart, and Timo Ojala. 2014. Tandem Browsing Toolkit: Distributed Multi-Display Interfaces with Web Technologies. In Proceedings of The International Symposium on Pervasive Displays (PerDis '14). ACM, New York, NY, USA, Article 142, 6 pages.
[16]
Otmar Hilliges, Lucia Terrenghi, Sebastian Boring, David Kim, Hendrik Richter, and Andreas Butz. 2007. Designing for Collaborative Creative Problem Solving. In Proceedings of the 6th ACM SIGCHI Conference on Creativity &Amp; Cognition (C&C '07). ACM, New York, NY, USA, 137--146.
[17]
Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1247--1256.
[18]
Maria Husmann, Daniel Huguenin, Matthias Geel, and Moira C. Norrie. 2017. Orchestrating Multi-device Presentations with OmniPresent. In Proceedings of the 6th ACM International Symposium on Pervasive Displays (PerDis '17). ACM, New York, NY, USA, Article 3, 8 pages.
[19]
Tero Jokela, Jarno Ojala, and Thomas Olsson. 2015. A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3903--3912.
[20]
Andreas Karrenbauer and Antti Oulasvirta. 2014. Improvements to Keyboard Optimization with Integer Programming. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 621--626.
[21]
Clemens N. Klokmose, James R. Eagan, Siemen Baader, Wendy Mackay, and Michel Beaudouin-Lafon. 2015. Webstrates: Shareable Dynamic Media. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 280--290.
[22]
Chinmay Eishan Kulkarni and Scott R. Klemmer. 2011. Automatically Adapting Web Pages to Heterogeneous Devices. In CHI '11 Extended Abstracts on Human Factors in Computing Systems (CHI EA '11). ACM, New York, NY, USA, 1573--1578.
[23]
R. Langner, T. Horak, and R. Dachselt. 2017. VISTILES: Coordinating and Combining Co-located Mobile Devices for Visual Data Exploration. IEEE Transactions on Visualization and Computer Graphics PP, 99 (2017), 1--1.
[24]
Andr´ es Lucero, Jussi Holopainen, and Tero Jokela. 2011. Pass-them-around: Collaborative Use of Mobile Phones for Photo Sharing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1787--1796.
[25]
Andr´ es Lucero, Matt Jones, Tero Jokela, and Simon Robinson. 2013. Mobile Collocated Interactions: Taking an Offline Break Together. interactions 20, 2 (March 2013), 26--32.
[26]
K. Luyten and K. Coninx. 2005. Distributed user interface elements to support smart interaction spaces. In Seventh IEEE International Symposium on Multimedia (ISM'05). 8 pp.--.
[27]
Nicolai Marquardt, Ken Hinckley, and Saul Greenberg. 2012. Cross-device Interaction via Micro-mobility and F-formations. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 13--22.
[28]
J´ er´ emie Melchior, Donatien Grolaux, Jean Vanderdonckt, and Peter Van Roy. 2009. A Toolkit for Peer-to-peer Distributed User Interfaces: Concepts, Implementation, and Applications. In Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '09). ACM, New York, NY, USA, 69--78.
[29]
Jan Meskens, Jo Vermeulen, Kris Luyten, and Karin Coninx. 2008. Gummy for Multi-platform User Interface Designs: Shape Me, Multiply Me, Fix Me, Use Me. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 233--240.
[30]
Brad A. Myers, Herb Stiel, and Robert Gargiulo. 1998. Collaboration Using Multiple PDAs Connected to a PC. In Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work (CSCW '98). ACM, New York, NY, USA, 285--294.
[31]
Michael Nebeling. 2016. Cross-Device Interfaces: Existing Research, Current Tools, Outlook. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 513--516.
[32]
Michael Nebeling. 2017. XDBrowser 2.0: Semi-Automatic Generation of Cross-Device Interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4574--4584.
[33]
Michael Nebeling and Anind K. Dey. 2016. XDBrowser: User-Defined Cross-Device Web Page Designs. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5494--5505.
[34]
Michael Nebeling, Theano Mintsi, Maria Husmann, and Moira Norrie. 2014. Interactive Development of Cross-device User Interfaces. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2793--2802.
[35]
Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joseph Hughes, Thomas K. Harris, Roni Rosenfeld, and Mathilde Pignol. 2002. Generating Remote Control Interfaces for Complex Appliances. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (UIST '02). ACM, New York, NY, USA, 161--170.
[36]
Jeffrey Nichols, Brad A. Myers, and Kevin Litwack. 2004. Improving Automatic Interface Generation with Smart Templates. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, New York, NY, USA, 286--288.
[37]
Katie O'Leary, Tao Dong, Julia Katherine Haines, Michael Gilbert, Elizabeth F. Churchill, and Jeffrey Nichols. 2017. The Moving Context Kit: Designing for Context Shifts in Multi-Device Experiences. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS '17). ACM, New York, NY, USA, 309--320.
[38]
A. Oulasvirta. 2017. User Interface Design with Combinatorial Optimization. Computer 50, 1 (Jan 2017), 40--47.
[39]
Fabio Patern` o and Carmen Santoro. 2012. A Logical Framework for Multi-device User Interfaces. In Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '12). ACM, New York, NY, USA, 45--50.
[40]
Martin Porcheron, Andr´ es Lucero, and Joel E. Fischer. 2016. Co-curator: Designing for Mobile Ideation in Groups. In Proceedings of the 20th International Academic Mindtrek Conference (AcademicMindtrek '16). ACM, New York, NY, USA, 226--234.
[41]
Roman R¨ adle, Hans-Christian Jetter, Nicolai Marquardt, Harald Reiterer, and Yvonne Rogers. 2014. HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM, New York, NY, USA, 45--54.
[42]
Roman R¨ adle, Hans-Christian Jetter, Mario Schreiner, Zhihao Lu, Harald Reiterer, and Yvonne Rogers. 2015. Spatially-aware or Spatially-agnostic: Elicitation and Evaluation of User-Defined Cross-Device Interactions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3913--3922.
[43]
Roman R¨ adle, Midas Nouwens, Kristian Antonsen, James R Eagan, and Clemens Nylandsted Klokmose. 2017. Codestrates: Literate Computing with Webstrates. In Proceedings of the 30th annual ACM symposium on User interface software & technology (UIST '17). New York, NY, USA.
[44]
Jun Rekimoto. 1998. A Multiple Device Approach for Supporting Whiteboard-based Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '98). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 344--351.
[45]
Simon Robinson, Jennifer Pearson, Matt Jones, Anirudha Joshi, and Shashank Ahire. 2017. Better Together: Disaggregating Mobile Services for Emergent Users. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 44, 13 pages.
[46]
Stephanie Santosa and Daniel Wigdor. 2013. A Field Study of Multi-device Workflows in Distributed Workspaces. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 63--72.
[47]
Mario Schreiner, Roman R¨ adle, Hans-Christian Jetter, and Harald Reiterer. 2015. Connichiwa: A Framework for Cross-Device Web Applications. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15). ACM, New York, NY, USA, 2163--2168.
[48]
Julia Schwarz, David Klionsky, Chris Harrison, Paul Dietz, and Andrew Wilson. 2012. Phone As a Pixel: Enabling Ad-hoc, Large-scale Displays Using Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2235--2238.
[49]
Lucia Terrenghi, Aaron Quigley, and Alan Dix. 2009. A Taxonomy for and Analysis of Multi-person-display Ecosystems. Personal Ubiquitous Comput. 13, 8 (Nov. 2009), 583--598.
[50]
Minna W¨ aljas, Katarina Segersthl, Kaisa V¨ a¨ an¨ anen-Vainio-Mattila, and Harri Oinas-Kukkonen. 2010. Cross-platform Service User Experience: A Field Study and an Initial Framework. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '10). ACM, New York, NY, USA, 219--228.
[51]
Jishuo Yang and Daniel Wigdor. 2014. Panelrama: Enabling Easy Specification of Cross-device Web Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2783--2792.

Cited By

View all

Index Terms

  1. AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
    April 2018
    8489 pages
    ISBN:9781450356206
    DOI:10.1145/3173574
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 April 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. cross-device interaction
    2. distributed user interface
    3. optimization
    4. ui adaptation

    Qualifiers

    • Research-article

    Funding Sources

    • SNF UFO
    • ERC OPTINT
    • ERC Computed

    Conference

    CHI '18
    Sponsor:

    Acceptance Rates

    CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)122
    • Downloads (Last 6 weeks)12
    Reflects downloads up to 22 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)MARLUI: Multi-Agent Reinforcement Learning for Adaptive Point-and-Click UIsProceedings of the ACM on Human-Computer Interaction10.1145/36611478:EICS(1-27)Online publication date: 17-Jun-2024
    • (2024)The CoExplorer Technology Probe: A Generative AI-Powered Adaptive Interface to Support Intentionality in Planning and Running Video MeetingsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661507(1638-1657)Online publication date: 1-Jul-2024
    • (2024)FrameKit: A Tool for Authoring Adaptive UIs Using KeyframesProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645176(660-674)Online publication date: 18-Mar-2024
    • (2024)CoExplorer: Generative AI Powered 2D and 3D Adaptive Interfaces to Support Intentionality in Video MeetingsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650797(1-10)Online publication date: 11-May-2024
    • (2024)Semi-Automatic Layout Adaptation for Responsive Multiple-View Visualization DesignIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324035630:7(3798-3812)Online publication date: Jul-2024
    • (2024)AI Sensors and DashboardsComputer10.1109/MC.2024.339405657:8(55-64)Online publication date: 1-Aug-2024
    • (2024)The SPATIAL Architecture: Design and Development Experiences from Gauging and Monitoring the AI Inference Capabilities of Modern Applications2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00092(947-959)Online publication date: 23-Jul-2024
    • (2024)Empowering Collaborative Application Development: A Robust Framework for Ad-Hoc Distributed SystemsIEEE Access10.1109/ACCESS.2024.339537012(62296-62316)Online publication date: 2024
    • (2023)BlendMR: A Computational Method to Create Ambient Mixed Reality InterfacesProceedings of the ACM on Human-Computer Interaction10.1145/36264727:ISS(217-241)Online publication date: 1-Nov-2023
    • (2023)WAMS: A Flexible API for Visual Workspaces Across Multiple SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932367:EICS(1-40)Online publication date: 19-Jun-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media