Article

Free access

Integration and synchronization of input modes during multimodal human-computer interaction

Authors:

Sharon Oviatt,

Antonella DeAngeli,

Karen KuhnAuthors Info & Claims

CHI '97: Proceedings of the ACM SIGCHI Conference on Human factors in computing systems

Pages 415 - 422

https://doi.org/10.1145/258549.258821

Published: 27 March 1997 Publication History

PDF eReader

References

[1]

Bolt, R. "Put-That-There": Voice and Gesture at the hics interface, Computer Graphics, 1980, 14 (3): 270.

Digital Library

Google Scholar

[2]

Cohen, P., Dalrymple, M., Moran, D. & Pereira, F. Synergistic use of direct manipulation and natural language, CHI 89 Conf. Proc., ACM: Addison Wesley, New York, 1989, 227-234.

Digital Library

Google Scholar

[3]

Kendon, A. Gesticulation and speech: Two aspects of the process of utterance, The Relationship of Verbal and Nonverbal Communication (ed. by M. Key), The Hague: Mouton, 1980, 207-227.

Crossref

Google Scholar

[4]

Koons, D., Sparrell, C. & Thorisson, K. Integrating simultaneous input from speech, gaze, and hand gestures, Intelligent Multimedia Interfaces, ed. by M. Maybury, MIT Press: Cambridge, MA, 1993, 257-76.

Digital Library

Google Scholar

[5]

Levelt, W., Richardson, G. & Heu, W. Pointing and voicing in deictic expressions, Jour. of Memory and Language, 1985, 24, 133-164.

Crossref

Google Scholar

[6]

McNeill, D. Hand and Mind: What gestures reveal about thought, Univ. of Chicago Press: Chicago, II1., 1992.

Google Scholar

[7]

McNeill, D. Language as gesture (Gesture as language), Proc. of the Workshop on the Integration of Gesture in Language & Speech, ed. by L. Messing, Univ. of Delaware, Oct. 1996, 1-20.

Google Scholar

[8]

Naughton, K. Spontaneous gesture and sign: A study of ASL signs co-occurring with speech, Proc. of the Workshop on the Integration of Gesture in Language & Speech, ed. by L. Messing, Univ. of Delaware, Oct. 1996, 125-34.

Google Scholar

[9]

Neal, J. & Shapiro, S. Intelligent multi-media interface technology, in Intelligent User Interfaces (J. Sullivan & S. Tyler, eds.), ACM: Addison Wesley, New York, 1991, ch. 3, 45-68.

Digital Library

Google Scholar

[10]

Oviatt, S.L. Multimodal interfaces for dynamic interactive maps, CHI '96 Conf Proc., New York, ACM Press, 1996, 95-102.

Digital Library

Google Scholar

[11]

Oviatt, S., Cohen, P., Fong, M., & Frank, M. A rapid semi-automatic simulation technique for investigating interactive speech and handwriting, Proc. of the Intl. Conf. on Spoken Language Processing, 1992, 2, 1351-54.

Google Scholar

[12]

Oviatt, S., Cohen, P. Johnston, M. & Kuhn, K. Multimodal language: Linguistic features and processing requirements, forthcoming.

Google Scholar

[13]

Oviatt, S., Cohen, P. & Wang, M. Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity, Speech Communication, 1994,15 (3-4), 283-300.

Digital Library

Google Scholar

[14]

Oviatt, S. & Olsen, E. Integration themes in multimodal human-computer interaction, Proc. of the lntl Conf. on Spoken Language Processing, 1994, 2, 551-554.

Google Scholar

[15]

Oviatt, S. L. & vanGent, R. Error resolution during multimodal human-computer interaction, Proc. of the Intl. Conf. on Spoken Language Processing, 1996.

Crossref

Google Scholar

[16]

Pittman, J., Cohen, P., Smith, I., Yang, T. & Oviatt, S. Quickset: A multimodal interface for distributed interactive simulations, Proc. of the 6th Conf. on Computer-Generated Forces & Behavior Representation, Univ. of Central Florida, Orlando, FL., 1996, 217-24.

Google Scholar

Cited By

View all

Rodriguez RSullivan BBarrera Machuca MBatmaz ATornatzky COrtega F(2024)An Artists' Perspectives on Natural Interactions for Virtual Reality 3D SketchingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642758(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642758
Coutaz J(2024)At the Confluence of Software Engineeringand Human-Computer Interaction:A Personal AccountThe French School of Programming10.1007/978-3-031-34518-0_5(89-122)Online publication date: 30-Apr-2024
https://doi.org/10.1007/978-3-031-34518-0_5
Liang PLyu YChhablani GJain NDeng ZWang XMorency LSalakhutdinov R(2023)MultiViz: Towards User-Centric Visualizations and Interpretations of Multimodal ModelsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585604(1-21)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585604
Show More Cited By

Index Terms

Integration and synchronization of input modes during multimodal human-computer interaction
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

Integration and synchronization of input modes during multimodal human-computer interaction
ReferringPhenomena '97: Referring Phenomena in a Multimedia Context and their Computational Treatment

Our ability to develop robust multimodal systems will depend on knowledge of the natural integration patterns that typify people's combined use of different input modes. To provide a foundation for theory and design, the present research analyzed ...
Toward a theory of organized multimodal integration patterns during human-computer interaction
ICMI '03: Proceedings of the 5th international conference on Multimodal interfaces

As a new generation of multimodal systems begins to emerge, one dominant theme will be the integration and synchronization requirements for combining modalities into robust whole systems. In the present research, quantitative modeling is presented on ...
A mobile multimodal dialogue system for public transportation navigation evaluated
MobileHCI '06: Proceedings of the 8th conference on Human-computer interaction with mobile devices and services

As the technical capabilities of latest mobile devices are combined with mobile broadband internet access, we are ready to make use of free and natural speech in mobile services by utilizing optional and complementary means of input. In these kinds of ...

Comments

Information & Contributors

Information

Published In

CHI '97: Proceedings of the ACM SIGCHI Conference on Human factors in computing systems

March 1997

565 pages

ISBN:0897918029

DOI:10.1145/258549

Co-chair:
Steven Pemberton

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 March 1997

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CHI97

Sponsor:

SIGCHI

CHI97: ACM Conference on Human Factors & Computing Systems

March 22 - 27, 1997

Georgia, Atlanta, USA

Acceptance Rates

CHI '97 Paper Acceptance Rate 55 of 234 submissions, 24%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

180
Total Citations
View Citations
2,091
Total Downloads

Downloads (Last 12 months)302
Downloads (Last 6 weeks)24

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Rodriguez RSullivan BBarrera Machuca MBatmaz ATornatzky COrtega F(2024)An Artists' Perspectives on Natural Interactions for Virtual Reality 3D SketchingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642758(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642758
Coutaz J(2024)At the Confluence of Software Engineeringand Human-Computer Interaction:A Personal AccountThe French School of Programming10.1007/978-3-031-34518-0_5(89-122)Online publication date: 30-Apr-2024
https://doi.org/10.1007/978-3-031-34518-0_5
Liang PLyu YChhablani GJain NDeng ZWang XMorency LSalakhutdinov R(2023)MultiViz: Towards User-Centric Visualizations and Interpretations of Multimodal ModelsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585604(1-21)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585604
Mich OSchiavo GFerron MMana N(2022)Framing the Design Space of Multimodal Mid-Air Gesture and Speech-Based Interaction With Mobile Devices for Older PeopleResearch Anthology on Supporting Healthy Aging in a Digital Society10.4018/978-1-6684-5295-0.ch028(496-519)Online publication date: 4-Feb-2022
https://doi.org/10.4018/978-1-6684-5295-0.ch028
Mahi MRakib MJoy MHridoy MRupai ANabil R(2022)A Proposed Design of a Lecture Material to Reduce Learning ComplexityProceedings of the 2nd International Conference on Computing Advancements10.1145/3542954.3542986(213-219)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1145/3542954.3542986
Molina León GLischka MLuo WBreiter A(2022)Mobile and Multimodal? A Comparative Evaluation of Interactive Workplaces for Visual Data ExplorationComputer Graphics Forum10.1111/cgf.1455141:3(417-428)Online publication date: 12-Aug-2022
https://doi.org/10.1111/cgf.14551
Oviatt S(2022)Multimodal Interaction, Interfaces, and AnalyticsHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_22-1(1-29)Online publication date: 1-Apr-2022
https://doi.org/10.1007/978-3-319-27648-9_22-1
Srinivasan ALee BStasko J(2021)Interweaving Multimodal Interaction With Flexible Unit Visualizations for Data ExplorationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.297805027:8(3519-3533)Online publication date: 1-Aug-2021
https://doi.org/10.1109/TVCG.2020.2978050
Marge MEspy-Wilson CWard NAlwan AArtzi YBansal MBlankenship GChai JDaumé HDey DHarper MHoward TKennington CKruijff-Korbayová IManocha DMatuszek CMead RMooney RMoore ROstendorf MPon-Barry HRudnicky AScheutz MAmant RSun TTellex STraum DYu Z(2021)Spoken language interaction with robots: Recommendations for future researchComputer Speech & Language10.1016/j.csl.2021.101255(101255)Online publication date: Jul-2021
https://doi.org/10.1016/j.csl.2021.101255
Wang INarayana PPatil DBangar RDraper BBeveridge RRuiz J(2021)It’s a Joint Effort: Understanding Speech and Gesture in Collaborative TasksHuman-Computer Interaction. Interaction Techniques and Novel Applications10.1007/978-3-030-78465-2_13(159-178)Online publication date: 3-Jul-2021
https://doi.org/10.1007/978-3-030-78465-2_13
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Integration and synchronization of input modes during multimodal human-computer interaction

Toward a theory of organized multimodal integration patterns during human-computer interaction

A mobile multimodal dialogue system for public transportation navigation evaluated