Article

Staging transformations for multimodal web interaction management

Authors:

Michael Narayan,

Christopher Williams,

Saverio Perugini,

Naren RamakrishnanAuthors Info & Claims

WWW '04: Proceedings of the 13th international conference on World Wide Web

Pages 212 - 223

https://doi.org/10.1145/988672.988702

Published: 17 May 2004 Publication History

Abstract

Multimodal interfaces are becoming increasingly ubiquitous with the advent of mobile devices, accessibility considerations, and novel software technologies that combine diverse interaction media. In addition to improving access and delivery capabilities, such interfaces enable flexible and personalized dialogs with websites, much like a conversation between humans. In this paper, we present a software framework for multimodal web interaction management that supports mixed-initiative dialogs between users and websites. A mixed-initiative dialog is one where the user and the website take turns changing the flow of interaction. The framework supports the functional specification and realization of such dialogs using staging transformations -- a theory for representing and reasoning about dialogs based on partial input. It supports multiple interaction interfaces, and offers sessioning, caching, and co-ordination functions through the use of an interaction manager. Two case studies are presented to illustrate the promise of this approach.

References

[1]

Speech Application Language Tags (SALT) Specification. Technical report, SALT Forum, July 2002. Version 1.0.

[2]

J. F. Allen, C. I. Guinn, and E. Horvitz. Mixed-Initiative Interaction. IEEE Intelligent Systems, Vol. 14(5):pages 14--23, September--October 1999.

Digital Library

[3]

E. André and T. Rist. From Adaptive Hypertext to Personalized Web Companions. CACM, Vol. 45(5):pages 43--46, May 2002.

Digital Library

[4]

Y. Aridor, D. Carmel, Y. S. Maarek, A. Soffer, and R. Lempel. Knowledge Encapsulation for Focused Search from Pervasive Devices. In Proc. WWW10, pages 754--764, 2001.

Digital Library

[5]

D. L. Atkins, T. Ball, G. Bruns, and K. Cox. Mawl: A Domain-Specific Language for Form-Based Services. IEEE Transactions on Software Engineering, Vol. 25(3):pages 334--346, May--June 1999.

Digital Library

[6]

J. Axelsson, C. Cross, H. Lie, G. McCobb, T. Raman, and L. Wilson (eds.). Xhtml+voice profile 1.0. W3C Note, December 2001.

[7]

N. J. Belkin, C. Cool, A Stein, and U. Thiel. Cases, Scripts, and Information Seeking Strategies: On the Design of Interactive Information Retrieval Systems. Expert Systems with Applications, Vol. 9(3):pages 379--395, 1995.

[8]

D. W. Binkley and K. B. Gallagher. Program Slicing. In Advances in Computers, volume 43, pages 1--50. 1996.

[9]

P. De Bra, P. Brusilovsky, and G.-J. Houben. Adaptive Hypermedia: From Systems to Framework. ACM Computing Surveys, Vol. 31(4es), December 1999. Article No. 12.

Digital Library

[10]

C. Brabrand, A. Møller, and M. I. Schwartzbach. The <bigwig> Project. ACM Transactions on Internet Technology, Vol. 2(2):pages 79--14, May 2002.

Digital Library

[11]

P. Brusilovsky. Adaptive Hypermedia. User Modeling and User-Adapted Interaction, Vol. 11(1--2):pages 87--110, 2001.

Digital Library

[12]

O. Buyukkokten, H. Garcia-Molina, and A. Paepcke. Seeing the Whole in Parts: Text Summarization for Web Browsing on Handheld Devices. In Proc. WWW10, pages 652--662, 2001.

Digital Library

[13]

R. Capra, M. Narayan, S. Perugini, N. Ramakrishnan, and M. A. P 233;rez-Qui 241;ones. The Staging Transformation Approach to Mixing Initiative. In G. Tecuci, editor, Working Notes of the IJCAI 2003 Workshop on Mixed-Initiative Intelligent Systems, pages 23--29. AAAI/MIT Press, August 2003.

[14]

R. Capra, M. A. Pérez-Quiñones, and N. Ramakrishnan. WebContext: Remote Access to Shared Context. In Proc. PUI, November 2001.

Digital Library

[15]

S. K. Card, T. P. Moran, and A. Newell. Computer Text-Editing: An Information-Processing Analysis of a Routine Cognitive Skill. Cognitive Psychology, Vol. 12:pages 32--74, 1980.

[16]

J. Chen, B. Zhou, J. Shi, H. Zhang, and Q. Fengwu. Function-Based Object Model Towards Website Adaptation. In Proc. WWW10, pages 587--596, 2001.

Digital Library

[17]

Y. Chen, W.-Y. Ma, and H.-J. Zhang. Detecting Web Page Structure for Adaptive Viewing on Small Form Factor Devices. In Proc. WWW12, pages 225--233, 2003.

Digital Library

[18]

D. Cohen, M. Herscovici, Y. Petruschka, Y. S. Maarek, and A. Soffer. Personalized Pocket Directories for Mobile Devices. In Proc. WWW11, pages 627--638, 2002.

Digital Library

[19]

A. Coles, E. Deliot, T. Melamed, and K. Lansard. A Framework for Coordinated Multi-Modal Browsing with Multiple Clients. In Proc. WWW12, pages 718--726, 2003.

Digital Library

[20]

S. Dumais. Tightly Coupling Structure and Search. In Proc. SIGIR Workshop on Information Reduction, July 1997.

[21]

J. Freire, B. Kumar, and D. Lieuwen. WebViews: Accessing Personalized Web Content and Services. In Proc. WWW10, pages 576--586, 2001.

Digital Library

[22]

P. Graunke, R. Findler, S. Krishnamurthi, and M. Felleisen. Automatically Restructuring Programs for the Web. In Proc. ASE, November 2001.

Digital Library

[23]

B. J. Grosz and C. L. Sidner. Attention, Intentions, and the Structure of Discourse. Computational Linguistics, Vol. 12:pages 175--204, 1986.

Digital Library

[24]

M. A. Hearst, A. Elliott, J. English, R. Sinha, K. Swearingen, and K.-P. Yee. Finding the Flow in Web Site Search. CACM, Vol. 45(9):pages 42--49, September 2002.

Digital Library

[25]

S. Horwitz, T. Reps, and D. W. Binkley. Interprocedural Slicing Using Dependency Graphs. ACM Transactions on Programming Languages and Systems, Vol. 12(1):pages 26--60, January 1990.

Digital Library

[26]

N. D. Jones. An Introduction to Partial Evaluation. ACM Computing Surveys, Vol. 28(3):pages 480--503, September 1996.

Digital Library

[27]

I.-Y. Ko, K.-T. Yao, and R. Neches. Dynamic Coordination of Information Management Services for Processing Dynamic Web Content. In Proc. WWW11, pages 355--365, 2002.

Digital Library

[28]

J. Lai. Conversation Interfaces. CACM, Vol. 43(9):pages 24--27, September 2000.

Digital Library

[29]

G. Marchionini. Information Seeking in Electronic Environments. Cambridge Series on Human-Computer Interaction. Cambridge University Press, 1997.

Digital Library

[30]

S. McGlashan, D. Burnett, P. Danielsen, J. Ferrans, A. Hunt, G. Karam, D. Ladd, B. Lucas, B. Porter, K. Rehor, and S. Tryphonas. Voice eXtensible Markup Language: VoiceXML. Technical report, VoiceXML Forum, October 2001. Version 2.0.

[31]

B. Mobashier, R. Cooley, and J. Srivastava. Automatic Personalization Based on Web Usage Mining. CACM, Vol. 43(8):pages 142--151, August 2000.

Digital Library

[32]

B. A. Myers and M. Beigl. Guest Editors' Introduction: Handheld Computing. IEEE Computer, Vol. 36(9):pages 27--29, September 2003.

Digital Library

[33]

S. Perugini, M. E. Pinney, N. Ramakrishnan, M. A. Pérez-Quiñones, and M. B. Rosson. Taking the Initiative with Extempore: Exploring Out-of-Turn Interactions with Websites. Technical Report cs.HC/0312016, Computing Research Repository (CoRR), December 2003.

[34]

S. Perugini and N. Ramakrishnan. Personalizing Interactions with Information Systems. In Advances in Computers, volume 57: Information Repositories, pages 323--382. September 2003.

[35]

S. Perugini and N. Ramakrishnan. Personalizing Web Sites with Mixed-Initiative Interaction. IEEE IT Professional, Vol. 5(2):pages 9--15, March--April 2003.

Digital Library

[36]

J. Pokorny. Static Pages are Dead: How a Modular Approach is Changing Interaction Design. ACM Interactions, Vol. 8(5):pages 19--24, September--October 2001.

Digital Library

[37]

D. Quan, D. Huynh, D. R. Karger, and R. Miller. User Interface Continuations. In Sixteenth ACM Symposium on User Interface Software and Technology (UIST), November 2003.

Digital Library

[38]

C. Queinnec. The Influence of Browsers on Evaluators or, Continuations to ProgramWeb Servers. In Proc. ICFP, pages 23--33, September 2000.

Digital Library

[39]

S. Rollins and N. Sundaresan. AVoN Calling: AXL for Voice-Enabled Web Navigation. In Proc. WWW9, 2000.

Digital Library

[40]

G. M. Sacco. Dynamic Taxonomies: A Model for Large Information Bases. IEEE Transactions on Knowledge and Data Engineering, Vol. 12(3):pages 468--479, May--June 2000.

Digital Library

[41]

S. Srinivasan and E. Brown. Is Speech Recognition Becoming Mainstream? IEEE Computer, Vol. 35(4):pages 38--41, April 2002.

Digital Library

[42]

J. Steinberg and J. Pasquale. A Web Middleware Architecture for Dynamic Customization of Content for Wireless Clients. In Proc. WWW11, pages 639--650, 2002.

Digital Library

[43]

J. Veitch. A Conversation with Paul Graham. CACM, Vol. 41(5):pages 52--54, May 1998.

Digital Library

[44]

K. Wang. A Study of Semantic Synchronous Understanding on Speech Interface Design. In Proc. UIST'03, November 2003.

[45]

T. Winograd and F. Flores, editors. Understanding Computers and Cognition -- A New Foundation for Design. Addison-Wesley, Reading, PA, 1987.

Digital Library

[46]

S. A. Wolfman, T. Lau, P. Domingos, and D. S. Weld. Mixed Initiative Interfaces for Learning Tasks: SMARTedit Talks Back. In Proc. IUI, pages 167--174, 2001.

Digital Library

[47]

N. Yankelovich. How Do Users Know What To Say? ACM Interactions, 3(6):pages 32--43, November--December 1996.

Digital Library

Cited By

Perugini SBuck JLuyten KPalanque PCampos JSchmidt A(2016)A language-based model for specifying and staging mixed-initiative dialogsProceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/2933242.2933262(204-216)Online publication date: 21-Jun-2016
https://dl.acm.org/doi/10.1145/2933242.2933262
Yeh TWhite BSan Pedro JKatz BDavis LSadagopan SRamamritham KKumar ARavindra MBertino EKumar R(2011)A case for query by image and text contentProceedings of the 20th international conference on World wide web10.1145/1963405.1963513(775-784)Online publication date: 28-Mar-2011
https://dl.acm.org/doi/10.1145/1963405.1963513
Gatius MGonzalez M(2011)Using domain knowledge for fostering the collaborative ability of a web dialogue system2011 7th International Conference on Next Generation Web Services Practices10.1109/NWeSP.2011.6088165(129-134)Online publication date: Oct-2011
https://doi.org/10.1109/NWeSP.2011.6088165
Show More Cited By

Index Terms

Recommendations

A study of out-of-turn interaction in menu-based, IVR, voicemail systems
CHI '07: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We present the first user study of out-of-turn interaction inmenu-based, interactive voice-response systems. Out-of-turn interaction is atechnique which empowers the user (unable to respond to the current prompt) totake the conversational initiative by ...
A language-based model for specifying and staging mixed-initiative dialogs
EICS '16: Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems

Specifying and implementing flexible human-computer dialogs, such as those used in kiosks, is complex because of the numerous and varied directions in which each user might steer a dialog. The objective of this research is to improve dialog ...
Natural Language, Mixed-initiative Personal Assistant Agents
IMCOM '18: Proceedings of the 12th International Conference on Ubiquitous Information Management and Communication

The increasing popularity and use of personal voice assistant technologies, such as Siri and Google Now, is driving and expanding progress toward the long-term and lofty goal of using artificial intelligence to build human-computer dialog systems ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '04: Proceedings of the 13th international conference on World Wide Web

May 2004

754 pages

ISBN:158113844X

DOI:10.1145/988672

Conference Chairs:
Stuart Feldman
IBM Research
,
Mike Uretsky
New York University
,
Program Chairs:
Marc Najork
Microsoft Research
,
Craig Wills
Worcester Polytechnic Institute

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 May 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

WWW04

Sponsor:

ACM

WWW04: The 2004 World Wide Web Conference ( in conjunction with ACM Conference on Electronic Commerce [EC'04])

May 17 - 20, 2004

NY, New York, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
950
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Perugini SBuck JLuyten KPalanque PCampos JSchmidt A(2016)A language-based model for specifying and staging mixed-initiative dialogsProceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/2933242.2933262(204-216)Online publication date: 21-Jun-2016
https://dl.acm.org/doi/10.1145/2933242.2933262
Yeh TWhite BSan Pedro JKatz BDavis LSadagopan SRamamritham KKumar ARavindra MBertino EKumar R(2011)A case for query by image and text contentProceedings of the 20th international conference on World wide web10.1145/1963405.1963513(775-784)Online publication date: 28-Mar-2011
https://dl.acm.org/doi/10.1145/1963405.1963513
Gatius MGonzalez M(2011)Using domain knowledge for fostering the collaborative ability of a web dialogue system2011 7th International Conference on Next Generation Web Services Practices10.1109/NWeSP.2011.6088165(129-134)Online publication date: Oct-2011
https://doi.org/10.1109/NWeSP.2011.6088165
Perugini S(2010)Supporting multiple paths to objects in information hierarchiesInformation Processing and Management: an International Journal10.1016/j.ipm.2009.06.00746:1(22-43)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.1016/j.ipm.2009.06.007
Perugini SRamakrishnan N(2010)Program transformations for information personalizationComputer Languages, Systems and Structures10.1016/j.cl.2009.09.00236:3(223-249)Online publication date: 1-Oct-2010
https://dl.acm.org/doi/10.1016/j.cl.2009.09.002
Perugini S(2008)Symbolic links in the Open Directory ProjectInformation Processing and Management: an International Journal10.5555/1347449.134753244:2(910-930)Online publication date: 1-Mar-2008
https://dl.acm.org/doi/10.5555/1347449.1347532
Antonacopoulos AHu J(2007)Web Document AnalysisDigital Document Processing10.1007/978-1-84628-726-8_18(407-419)Online publication date: 2007
https://doi.org/10.1007/978-1-84628-726-8_18
Perugini SRamakrishnan N(2007)Mining Web functional dependencies for flexible information accessJournal of the American Society for Information Science and Technology10.1002/asi.2062858:12(1805-1819)Online publication date: 6-Sep-2007
https://doi.org/10.1002/asi.20628
Perugini SRamakrishnan N(2006)Interacting with Web HierarchiesIT Professional10.1109/MITP.2006.918:4(19-28)Online publication date: 1-Jul-2006
https://dl.acm.org/doi/10.1109/MITP.2006.91
Perugini SRamakrishnan N(2005)A generative programming approach to interactive information retrievalProceedings of the 4th international conference on Generative Programming and Component Engineering10.1007/11561347_15(205-220)Online publication date: 29-Sep-2005
https://dl.acm.org/doi/10.1007/11561347_15
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents