Article

Towards a computational model of sketching

Authors:

Kenneth D. Forbus,

Ronald W. Ferguson,

Jeffery M. UsherAuthors Info & Claims

IUI '01: Proceedings of the 6th international conference on Intelligent user interfaces

Pages 77 - 83

https://doi.org/10.1145/359784.360278

Published: 01 January 2001 Publication History

Abstract

Sketching is a powerful means of interpersonal communication. While many useful multimodal systems have been created, current systems are far from achieving human-like participation in sketching. A computational model of sketching would help characterize these differences and help us better understand how to overcome them. This paper is a first step towards such a model. We start with an example of a sketching system(nuSketch COA Creator)designed to aid military planners, to provide context and a source of examples. We then describe four dimensions of sketching,visual understanding, conceptual understanding, language understanding,anddrawing,that can be used to characterize the competence of existing systems and identify open problems. The issues involved will be illustrated by examples from our experience with nuSketch. Three research challenges are posed, to serve as milestones towards a computational model of sketching that can explain and replicate human abilities in this area.

References

[1]

Allen, J.F. et al, The TRAINS Project: A Case Study in Defining a Conversational Planning Agent, Journal of Experimental and Theoretical AI, 1995.

[2]

Bolt, R. A. (1980) Put-That-There: Voice and gesture at the graphics interface. Computer Graphics. 14(3), 262-270.

Digital Library

[3]

Clark, H. H. 1996. Using language. Cambridge University Press.

[4]

Clark, H.H. 1999. Speaking in Time. Proceedings of the ESCA workshop on dialogue and prosody, September 1-3, Veldhoven, the Netherlands.

[5]

Cohen, P. (1992) The role of natural language in a multimodal interface. UIST92, pp 143-149.

Digital Library

[6]

Cohen, P., Dalrymple, M., Moran, D., Pereira, F., Sullivan, J., Gargan, R., Schlossberg, J. and Tyler, S. (1989) Synergistic use of direct manipulation and natural language. Proceedings of CHI-89, pp 227-233.

Digital Library

[7]

Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. (1997). QuickSet: Multimodal interaction for distributed applications, Proceedings of the Fifth Annual International Multimodal Conference (Multimedia '97), (Seattle, WA, November 1997), ACM Press, pp 31-40.

Digital Library

[8]

Cohn, A. (1996) Calculi for Qualitative Spatial Reasoning. In Artificial Intelligence and Symbolic Mathematical Computation, LNCS 1138, eds: J Calmet, J A Campbell, J Pfalzgraf, Springer Verlag, 124-143, 1996.

Digital Library

[9]

Egenhofer, M. (1997) Query Processing in Spatial-Query-by-Sketch in Journal of Visual Languages and Computing 8(4), 403-424 pp.

[10]

Falkenhainer, B., Forbus, K., Gentner, D. (1989) The Structure-Mapping Engine: Algorithm and examples. Artificial Intelligence, 41, pp 1-63.

Digital Library

[11]

Ferguson, R.W. and Forbus, K.D. 2000. GeoRep: A Flexible Tool for Spatial Representation of Line Drawings. Proceedings of AAAI-2000. Austin, Texas.

Digital Library

[12]

Ferguson, R.W., Rasch, R.A., Turmel, W., & Forbus, K.D. (2000) Qualitative Spatial Interpretation of Course-of-Action Diagrams. Proceedings of the 14th International Workshop on Qualitative Reasoning. Morelia, Mexico. June, 2000.

Digital Library

[13]

Forbus, K. 1995. Qualitative Spatial Reasoning: Framework and Frontiers. In Glasgow, J., Narayanan, N., and Chandrasekaran, B. Diagrammatic Reasoning: Cognitive and Computational Perspectives. MIT Press, pp. 183-202.

[14]

Forbus, K., Ferguson, R. and Gentner, D. (1994) Incremental structure-mapping. Proceedings of the Cognitive Science Society, August.

[15]

Forbus, K., Nielsen, P. and Faltings, B. "Qualitative Spatial Reasoning: The CLOCK Project", Artificial Intelligence, 51 (1-3), October, 1991.

Digital Library

[16]

Gross, M. (1996) The Electronic Cocktail Napkin - computer support for working with diagrams. Design Studies. 17(1), 53- 70.

[17]

Gross, M. and Do, E. (1995) Drawing Analogies - Supporting Creative Architectural Design with Visual References. in 3d International Conference on Computational Models of Creative Design, M-L Maher and J. Gero (eds), Sydney: University of Sydney, 37-58.

[18]

Grosz, B.J., and Sidner, C.L., "Attention, Intentions, and the Structure of Discourse", Computational Linguistics, 12:3 (1986)

Digital Library

[19]

Henderson, K. 1999. On Line and On Paper: Visual representations, visual culture, and computer graphics in design engineering. MIT Press.

Digital Library

[20]

Julia, L. and Faure, C. 1995. Pattern Recogntion and Beautification for a pen-based interface. Proceedings of ICDAR'95: Montreal (Canada), pp 58-63.

Digital Library

[21]

Landay, J. and Myers, B. 1996. Sketching storyboards to illustrate interface behaviors. CHI'96 Conference Companion: Human Factors in Computing Systems, Vancouver, Canada.

Digital Library

[22]

Lakin, F. 1986. Spatial parsing for visual languages. In Chang, S., Ichikawa, T. and Ligomendes, P. (Eds.) Visual Languages. Plenum Press, NY.

[23]

Luperfoy, S. (editor) Automated Spoken Dialogue Systems. MIT Press (in preparation).

[24]

Maybury, M. and Whalster, W. 1998. Readings in Intelligent User Interfaces. Morgan Kaufmann.

Digital Library

[25]

McCloud, Scott. 1993. Understanding Comics: The Invisible Art. Kitchen Sink Press, NY.

[26]

Moran, D.B., Cheyer, A.J., Julia, L.E., Martin, D.L., and Park, S. (1997) Multimodal user interfaces in the Open Agent Architecture. Proceedings of IUI97. pp 61-68.

Digital Library

[27]

Oviatt, S. L. 1999. Ten myths of multimodal interaction, Communications of the ACM, Vol. 42, No. 11, November, 1999, pp. 74-81.

Digital Library

[28]

Saund, E., and Moran, T, (1995) Perceptual Organization in an Interactive Sketch Editing Application. ICCV '95

Digital Library

[29]

Stahovich, T. F., Davis, R., and Shrobe, H., "Generating Multiple New Designs from a Sketch," in Proceedings Thirteenth National Conference on Artificial Intelligence, AAAI-96, pp. 1022-29, 1996.

Digital Library

[30]

Waibel, A., Suhm, B., Vo, M. and Yang, J. 1996. Multimodal interfaces for multimedia information agents. Proc. of ICASSP 97.

Digital Library

Cited By

Pourjafarian NMjaku FKoelle MSchmitz MBorchers JSteimle J(2023)Handheld Tools Unleashed: Mixed-Initiative Physical Sketching with a Robotic PrinterProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580691(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580691
Brathen KSeehuus RMevassvik O(2022)Simulation Support to Wargaming for Tactical Operations PlanningSimulation and Wargaming10.1002/9781119604815.ch9(223-247)Online publication date: 7-Jan-2022
https://doi.org/10.1002/9781119604815.ch9
Abdalla SRashid MAra D(2021)Plausibility of CAAD in Conceptual Design: Challenges in Architectural Engineering for Early-Stage Digital Design ToolsJournal of Architectural Engineering10.1061/(ASCE)AE.1943-5568.000045727:2Online publication date: Jun-2021
https://doi.org/10.1061/(ASCE)AE.1943-5568.0000457
Show More Cited By

Index Terms

Towards a computational model of sketching

Recommendations

Sketching user experiences tutorial: stories, strategies, surfaces
ITS '13: Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces

Paper-pencil sketches are a valuable tool during different stages of experience design in human-computer interaction. This hands-on tutorial will demonstrate how to integrate sketching into researchers' and interaction designers' everyday practice -- ...
Hands-on sketching course
CHI EA '14: CHI '14 Extended Abstracts on Human Factors in Computing Systems

Sketching as a technique to quickly draw something on a piece of paper is used to explore and communicate ideas. This course introduces basic sketching techniques and a visual language which participants can apply in research and practice. It is a hands-...
Sketching interactive systems with sketchify
INTERACT'11: Proceedings of the 13th IFIP TC 13 international conference on Human-computer interaction - Volume Part IV

Sketching is at the heart of design and creativity, an omnipresent element of any disciplined activity of design. In this tutorial we will summarize many of the existing studies of sketching, and emphasize its role in supporting creativity. We will look ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '01: Proceedings of the 6th international conference on Intelligent user interfaces

January 2001

174 pages

ISBN:1581133251

DOI:10.1145/359784

Chairmen:
Candy Sidner
MERL
,
Johanna Moore
Univ. of Edinburgh, Edinburgh, Scotland

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGCHI: ACM Special Interest Group on Computer-Human Interaction
AAAI: Am Assoc for Artifical Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

IUI01

Sponsor:

IUI01: International Conference on Intelligent User Interfaces 2001

January 14 - 17, 2001

New Mexico, Santa Fe, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
860
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pourjafarian NMjaku FKoelle MSchmitz MBorchers JSteimle J(2023)Handheld Tools Unleashed: Mixed-Initiative Physical Sketching with a Robotic PrinterProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580691(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580691
Brathen KSeehuus RMevassvik O(2022)Simulation Support to Wargaming for Tactical Operations PlanningSimulation and Wargaming10.1002/9781119604815.ch9(223-247)Online publication date: 7-Jan-2022
https://doi.org/10.1002/9781119604815.ch9
Abdalla SRashid MAra D(2021)Plausibility of CAAD in Conceptual Design: Challenges in Architectural Engineering for Early-Stage Digital Design ToolsJournal of Architectural Engineering10.1061/(ASCE)AE.1943-5568.000045727:2Online publication date: Jun-2021
https://doi.org/10.1061/(ASCE)AE.1943-5568.0000457
Chandrasegaran SRamanujan DElmqvist NKoskinen ILim YCerratto-Pargman TChow KOdom W(2018)How Do Sketching and Non-Sketching Actions Convey Design Intent?Proceedings of the 2018 Designing Interactive Systems Conference10.1145/3196709.3196723(373-385)Online publication date: 8-Jun-2018
https://dl.acm.org/doi/10.1145/3196709.3196723
Kang BLaViola Jr. JWisniewski PMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)Structured Input Improves Usability and Precision for Solving Geometry-based Algebraic ProblemsProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3025468(4692-4702)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3025468
Kang BKulshreshth ALaViola JNichols JMahmud JO'Donovan JConati CZancanaro M(2016)AnalyticalInkProceedings of the 21st International Conference on Intelligent User Interfaces10.1145/2856767.2856789(419-430)Online publication date: 7-Mar-2016
https://dl.acm.org/doi/10.1145/2856767.2856789
Lee WStahovich T(2014)PaperCADAdvances in Human-Computer Interaction10.1155/2014/9086902014(14-14)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1155/2014/908690
Richter KWinter SRichter KWinter S(2014)Communication Aspects: How Landmarks Enrich the Communication Between Human and MachineLandmarks10.1007/978-3-319-05732-3_6(175-203)Online publication date: 3-Apr-2014
https://doi.org/10.1007/978-3-319-05732-3_6
Ratzka A(2013)User Interface Patterns for Multimodal InteractionTransactions on Pattern Languages of Programming III10.1007/978-3-642-38676-3_4(111-167)Online publication date: 2013
https://doi.org/10.1007/978-3-642-38676-3_4
Khatib AMahdjoub MBluntzer JSagot J(2013)A Tool Proposition to Support Multidisciplinary Convergence in Immersive Virtual Environment: VirtusketchesSmart Product Engineering10.1007/978-3-642-30817-8_78(795-804)Online publication date: 2013
https://doi.org/10.1007/978-3-642-30817-8_78
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents