research-article

Combining Sketching and Traditional Diagram Editing Tools

Authors:

Peter RodgersAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 6, Issue 1

Article No.: 10, Pages 1 - 29

https://doi.org/10.1145/2631925

Published: 11 March 2015 Publication History

Abstract

The least cognitively demanding way to create a diagram is to draw it with a pen. Yet there is also a need for more formal visualizations, that is, diagrams created using both traditional keyboard and mouse interaction. Our objective is to allow the creation of diagrams using traditional and stylus-based input. Having two diagram creation interfaces requires that changes to a diagram should be automatically rendered in the other visualization. Because sketches are imprecise, there is always the possibility that conversion between visualizations results in a lack of syntactic consistency between the two visualizations. We propose methods for converting diagrams between forms, checking them for equivalence, and rectifying inconsistencies. As a result of our theoretical contributions, we present an intelligent software system allowing users to create and edit diagrams in sketch or formal mode. Our proof-of-concept tool supports diagrams with connected and spatial syntactic elements. Two user studies show that this approach is viable and participants found the software easy to use. We conclude that supporting such diagram creation is now possible in practice.

Supplementary Material

a10-stapleton-apndx.pdf (stapleton.zip)

Supplemental movie, appendix, image and software files for, Combining Sketching and Traditional Diagram Editing Tools

Download
220.08 KB

References

[1]

T. Baar. 2006a. Correctly defined concrete syntax for visual modeling languages. In MODELS. Springer, 111--125.

Digital Library

[2]

T. Baar. 2006b. Correctly defined concrete syntax for visual models. In MoDELS/UML. Springer, 111--125.

Digital Library

[3]

R. Blagojevic, B. Plimmer, J. Grundy, and Y. Wang. 2008. A data collection tool for sketched diagrams. In SBIM. 80--93.

Digital Library

[4]

P. Bottoni, M. Koch, F. Parisi-Presicce, and G. Taentzer. 2001. A visualization of OCL using collaborations. In UML 2001. Springer, 257--271.

Digital Library

[5]

S. H. H. Chang, R. Blagojevic, and B. Plimmer. 2012. RATA.Gesture: A gesture recognizer developed using data mining. Artificial Intelligence for Engineering, Analysis and Manufacturing 26, 3 (2012), 351--366.

Digital Library

[6]

R. Clarke. 2007. Fast zone discrimination. In Visual Languages and Logic. 41--54.

[7]

G. Cordasco, R. De Chiara, and A. Fish. 2011. Efficient on-line algorithms for Euler diagram region computation. Computational Geometry: Theory and Applications 44, 1 (2011), 52--68.

Digital Library

[8]

G. Costagliola, A. De Lucia, S. Orefice, and G. Polese. 2002. A classification framework to support the design of visual languages. Journal of Visual Languages and Computing 13, 6 (2002), 573--600.

[9]

C. Damm, K. Hansen, and M. Thomsen. 2000. Tool support for cooperative object-oriented design: Gesture based modelling on an electronic whiteboard. In SIGCHI Conference on Human Factors in Computing Systems. ACM, 518--525.

Digital Library

[10]

F. Dau. 2007. Constants and functions in Peirce’s existential graphs. In Conceptual Structures. 429--442.

Digital Library

[11]

R. DeChiara, U. Erra, and V. Scarano. 2003. VennFS: A Venn diagram file manager. In Proceedings of Information Visualisation. IEEE Computer Society, 120--126.

Digital Library

[12]

H. Dunn-Davies and R. Cunningham. 2005. Propositional statecharts for agent interaction protocols. In Proceedings of Euler Diagrams 2004, Brighton, UK (ENTCS), Vol. 134. 55--75.

Digital Library

[13]

M. Erwig. 1998. Abstract syntax and semantics of visual languages. Journal of Visual Languages and Computing 9 (1998), 461--483.

[14]

G. Farrell and W. Sousa. 2001. Repeat victimization and hot spots: The overlap and its implication for crime control and problem-oriented policing. Crime Prevention Studies 12 (2001), 221--240.

[15]

M. Field, S. Gordon, E. Peterson, R. Robinson, T. Stahovich, and C. Alvarado. 2009. The effect of task on classification accuracy: Using gesture recognition techniques in free-sketch recognition. Computers and Graphics 34, 5 (2009), 499--512.

Digital Library

[16]

A. Fish, J. Flower, and J. Howse. 2005. The semantics of augmented constraint diagrams. Journal of Visual Languages and Computing 16, 6 (2005), 541--573.

Digital Library

[17]

J. Gil, J. Howse, and S. Kent. 1999. Formalising spider diagrams. In Proceedings of IEEE Symposium on Visual Languages (VL’99). IEEE Computer Society Press, 130--137.

Digital Library

[18]

V. Goel. 1995. Sketches of Thought. MIT Press.

[19]

G. Goldschmidt. 1999. The backtalk of self-generated sketches. In Visual and Spatial Reasoning in Design. University of Sydney, 163--184.

[20]

M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. Witten. 2009. The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter 11, 1 (2009), 10--18.

Digital Library

[21]

T. Hammond and R. Davis. 2002. Tahuti: A geometrical sketch recognition system for UML class diagrams. In AAAI Spring Symposium on Sketch Understanding.

[22]

J. Howse, F. Molina, S.-J. Shin, and J. Taylor. 2001. Type-syntax and token-syntax in diagrammatic systems. In Proceedings Fof the 2nd International Conference on Formal Ontology in Information Systems (FOIS’01). Maine, USA. ACM Press, 174--185.

Digital Library

[23]

J. Howse, G. Stapleton, and J. Taylor. 2005. Spider diagrams. LMS Journal of Computation and Mathematics 8 (2005), 145--194.

[24]

J. Howse, G. Stapleton, K. Taylor, and P. Chapman. 2011. Visualizing ontologies: A case study. In International Semantic Web Conference. Springer, 257--272.

Digital Library

[25]

E. Ip. 2001. Visualizing multiple regression. Journal of Statistics Education 9, 1 (2001).

[26]

Y. Jiang, F. Tian, X. L. Zhang, G. Dai, and H. Wang. 2011. Understanding, manipulating and searching hand-drawn concept maps. ACM Transactions on Intelligent Systems and Technology 3, 11 (2011), 21.

Digital Library

[27]

G. Johnson, M. Gross, and J. Hong. 2009. Computational Support for Sketching in Design. Now Publishers.

[28]

S. Kent. 1997. Constraint diagrams: Visualizing invariants in object oriented models. In Proceedings of OOPSLA97. ACM Press, 327--341.

Digital Library

[29]

H. Kestler, A. Muller, J. Kraus, M. Buchholz, T. Gress, H. Liu, D. Kane, B. Zeeberg, and J. Weinstein. 2008. VennMaster: Area-proportional euler diagrams for functional GO analysis of microarrays. BMC Bioinformatics 9, 67 (2008).

[30]

H. Kestler, A. Muller, H. Liu, D. Kane, B. Zeeberg, and J. Weinstein. 2005. Euler diagrams for visualizing annotated gene expression data. In Proceedings of Euler Diagrams 2005, Paris.

[31]

S. Maier, S. Mazanek, and M. Minas. 2008. Layout specification on the concrete and abstract syntax level of a diagram language. In Layout of (Software) Engineering Diagrams, Vol. 13. ECEASST.

[32]

K. Mineshima, M. Okada, and R. Takemura. 2012. A diagrammatic inference system with euler circles. Journal of Logic, Language and Information 21, 3 (2012), 365--391.

Digital Library

[33]

Petri Nets. 2013. Standardisation. Retrieved from http://www.petrinets.info/.

[34]

C. Peirce. 1933. Collected Papers, Vol. 4. Harvard University Press.

[35]

B. Plimmer. 2008. Experiences with digital pen, keyboard and mouse usability. Journal of Multi-modal User Interfaces 2, 1 (2008), 13--23.

[36]

B. Plimmer and I. Freeman. 2007. A toolkit approach to sketched diagram recognition. In HCI. British Computer Society, 205--213.

Digital Library

[37]

B. Plimmer, H. Purchase, and H. Laycock. 2009. Preserving the hand-drawn appearance of graphs. In Visual Languages and Computing. 347--352.

[38]

B. Plimmer, H. Purchase, and H. Yang. 2010. SketchNode: Intelligent sketching support and formal diagramming. In OZCHI 2010. ACM, 136--143.

Digital Library

[39]

D. Rubine. 1991. Specifying gestures by example. ACM SIGGRAPH Computer Graphics 25, 4 (1991), 329--337.

Digital Library

[40]

T. Sezgin, T. Stahovich, and R. Davis. 2001. Sketch based interfaces: Early processing for sketch understanding. In Workshop on Perceptive User Interfaces.

Digital Library

[41]

S.-J. Shin. 1994. The Logical Status of Diagrams. Cambridge University Press.

[42]

J. Soriano, K. Davis B. Coleman, G. Visick, D. Mannino, and N. Pride. 2003. The proportional Venn diagram of obstructive lung disease. Chest 124 (2003), 474--481.

[43]

J. Sowa. 2013. Conceptual Graphs. Retrieved from http://www.jfsowa.com/cg/.

[44]

G. Stapleton and J. Masthoff. 2007. Incorporating negation into visual logics: A case study using euler diagrams. In Visual Languages and Computing 2007. Knowledge Systems Institute, 187--194.

[45]

G. Stapleton, P. Rodgers, J. Howse, and J. Taylor. 2007. Properties of euler diagrams. In Proceedings of Layout of Software Engineering Diagrams. EASST, 2--16.

[46]

N. Swoboda and G. Allwein. 2005. Heterogeneous reasoning with euler/venn diagrams containing named constants and FOL. In Proceedings of Euler Diagrams 2004 (ENTCS), Vol. 134. Elsevier Science.

Digital Library

[47]

J. Thièvre, M. Viaud, and A. Verroust-Blondet. 2005. Using euler diagrams in traditional library environments. In Euler Diagrams 2004 (ENTCS), Vol. 134. ENTCS, 189--202.

Digital Library

[48]

Unified Modeling Language. 2006. Unified Modeling Language (UML) Resource Page. Retrieved from http://www.uml.org/.

[49]

M. Wang, B. Plimmer, P. Schmieder, G. Stapleton, P. Rodgers, and A. Delaney. 2011. SketchSet: Creating euler diagrams using pen or mouse. In IEEE Symposium on Visual Languages and Computing. IEEE, 75--82.

[50]

L. Wilkinson. 2012. Exact and approximate area-proportional circular Venn and Euler diagrams. IEEE Transactions on Visualization and Computer Graphics 18, 2 (2012), 321--331.

Digital Library

[51]

R. Wilson. 1996. Introduction to Graph Theory. Prentice Hall.

Digital Library

[52]

J. Wobbrock, A. Wilson, and L. Yang. 2007. Gestures without libraries, toolkits or training: A &dollar;1 recognizer for user interface prototypes. In 20th Annual ACM Symposium on User Interface Software and Technology. ACM.

Digital Library

[53]

A. Wolin, M. Field, and T. Hammond. 2011. Combining corners from multiple segmenters. In 8th Eurographics Symposium on Sketch-Based Interfaces and Modeling.

Digital Library

[54]

L. Yeung, B. Plimmer, B. Lobb, and D. Elliffe. 2008. Effect of fidelity in diagram presentation. In 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction, Vol. 1. British Computer Society, 35--44.

Digital Library

Cited By

Prange ASonntag D(2022)Modeling Users' Cognitive Performance Using Digital Pen FeaturesFrontiers in Artificial Intelligence10.3389/frai.2022.7871795Online publication date: 3-May-2022
https://doi.org/10.3389/frai.2022.787179
Prange ASonntag D(2021)Assessing Cognitive Test Performance Using Automatic Digital Pen Features AnalysisProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3450613.3456812(33-43)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3450613.3456812
Chen S(2019)Exploration of artistic creation of Chinese ink style painting based on deep learning framework and convolutional neural network modelSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-019-03985-624:11(7873-7884)Online publication date: 11-Apr-2019
https://dl.acm.org/doi/10.1007/s00500-019-03985-6
Show More Cited By

Index Terms

Combining Sketching and Traditional Diagram Editing Tools
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Investigating multi-touch and pen gestures for diagram editing on interactive surfaces
ITS '09: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces

Creating and editing large graphs and node-link diagrams are crucial activities in many application areas. For them, we consider multi-touch and pen input on interactive surfaces as very promising. This fundamental work presents a user study ...
Diagram Editing with Hypergraph Parser Support
VL '97: Proceedings of the 1997 IEEE Symposium on Visual Languages (VL '97)

Diagrams are always used when communicating complex situations. Diagram editors support the user when editing diagrams on a computer. However, creating diagram editors is expensive and time-consuming. Frameworks that can be customized for the specific ...
Diagram definition: a case study with the UML class diagram
MODELS'11: Proceedings of the 14th international conference on Model driven engineering languages and systems

The abstract syntax of a graphical modeling language is typically defined with a metamodel while its concrete syntax (diagram) is informally defined with text and figures. Recently, the Object Management Group (OMG) released a beta specification, called ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 6, Issue 1

April 2015

255 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/2745393

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 March 2015

Accepted: 01 June 2014

Revised: 01 May 2014

Received: 01 September 2013

Published in TIST Volume 6, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
204
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Prange ASonntag D(2022)Modeling Users' Cognitive Performance Using Digital Pen FeaturesFrontiers in Artificial Intelligence10.3389/frai.2022.7871795Online publication date: 3-May-2022
https://doi.org/10.3389/frai.2022.787179
Prange ASonntag D(2021)Assessing Cognitive Test Performance Using Automatic Digital Pen Features AnalysisProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3450613.3456812(33-43)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3450613.3456812
Chen S(2019)Exploration of artistic creation of Chinese ink style painting based on deep learning framework and convolutional neural network modelSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-019-03985-624:11(7873-7884)Online publication date: 11-Apr-2019
https://dl.acm.org/doi/10.1007/s00500-019-03985-6
Lu ZFan MWang YZhao JAnnett MWigdor D(2018)InkPlanner: Supporting Prewriting via Intelligent Visual DiagrammingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2018.286488725:1(277-287)Online publication date: 7-Dec-2018
https://dl.acm.org/doi/10.1109/TVCG.2018.2864887
Chechik MGrünbacher P(2018)Guest editorialAutomated Software Engineering10.1007/s10515-017-0222-525:1(45-46)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s10515-017-0222-5
Kleffmann MRöhl SBook MGruhn V(2018)Evaluation of a traceability approach for informal freehand sketchesAutomated Software Engineering10.1007/s10515-017-0221-625:1(1-43)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s10515-017-0221-6
Oppl SStary CVogl S(2017)Recognition of Paper-Based Conceptual Models Captured Under Uncontrolled ConditionsIEEE Transactions on Human-Machine Systems10.1109/THMS.2016.261194347:2(206-220)Online publication date: Apr-2017
https://doi.org/10.1109/THMS.2016.2611943
Cordasco GDe Chiara RFish A(2017)Online region computations for Euler diagrams with relaxed drawing conventionsJournal of Visual Languages and Computing10.1016/j.jvlc.2016.10.00638:C(18-37)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1016/j.jvlc.2016.10.006

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents