research-article

RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching

Authors:

Rubaiat Habib Kazi,

Stephen DiVerdi,

Daniel LeithingerAuthors Info & Claims

UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Pages 166 - 181

https://doi.org/10.1145/3379337.3415892

Published: 20 October 2020 Publication History

Abstract

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid-air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.

Supplementary Material

VTT File (3379337.3415892.vtt)

Download
6.41 KB

ZIP File (ufp8798aux.mp4.zip)

video

Download
57.32 MB

MP4 File (ufp8798pv.mp4)

Preview video

Download
84.91 MB

MP4 File (ufp8798vf.mp4)

Video figure

Download
57.52 MB

MP4 File (3379337.3415892.mp4)

Presentation Video

Download
51.72 MB

References

[1]

Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. Symbiosissketch: Combining 2d & 3d sketching for designing detailed 3d objects in situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 185.

Digital Library

[2]

Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George W Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In CHI, Vol. 17. 5643--5654.

[3]

Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 151--160.

Digital Library

[4]

Connelly Barnes, David E Jacobs, Jason Sanders, Dan B Goldman, Szymon Rusinkiewicz, Adam Finkelstein, and Maneesh Agrawala. 2008. Video puppetry: a performative interface for cutout animation. In ACM Transactions on Graphics (TOG), Vol. 27. 124.

Digital Library

[5]

Mayra D Barrera Machuca, Paul Asente, Jingwan Lu, Byungmoon Kim, and Wolfgang Stuerzlinger. 2017. Multiplanes: Assisted freehand VR drawing. In Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology. 1--3.

Digital Library

[6]

Jacques Bertin. 1983. Semiology of Graphics. University of Wisconsin Press.

Digital Library

[7]

Alan Borning. 1981. The programming language aspects of ThingLab, a constraint-oriented simulation laboratory. ACM Transactions on Programming Languages and Systems (TOPLAS) 3, 4 (1981), 353--387.

Digital Library

[8]

Alan Borning and Robert Duisberg. 1986. Constraint-based tools for building user interfaces. ACM Transactions on Graphics (TOG) 5, 4 (1986), 345--374.

Digital Library

[9]

John Brosz, Miguel A Nacenta, Richard Pusch, Sheelagh Carpendale, and Christophe Hurter. 2013. Transmogrification: causal manipulation of visualizations. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 97--106.

Digital Library

[10]

Nicky Case. 2015. Explorable Explnations. (2015). https://explorabl.es

[11]

William O Chao, T Munzner, and MVD Panne. 2010. Napkinvis: Rapid pen-centric authoring of improvisational visualizations. In IEEE Infovis.

[12]

Hua-Tsung Chen, Yu-Zhen He, and Chun-Chieh Hsu. 2018. Computer-assisted yoga training system. Multimedia Tools and Applications 77, 18 (2018), 23969--23991.

Digital Library

[13]

Zhutian Chen, Yijia Su, Yifang Wang, Qianwen Wang, Huamin Qu, and Yingcai Wu. 2019. MARVisT: Authoring Glyph-based Visualization in Mobile Augmented Reality. IEEE transactions on visualization and computer graphics (2019).

[14]

Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. 2016. Programmatic and direct manipulation, together at last. ACM SIGPLAN Notices 51, 6 (2016), 341--354.

Digital Library

[15]

Pierre Dillenbourg. 2013. Design for classroom orchestration. Computers & Education 69 (2013), 485--492.

Digital Library

[16]

Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.

Digital Library

[17]

Matt Dunleavy and Chris Dede. 2014. Augmented reality teaching and learning. In Handbook of research on educational communications and technology. Springer, 735--745.

[18]

Euphrates. 2008. Mathematical Curves Hidden Around Us. (2008). https://euphrates.jp/1849115

[19]

Eurosport. 2019. Sports Explainer. (2019). https://video.eurosport.com/all-sports/video-channel/sports-explainer/

[20]

David Furió, Stéphanie Fleck, Bruno Bousquet, Jean-Paul Guillet, Lionel Canioni, and Martin Hachet. 2017. Hobit: Hybrid optical bench for innovative teaching. In Proceedings of the 2017 chi conference on human factors in computing systems. ACM, 949--959.

Digital Library

[21]

Emiliano Gambaretto and Charles Pi na. 2014. Real-time Animation of Cartoon Character Faces. In ACM SIGGRAPH 2014 Computer Animation Festival (SIGGRAPH '14). 1--1. http://dx.doi.org/10.1145/2633956.2658830

[22]

Danilo Gasques, Janet G Johnson, Tommy Sharkey, and Nadir Weibel. 2019. What you sketch is what you get: Quick and easy augmented reality prototyping with pintar. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1--6.

Digital Library

[23]

Oliver Glauser, Wan-Chun Ma, Daniele Panozzo, Alec Jacobson, Otmar Hilliges, and Olga Sorkine-Hornung. 2016. Rig animation with a tangible and modular input device. ACM Transactions on Graphics (TOG) 35, 4 (2016), 144.

Digital Library

[24]

Mark D Gross and Ellen Yi-Luen Do. 1996. Ambiguous intentions: a paper-like interface for creative design. In Proceedings of the 9th annual ACM symposium on User interface software and technology. 183--192.

Digital Library

[25]

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961--2969.

[26]

Robert Held, Ankit Gupta, Brian Curless, and Maneesh Agrawala. 2012. 3D puppetry: a kinect-based interface for 3D animation. In UIST '12. 423--434.

Digital Library

[27]

Valentin Heun, James Hobin, and Pattie Maes. 2013a. Reality editor: programming smarter objects. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. 307--310.

Digital Library

[28]

Valentin Heun, Shunichi Kasahara, and Pattie Maes. 2013b. Smarter Objects: Using AR Technology to Program Physical Objects and Their Interactions. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). Association for Computing Machinery, New York, NY, USA, 961--966. http://dx.doi.org/10.1145/2468356.2468528

Digital Library

[29]

Otmar Hilliges, David Kim, Shahram Izadi, Malte Weiss, and Andrew Wilson. 2012. HoloDesk: direct 3d interactions with a situated see-through display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2421--2430.

Digital Library

[30]

Derek Hodson. 1996. Laboratory work as scientific method: Three decades of confusion and distortion. Journal of Curriculum studies 28, 2 (1996), 115--135.

[31]

Joshua Horowitz. 2015. Gallery of Concept Visualization. (2015). https://conceptviz.github.io/

[32]

Takeo Igarashi and John F Hughes. 2007. A suggestive interface for 3D drawing. In ACM SIGGRAPH 2007 courses. 20--es.

Digital Library

[33]

Takeo Igarashi, Satoshi Matsuoka, Sachiko Kawachiya, and Hidehiko Tanaka. 2007. Interactive beautification: a technique for rapid geometric design. In ACM SIGGRAPH 2007 courses. 18--es.

Digital Library

[34]

Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 2006. Teddy: a sketching interface for 3D freeform design. In ACM SIGGRAPH 2006 Courses. 11--es.

Digital Library

[35]

Byju's Inc. 2019a. Byju's The Learning App. (2019). https://byjus.com

[36]

Google Inc. 2016a. Google Science Journal. (2016). https://sciencejournal.withgoogle.com

[37]

Google Inc. 2016b. TiltBrush. (2016). https://www.tiltbrush.com/

[38]

Google Inc. 2018. Just a Line. (2018). https://justaline.withgoogle.com/

[39]

Gravity Sketch Inc. 2017a. Gravity Sketch. (2017). https://www.gravitysketch.com/

[40]

NEX Team Inc. 2019b. HomeCourt. (2019). https://www.homecourt.ai/

[41]

PTC Inc. 2017b. Vuforia Chalk AR. (2017). https://chalk.vuforia.com/

[42]

Shapr3D Inc. Shapr3D Constraint. (????). https://support.shapr3d.com/hc/en-us/articles/115001416214-Constraints

[43]

Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. 234--241.

Digital Library

[44]

Hiroshi Ishii, Craig Wisneski, Julian Orbanes, Ben Chun, and Joe Paradiso. 1999. PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 394--401.

Digital Library

[45]

Robert JK Jacob, Audrey Girouard, Leanne M Hirshfield, Michael S Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 201--210.

Digital Library

[46]

Gabe Johnson, Mark Gross, Ellen Yi-Luen Do, and Jason Hong. 2012. Sketch it, make it: sketching precise drawings for laser cutting. In CHI'12 Extended Abstracts on Human Factors in Computing Systems. ACM, 1079--1082.

Digital Library

[47]

Seokbin Kang, Ekta Shokeen, Virginia L Byrne, Leyla Norooz, Elizabeth Bonsignore, Caro Williams-Pierce, and Jon E Froehlich. 2020. ARMath: Augmenting Everyday Life with Math Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--15.

Digital Library

[48]

Shunichi Kasahara, Valentin Heun, Austin S Lee, and Hiroshi Ishii. 2012. Second surface: multi-user spatial collaboration system based on augmented reality. In SIGGRAPH Asia 2012 Emerging Technologies. 1--4.

[49]

Shunichi Kasahara, Ryuma Niiyama, Valentin Heun, and Hiroshi Ishii. 2013. exTouch: spatially-aware embodied manipulation of actuated objects mediated by augmented reality. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction. 223--228.

Digital Library

[50]

Alan Kay. 2005. Squeak etoys, children & learning. online article 2006 (2005).

[51]

Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice. 2014a. Kitty: sketching dynamic and interactive illustrations. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 395--405.

Digital Library

[52]

Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, Shengdong Zhao, and George Fitzmaurice. 2014b. Draco: bringing life to illustrations with kinetic textures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 351--360.

Digital Library

[53]

Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Motion amplifiers: sketching dynamic illustrations using the principles of 2D animation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4599--4609.

Digital Library

[54]

Rubaiat Habib Kazi, Takeo Igarashi, Shengdong Zhao, and Richard Davis. 2012. Vignette: interactive texture design and manipulation with freeform gestures for pen-and-ink illustration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1727--1736.

Digital Library

[55]

Annie Kelly, R Benjamin Shapiro, Jonathan de Halleux, and Thomas Ball. 2018. ARcadia: A rapid prototyping platform for real-time tangible interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--8.

Digital Library

[56]

Yongkwan Kim and Seok-Hyung Bae. 2016. SketchingWithHands: 3D sketching handheld products with first-person hand posture. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 797--808.

Digital Library

[57]

Yoonji Kim, Youngkyung Choi, Hyein Lee, Geehyuk Lee, and Andrea Bianchi. 2019. VirtualComponent: a Mixed-Reality Tool for Designing and Tuning Breadboarded Circuits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 177.

Digital Library

[58]

Kin Chung Kwan and Hongbo Fu. 2019. Mobi3DSketch: 3D Sketching in Mobile AR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 176.

Digital Library

[59]

David Lakatos, Matthew Blackshaw, Alex Olwal, Zachary Barryte, Ken Perlin, and Hiroshi Ishii. 2014. T (ether): spatially-aware handhelds, gestures and proprioception for multi-user 3D modeling and animation. In Proceedings of the 2nd ACM symposium on Spatial user interaction. 90--93.

Digital Library

[60]

James A Landay. 1996. SILK: sketching interfaces like krazy. In Conference companion on Human factors in computing systems. 398--399.

Digital Library

[61]

James A Landay and Brad A Myers. 1995. Interactive sketching for the early stages of user interface design. In Proceedings of the SIGCHI conference on Human factors in computing systems. 43--50.

Digital Library

[62]

James A Landay and Brad A Myers. 2001. Sketching interfaces: Toward more human interface design. Computer 34, 3 (2001), 56--64.

Digital Library

[63]

Joseph J LaViola Jr and Robert C Zeleznik. 2004. MathPad 2: a system for the creation and exploration of mathematical sketches. ACM Transactions on Graphics (TOG) 23, 3 (2004), 432--440.

Digital Library

[64]

Bongshin Lee, Rubaiat Habib Kazi, and Greg Smith. 2013. SketchStory: Telling more engaging stories with data through freeform sketching. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2416--2425.

Digital Library

[65]

Joon Hyub Lee, Hyung-Gi Ham, and Seok-Hyung Bae. 2020. 3D Sketching for Multi-Pose Products. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1--8.

Digital Library

[66]

Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2020. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--13.

Digital Library

[67]

Yuwei Li, Xi Luo, Youyi Zheng, Pengfei Xu, and Hongbo Fu. 2017. SweepCanvas: Sketch-based 3D prototyping on an RGB-D image. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 387--399.

Digital Library

[68]

Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman. 2019. Learning the depths of moving people by watching frozen people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4521--4530.

[69]

Jingyuan Liu, Hongbo Fu, and Chiew-Lan Tai. 2020. PoseTween: Pose-driven Tween Animation. In UIST '20.

Digital Library

[70]

Zhicheng Liu, John Thompson, Alan Wilson, Mira Dontcheva, James Delorey, Sam Grigg, Bernard Kerr, and John Stasko. 2018. Data Illustrator: Augmenting vector design tools with lazy data binding for expressive visualization authoring. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 123.

Digital Library

[71]

William F McComas, Michael P Clough, and Hiya Almazroa. 1998. The role and character of the nature of science in science education. In The nature of science in science education. Springer, 3--39.

[72]

Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321--1329.

[73]

Elizabeth D Mynatt, Takeo Igarashi, W Keith Edwards, and Anthony LaMarca. 1999. Flatland: new dimensions in office whiteboards. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 346--353.

Digital Library

[74]

Koya Narumi, Steve Hodges, and Yoshihiro Kawahara. 2015. ConductAR: an augmented reality based tool for iterative design of conductive ink circuits. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 791--800.

Digital Library

[75]

Michael Nebeling and Katy Madier. 2019. 360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--13.

Digital Library

[76]

Michael Nebeling, Janet Nebeling, Ao Yu, and Rob Rumble. 2018. Protoar: Rapid physical-digital prototyping of mobile augmented reality applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--12.

Digital Library

[77]

Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D Wilson. 2016. Snaptoreality: Aligning augmented reality to the real world. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1233--1244.

Digital Library

[78]

Katherine Perkins, Wendy Adams, Michael Dubson, Noah Finkelstein, Sam Reid, Carl Wieman, and Ron LeMaster. 2006. PhET: Interactive simulations for teaching and learning physics. The physics teacher 44, 1 (2006), 18--23.

[79]

Ken Perlin. 2016. Future Reality: How emerging technologies will change language itself. IEEE computer graphics and applications 36, 3 (2016), 84--89.

[80]

Ken Perlin, Zhenyi He, and Karl Rosenberg. 2018a. Chalktalk: A Visualization and Communication Language--As a Tool in the Domain of Computer Science Education. arXiv preprint arXiv:1809.07166 (2018).

[81]

Ken Perlin, Zhenyi He, and Fengyuan Zhu. 2018b. Chalktalk VR/AR. International SERIES on Information Systems and Management in Creative eMedia (CreMedia) 2017/2 (2018), 30--31.

[82]

Ben Piper, Carlo Ratti, and Hiroshi Ishii. 2002. Illuminating clay: a 3-D tangible interface for landscape analysis. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 355--362.

Digital Library

[83]

Iulian Radu and Bertrand Schneider. 2019. What Can We Learn from Augmented Reality (AR)?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 544.

Digital Library

[84]

Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, and Thomas Willwacher. 2004. RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors. In ACM SIGGRAPH 2004 Papers. 406--415.

Digital Library

[85]

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 779--788.

[86]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems. 91--99.

[87]

Nazmus Saquib, Rubaiat Habib Kazi, Li-Yi Wei, and Wilmot Li. 2019. Interactive Body-Driven Graphics for Augmented Video Performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 622.

Digital Library

[88]

John Sarracino, Odaris Barrios-Arciga, Jasmine Zhu, Noah Marcus, Sorin Lerner, and Ben Wiedermann. 2017. User-Guided Synthesis of Interactive Diagrams. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 195--207.

Digital Library

[89]

Toby Schachman. 2015. Apparatus: a hybrid graphics editor/programming environment for creating interactive diagrams. Strange Loop (2015).

[90]

Steven Schkolne, Michael Pruett, and Peter Schröder. 2001. Surface drawing: creating organic 3D shapes with the hand and tangible tools. In Proceedings of the SIGCHI conference on Human factors in computing systems. 261--268.

Digital Library

[91]

Jeremy Scott and Randall Davis. 2013. Physink: sketching physical behavior. In Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology. ACM, 9--10.

Digital Library

[92]

PBS: Public Broadcasting Service. 2019. Nova. (2019). https://www.pbs.org/wgbh/nova/

[93]

L Sousa, R Alves, and JMF Rodrigues. 2016. Augmented reality system to assist inexperienced pool players. Computational Visual Media 2, 2 (2016), 183--193.

[94]

Ivan E Sutherland. 1964. Sketchpad a man-machine graphical communication system. Simulation 2, 5 (1964), R--3.

[95]

Ryo Suzuki, Jun Kato, Mark D Gross, and Tom Yeh. 2018. Reactile: Programming Swarm User Interfaces through Direct Physical Manipulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 199.

Digital Library

[96]

Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Björn Hartmann. 2019. TutoriVR: A Video-Based Tutorial System for Design Applications in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.

Digital Library

[97]

John Underkoffler and Hiroshi Ishii. 1999. Urp: a luminous-tangible workbench for urban planning and design. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 386--393.

Digital Library

[98]

Bret Victor. 2011. Explorable explanations. Bret Victor 10 (2011).

[99]

Bret Victor. 2012. Stop drawing dead fish. In ACM SIGGRAPH.

[100]

Bret Victor. 2013. Drawing dynamic visualizations. URL: worrydream. com (2013).

[101]

Bret Victor. 2014a. Humane representation of thought: a trail map for the 21st century. In Proceedings of the companion publication of the 2014 ACM SIGPLAN conference on Systems, Programming, and Applications: Software for Humanity. 5--5.

Digital Library

[102]

Bret Victor. 2014b. Seeing spaces. In Talk at EG conference.

[103]

Jerry Wellington. 1998. Practical work in school science: Which way now? Routledge.

[104]

Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (1993), 87--96.

Digital Library

[105]

Gerold Wesche and Hans-Peter Seidel. 2001. FreeDrawer: a free-form sketching system on the responsive workbench. In Proceedings of the ACM symposium on Virtual reality software and technology. 167--174.

Digital Library

[106]

Carl E Wieman, Wendy K Adams, and Katherine K Perkins. 2008. PhET: Simulations that enhance learning. Science 322, 5902 (2008), 682--683.

[107]

Wesley Willett, Yvonne Jansen, and Pierre Dragicevic. 2016. Embedded data representations. IEEE transactions on visualization and computer graphics 23, 1 (2016), 461--470.

Digital Library

[108]

Aidan Wolf. 2019. DoodleLens. (2019). https://www.doodlelens.app/

[109]

Haijun Xia, Nathalie Henry Riche, Fanny Chevalier, Bruno De Araujo, and Daniel Wigdor. 2018. DataInk: Direct and Creative Data-Oriented Drawing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 223, 13 pages. http://dx.doi.org/10.1145/3173574.3173797

Digital Library

[110]

Jun Xing, Rubaiat Habib Kazi, Tovi Grossman, Li-Yi Wei, Jos Stam, and George Fitzmaurice. 2016. Energy-brushes: Interactive tools for illustrating stylized elemental dynamics. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 755--766.

Digital Library

[111]

Bo Zhu, Michiaki Iwata, Ryo Haraguchi, Takashi Ashihara, Nobuyuki Umetani, Takeo Igarashi, and Kazuo Nakazawa. 2011. Sketch-based dynamic illustration of fluid systems. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1--8.

Digital Library

Cited By

Tong WShigyo KYuan LFan MPong TQu HXia M(2025)VisTellAR: Embedding Data Visualization to Short-Form Videos Using Mobile Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210431:3(1862-1874)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3372104
Takeyama YNagayama KChoi MShizuki B(2024)PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a CanvasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682078(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682078
Suh SDang HYen RPollock JArawjo IKazi RSubramonyam HLi JSaquib NSatyanarayan A(2024)Dynamic Abstractions: Building the Next Generation of Cognitive Tools and InterfacesAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686706(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686706
Show More Cited By

Index Terms

RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality

Recommendations

RealitySketch: Embedding Responsive Graphics and Visualizations in AR with Dynamic Sketching
UIST '20 Adjunct: Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the ...
RealitySketch: augmented reality sketching for real-time embedded and responsive visualizations
SA '21: SIGGRAPH Asia 2021 Real-Time Live!

In this Real-Time Live, we demonstrate RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations [Suzuki et al. 2020]. In recent years, an increasing number of AR sketching tools enable users to draw and embed ...
RealityCanvas: Augmented Reality Sketching for Embedded and Responsive Scribble Animation Effects
UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

We introduce RealityCanvas, a mobile AR sketching tool that can easily augment real-world physical motion with responsive hand-drawn animation. Recent research in AR sketching tools has enabled users to not only embed static drawings into the real world ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

October 2020

1297 pages

ISBN:9781450375146

DOI:10.1145/3379337

General Chairs:
Shamsi Iqbal
Microsoft Research, USA
,
Karon MacLean
University of British Columbia, Canada
,
Program Chairs:
Fanny Chevalier
University of Toronto, Canada
,
Stefanie Mueller
MIT CSAIL, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Funding Sources

Nakajima Foundation scholarship

Conference

UIST '20

Sponsor:

UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology

October 20 - 23, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

68
Total Citations
View Citations
1,838
Total Downloads

Downloads (Last 12 months)291
Downloads (Last 6 weeks)17

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tong WShigyo KYuan LFan MPong TQu HXia M(2025)VisTellAR: Embedding Data Visualization to Short-Form Videos Using Mobile Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210431:3(1862-1874)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3372104
Takeyama YNagayama KChoi MShizuki B(2024)PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a CanvasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682078(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682078
Suh SDang HYen RPollock JArawjo IKazi RSubramonyam HLi JSaquib NSatyanarayan A(2024)Dynamic Abstractions: Building the Next Generation of Cognitive Tools and InterfacesAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686706(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686706
Su XFroehlich JKoh EXiao C(2024)SonifyAR: Context-Aware Sound Generation in Augmented RealityProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676406(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676406
Zhang HChen PXie XLin CLiu LLi ZYou WSun L(2024)ProtoDreamer: A Mixed-prototype Tool Combining Physical Model and Generative AI to Support Conceptual DesignProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676399(1-18)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676399
Gunturu AWen YZhang NThundathil JKazi RSuzuki R(2024)Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook DiagramsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676392(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676392
Dogan MGonzalez EAhuja KDu RColaço ALee JGonzalez-Franco MKim D(2024)Augmented Object Intelligence with XR-ObjectsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676379(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676379
Rosenberg KKazi RWei LXia HPerlin K(2024)DrawTalking: Building Interactive Worlds by Sketching and SpeakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676334(1-25)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676334
Li BYuan LYan ZLiu QShen YWang Z(2024)AniCraft: Crafting Everyday Objects as Physical Proxies for Prototyping 3D Character Animation in Mixed RealityProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676325(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676325
Nghiem CBousseau ASypesteyn MHoftijzer JTsandilas T(2024)Sketch Presentation for Product DesignProceedings of the 35th International Francophone Conference on Human-Computer Interaction10.1145/3650104.3652907(1-4)Online publication date: 25-Mar-2024
https://dl.acm.org/doi/10.1145/3650104.3652907
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten