research-article

Proxy Painting: Digital Colorization of Real-world Objects

Authors:

Benjamin Keinert,

Marc Stamminger,

Frank BauerAuthors Info & Claims

Journal on Computing and Cultural Heritage (JOCCH), Volume 13, Issue 3

Article No.: 16, Pages 1 - 20

https://doi.org/10.1145/3377145

Published: 20 July 2020 Publication History

Abstract

For archaeologists, it is often desirable to present statues in their original material and coloration. With projection mapping, real-world surfaces are augmented by digital content to create compelling alterations of the scene’s visual appearance without actually altering or even damaging the object. While there are frequent advances in projection quality, content creation is still a challenging and often unintuitive task, especially for non-experts. In our system, we combine the advantages of digital content creation, such as rapid prototyping, with the convenience of an analog workflow. Users paint on smaller versions of the projection mapping target, employing real-world brushes and pencils, while the results are presented live on its large counterpart. We further demonstrate the integration of our system into a state-of-art game engine. By leveraging a powerful rendering and material workflow, we make creating compelling materials and lighting situations an intuitive experience.

References

[1]

Maneesh Agrawala, Andrew C. Beers, and Marc Levoy. 1995. 3D painting on scanned surfaces. In Proceedings of the 1995 Symposium on Interactive 3D Graphics (I3D’95). ACM, New York, NY, 145--ff.

Digital Library

[2]

H. Asayama, D. Iwai, and K. Sato. 2018. Fabricating diminishable visual markers for geometric registration in projection mapping. IEEE Trans. Vis. Comput. Graph. 24, 2 (February 2018), 1091--1102.

[3]

D. Bandyopadhyay, R. Raskar, and H. Fuchs. 2001. Dynamic shader lamps : Painting on movable objects. In Proceedings of the IEEE and ACM International Symposium on Augmented Reality. 207--216.

[4]

Bill Baxter, Vincent Scheib, Ming C. Lin, and Dinesh Manocha. 2001. DAB: Interactive haptic painting with 3D virtual brushes. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’01). ACM, New York, NY, 461--468.

Digital Library

[5]

William Baxter, Jeremy Wendt, and Ming C. Lin. 2004. IMPaSTo: A realistic, interactive model for paint. In Proceedings of the 3rd International Symposium on Non-photorealistic Animation and Rendering (NPAR’04). ACM, New York, NY, 45--148.

[6]

Oliver Bimber, Daisuke Iwai, Gordon Wetzstein, and Anselm Grundhöfer. 2008. The visual computing of projector-camera systems. In Computer Graphics Forum, Vol. 27. Wiley Online Library.

[7]

Ryan Brucks. 2016. Automated and Improved UV Dilation. Retrieved May 23 from http://shaderbits.com/blog/uv-dilation.

[8]

Renée Charrière, Mathieu Hébert, Alain Trémeau, and Nathalie Destouches. 2013. Color calibration of an RGB camera mounted in front of a microscope with strong color distortion. Appl. Opt. 52, 21 (July 2013), 5262--5271.

[9]

Patrick Tobias Fischer, Anke von der Heide, Eva Hornecker, Sabine Zierold, Andreas Kästner, Matti Dondera, Felix andWiegmann, Fernando Millán, Jonas Lideikis, Aidas Čergelis, Reinaldo Verde, Christoph Drews, Till Fastnacht, Kai Gerrit Lünsdorf, Djamel Merad, Aryan Khosravani, and Hesam Jannesar. 2015. Castle-sized interfaces: An interactive façade mapping. In Proceedings of the 4th International Symposium on Pervasive Displays (PerDis’15). ACM, New York, NY, 91--97.

[10]

Anselm Grundhöfer and Iwai Daisuke. 2018. Recent advances in projection mapping algorithms, hardware and applications. Comput. Graph. Forum 37, 2 (2018), 653--675.

[11]

D. Johnson, T. V. Thompson, M. Kaplan, D. Nelson, and E. Cohen. 1999. Painting textures with a haptic interface. In Proceedings of the IEEE Conference on Virtual Reality. 282--285.

[12]

Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira. 2014. RoomAlive: Magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST’14). ACM, NY, 8.

Digital Library

[13]

Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2013. IllumiRoom: Peripheral projected illusions for interactive experiences. In Proceedings of the ACM SIGGRAPH 2013 Emerging Technologies (SIGGRAPH’13). ACM, New York, NY, Article 7, 1 pages.

[14]

Philipp Kurth, Vanessa Lange, Christian Siegl, Marc Stamminger, and Frank Bauer. 2018. Auto-calibration for dynamic multi-projection mapping on arbitrary surfaces. IEEE Transactions on Visualization and Computer Graphics 24, 11 (Nov. 2018), 2886--2894.

[15]

Vanessa Lange, Philipp Kurth, Benjamin Keinert, Martin Boss, Marc Stamminger, and Frank Bauer. 2018. Proxy painting. In Eurographics Workshop on Graphics and Cultural Heritage, Robert Sablatnig and Michael Wimmer (Eds.). The Eurographics Association.

[16]

Vanessa Lange, Christian Siegl, Matteo Colaianni, Philipp Kurth, Marc Stamminger, and Frank Bauer. 2016. Interactive painting and lighting in dynamic multi-projection mapping. In Proceedings of the Annual Conference on Augmented Reality, Virtual Reality, and Computer Graphics (AVR’16). Springer, Cham, 113--125.

[17]

Vanessa Lange, Christian Siegl, Matteo Colaianni, Marc Stamminger, and Frank Bauer. 2017. Robust blending and occlusion compensation in dynamic multi-projection mapping. In Proceedings of the Annual Conference of the European Association for Computer Graphics (Eurographics’17).

[18]

Yong Yi Lee, Jong Hun Lee, Bilal Ahmed, Moon Gu Son, and Kwan H. Lee. 2019. A new projection-based exhibition system for a museum. J. Comput. Cult. Herit. 12, 2, Article 10 (May 2019), 17 pages.

Digital Library

[19]

Paolo Liverani. 2003. Die polychromie des augustus von prima porta, vorläufiger bericht. Neue Forschungen zur hellenistischen Plastik: Kolloquium zum 70. Geburtstag von Georg Daltrop (2003), 121--140.

[20]

M. R. Marner, B. H. Thomas, and C. Sandor. 2009. Physical-virtual tools for spatial augmented reality user interfaces. In Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality. 205--206.

Digital Library

[21]

M. E. Munich and P. Perona. 2002. Visual input for pen-based computers. IEEE Trans. Pattern Anal. Mach. Intell. 24, 3 (March 2002), 313--328.

Digital Library

[22]

Fred. G. Paas. 1992. Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. J. Educ. Psychol. 84, 4 (1992), 429--434.

[23]

Rosa Peral, Diego Sagasti, and Sara Sillaurren. 2005. Virtual restoration of cultural heritage through real-time 3D models projection. In Proceedings of the 6th International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST'05), M. Mudge, N. Ryan, and R. Scopigno (Eds.). Pisa, Italy.

[24]

G. T. Pfeiffer, R. G. Marroquim, and A. A. F. d. Oliveira. 2014. WebcamPaperPen: A low-cost graphics tablet. In Proceedings of the 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images. 87--94.

Digital Library

[25]

Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. 2001. Shader lamps: Animating real objects with image-based illumination. In Proceedings of the 12th Eurographics Workshop on Rendering Techniques. Springer-Verlag, London, 14.

[26]

Brett Ridel, Patrick Reuter, Jeremy Laviole, Nicolas Mellado, Nadine Couture, and Xavier Granier. 2014. The revealing flashlight: Interactive spatial augmented reality for detail exploration of cultural heritage artifacts. J. Comput. Cult. Herit. 7, 2, Article 6 (June 2014), 18 pages.

Digital Library

[27]

Christian Siegl, Matteo Colaianni, Marc Stamminger, and Frank Bauer. 2017. Adaptive stray-light compensation in dynamic multi-projection mapping. Computational Visual Media 3, 3 (2017), 263--271.

[28]

Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollhöfer, Shahram Izadi, Marc Stamminger, and Frank Bauer. 2015. Real-time pixel luminance optimization for dynamic multi-projection mapping. ACM Trans. Graph. 34, 6, Article 237 (Oct. 2015), 11 pages.

Digital Library

[29]

Richard Swinbank and R. James Purser. 2006. Fibonacci grids: A novel approach to global modelling. Quart. J. Roy. Meteorol. Soc. 132, 619 (2006), 1769--1793.

[30]

Yi Zhou, Shuangjiu Xiao, Ning Tang, Zhiyong Wei, and Xu Chen. 2016. Pmomo: Projection mapping on movable 3d object. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 781--790.

Digital Library

Cited By

Basu APaul SGhosh SDas SChanda BBhagvati CSnasel V(2023)Digital Restoration of Cultural Heritage With Data-Driven Computing: A SurveyIEEE Access10.1109/ACCESS.2023.328063911(53939-53977)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3280639
Huang SJin XJiang QLiu L(2022)Deep learning for image colorization: Current and future prospectsEngineering Applications of Artificial Intelligence10.1016/j.engappai.2022.105006114(105006)Online publication date: Sep-2022
https://doi.org/10.1016/j.engappai.2022.105006
Caron GBelghiti MDessaux A(2022)3D Model-Based Tracking of Puppet in Depth Images for the Dynamic Video-Mapping of Its SuitOptoelectronic Devices in Robotic Systems10.1007/978-3-031-09791-1_1(1-21)Online publication date: 15-Jun-2022
https://doi.org/10.1007/978-3-031-09791-1_1
Show More Cited By

Index Terms

Proxy Painting: Digital Colorization of Real-world Objects
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Mixed / augmented reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality

Recommendations

A New Projection-based Exhibition System for a Museum

In this article, we propose a new projection-based system to exhibit a realistic virtual cultural object in a museum. We introduce a new design by combining the multi-projection mapping technology with an optical see-through display that overcomes the ...
MAI painting brush++: augmenting the feeling of painting with new visual and tactile feedback mechanisms
UIST '11 Adjunct: Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology

We have developed a mixed-reality (MR) painting system named the MR-based Artistic Interactive (MAI) Painting Expert and MAI Painting Brush which simulates the painting of physical objects in the real world. In this paper, we describe how the MAI ...
MAI painting brush: an interactive device that realizes the feeling of real painting
UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology

Many digital painting systems have been proposed and their quality is improving. In these systems, graphics tablets are widely used as input devices. However, because of its rigid nib and indirect manipulation, the operational feeling of a graphics ...

Comments

Information & Contributors

Information

Published In

cover image Journal on Computing and Cultural Heritage

Journal on Computing and Cultural Heritage Volume 13, Issue 3

October 2020

211 pages

ISSN:1556-4673

EISSN:1556-4711

DOI:10.1145/3411173

Editor:
Franco Niccolucci
VAST-LAB at PIN, University of Florence, Italy

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2020

Online AM: 07 May 2020

Accepted: 01 December 2019

Revised: 01 October 2019

Received: 01 April 2019

Published in JOCCH Volume 13, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
309
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Basu APaul SGhosh SDas SChanda BBhagvati CSnasel V(2023)Digital Restoration of Cultural Heritage With Data-Driven Computing: A SurveyIEEE Access10.1109/ACCESS.2023.328063911(53939-53977)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3280639
Huang SJin XJiang QLiu L(2022)Deep learning for image colorization: Current and future prospectsEngineering Applications of Artificial Intelligence10.1016/j.engappai.2022.105006114(105006)Online publication date: Sep-2022
https://doi.org/10.1016/j.engappai.2022.105006
Caron GBelghiti MDessaux A(2022)3D Model-Based Tracking of Puppet in Depth Images for the Dynamic Video-Mapping of Its SuitOptoelectronic Devices in Robotic Systems10.1007/978-3-031-09791-1_1(1-21)Online publication date: 15-Jun-2022
https://doi.org/10.1007/978-3-031-09791-1_1
Klein VLeuschner MLangen TKurth PStamminger MBauer F(2021)Scan&Paint: Image-based Projection Painting2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR52148.2021.00069(517-525)Online publication date: Oct-2021
https://doi.org/10.1109/ISMAR52148.2021.00069

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents