research-article

Interactive Sketch-Based Normal Map Generation with Deep Neural Networks

Authors:

Hongbo FuAuthors Info & Claims

Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 1, Issue 1

Article No.: 22, Pages 1 - 17

https://doi.org/10.1145/3203186

Published: 25 July 2018 Publication History

Abstract

High-quality normal maps are important intermediates for representing complex shapes. In this paper, we propose an interactive system for generating normal maps with the help of deep learning techniques. Utilizing the Generative Adversarial Network (GAN) framework, our method produces high quality normal maps with sketch inputs. In addition, we further enhance the interactivity of our system by incorporating user-specified normals at selected points. Our method generates high quality normal maps in real time. Through comprehensive experiments, we show the effectiveness and robustness of our method. A thorough user study indicates the normal maps generated by our method achieve a lower perceptual difference from the ground truth compared to the alternative methods.

References

[1]

M. Arjovsky and L. Bottou. 2017. Towards Principled Methods for Training Generative Adversarial Networks. ArXiv e-prints (Jan. 2017). arXiv:1701.04862

[2]

M. Arjovsky, S. Chintala and L. Bottou. 2017. Wasserstein GAN. ArXiv e-prints (Jan. 2017). arXiv:1701.07875

[3]

Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A Non-Local Algorithm for Image Denoising. In CVPR '05. 60--65.

Digital Library

[4]

Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 2016. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. In ECCV '16. 628--644.

[5]

Doug DeCarlo, Adam Finkelstein, Szymon Rusinkiewicz, and Anthony Santella. 2003. Suggestive Contours for Conveying Shape. ACM Trans. Graph. 22, 3 (July 2003), 848--855.

Digital Library

[6]

Emily L Denton, Soumith Chintala, arthur szlam, and Rob Fergus. 2015. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. In Advances in Neural Information Processing Systems 28. 1486--1494.

Digital Library

[7]

Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In SIGGRAPH '01. 341--346.

Digital Library

[8]

David Eigen and Rob Fergus. 2015. Predicting Depth, Surface Normals and Semantic Labels With a Common Multi-Scale Convolutional Architecture. In ICCV '15.

Digital Library

[9]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). 2672--2680.

Digital Library

[10]

X. Han, C. Gao, and Y. Yu. 2017. DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling. ACM Trans. Graph. 36, 4 (July 2017).

Digital Library

[11]

Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In SIGGRAPH '01. 327--340.

Digital Library

[12]

G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786 (2006), 504--507.

[13]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and Locally Consistent Image Completion. ACM Trans. Graph. 36, 4 (July 2017), 107:1--107:14.

Digital Library

[14]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR '17. 5967--5976.

[15]

Scott F. Johnston. 2002. Lumo: Illumination for Cel Animation. In NPAR '02. 45--52.

Digital Library

[16]

Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. 2010. Learning 3D Mesh Segmentation and Labeling. ACM Trans. Graph. 29, 4 (July 2010), 102:1--102:12.

Digital Library

[17]

T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim. 2017. Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. ArXiv e-prints (March 2017). arXiv:1703.05192

[18]

Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. 2017. BendSketch: Modeling Freeform Surfaces Through 2D Sketching. ACM Trans. Graph. 36, 4 (July 2017), 125:1--125:14.

Digital Library

[19]

Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. CoRR abs/1707.06375 (2017). arXiv:1707.06375

[20]

M. Mirza and S. Osindero. 2014. Conditional Generative Adversarial Nets. ArXiv e-prints (nov 2014). arXiv:1411.1784

[21]

Luke Olsen, Faramarz F. Samavati, Mario Costa Sousa, and Joaquim A. Jorge. 2009. Sketch-based modeling: A survey. Computers 8 Graphics 33, 1 (2009), 85--103.

Digital Library

[22]

Hao Pan, Yang Liu, Alla Sheffer, Nicholas Vining, Chang-Jian Li, and Wenping Wang. 2015. Flow Aligned Surfacing of Curve Networks. ACM Trans. Graph. 34, 4 (July 2015), 127:1--127:10.

Digital Library

[23]

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context Encoders: Feature Learning by Inpainting. In CVPR '16. 2536--2544.

[24]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI '15. 234--241.

[25]

T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. 2016. Improved Techniques for Training GANs. ArXiv e-prints (June 2016). arXiv:1606.03498

[26]

Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic Drawing of 3D Scaffolds. ACM Trans. Graph. 28, 5 (Dec. 2009), 149:1--149:10.

Digital Library

[27]

Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: Shading Concept Sketches Using Cross-section Curves. ACM Trans. Graph. 31, 4 (July 2012), 45:1--45:11.

Digital Library

[28]

Daniel Sýkora, Ladislav Kavan, Martin Čadík, Ondřej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-ray: Bas-relief Meshes for Adding Global Illumination Effects to Hand-drawn Characters. ACM Trans. Graph. 33, 2 (April 2014), 16:1--16:15.

Digital Library

[29]

Xiaolong Wang, David Fouhey, and Abhinav Gupta. 2015. Designing Deep Networks for Surface Normal Estimation. In CVPR '15.

[30]

Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In NIPS '16. 82--90.

Digital Library

[31]

Tai-Pang Wu, Chi-Keung Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. ShapePalettes: interactive normal transfer via sketching. In SIGGRAPH '07. 44.

Digital Library

[32]

Q. Xu, Y. Gingold, and K. Singh. 2015. Inverse Toon Shading: Interactive Normal Field Modeling with Isophotes. In SBIM '15. 15--25.

Digital Library

[33]

Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision. (12 2016).

[34]

Z. Yi, H. Zhang, P. Tan, and M. Gong. 2017. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation. ArXiv e-prints (April 2017). arXiv:1704.02510

[35]

H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas. 2016b. StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. ArXiv e-prints (Dec. 2016). arXiv:1612.03242

[36]

Richard Zhang, Phillip Isola and Alexei A. Efros. 2016a. Colorful Image Colorization. In ECCV '16. 649--666.

[37]

Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Trans. Graph. 9, 4 (2017).

Digital Library

[38]

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ArXiv e-prints (March 2017). arXiv:1703.10593

Cited By

Puhachov IMartens CKry PBessmeltsev M(2023)Reconstruction of Machine-Made Shapes from Bitmap SketchesACM Transactions on Graphics10.1145/361836142:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618361
Xiao CSu WLiao JLian ZSong YFu H(2022)DifferSketchingACM Transactions on Graphics10.1145/3550454.355549341:6(1-16)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555493
Hähnlein FLi CMitra NBousseau A(2022)CAD2SketchACM Transactions on Graphics10.1145/3550454.355548841:6(1-18)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555488
Show More Cited By

Index Terms

Interactive Sketch-Based Normal Map Generation with Deep Neural Networks
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
        Shape representations
2. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. User centered design

Recommendations

Sketch2normal: deep networks for normal map generation
SA '17: SIGGRAPH Asia 2017 Posters

Normal maps are of great importance for many 2D graphics applications such as surface editing, re-lighting, texture mapping and 2D shading etc. Automatically inferring normal map is highly desirable for graphics designers. Many researchers have ...
Robust flow-guided neural prediction for sketch-based freeform surface modeling

Sketching provides an intuitive user interface for communicating free form shapes. While human observers can easily envision the shapes they intend to communicate, replicating this process algorithmically requires resolving numerous ambiguities. ...
‘CADSketchNet’ - An Annotated Sketch dataset for 3D CAD Model Retrieval with Deep Neural Networks▪
Highlights
- The goal of this paper is to create a sketch dataset that is suitable for developing deep learning-based solutions to the problem of search and retrieval in 3D CAD models.
- A sketch dataset of query images, called ‘CADSketchNet’ has ...
Graphical abstract

Display Omitted

Abstract
Ongoing advancements in the fields of 3D modelling and digital archiving have led to an outburst in the amount of data stored digitally. Consequently, several retrieval systems have been developed depending on the type of data stored in these ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques

Proceedings of the ACM on Computer Graphics and Interactive Techniques Volume 1, Issue 1

July 2018

378 pages

EISSN:2577-6193

DOI:10.1145/3242771

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2018

Published in PACMCGIT Volume 1, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China
RGC of Hong Kong SAR
Science Foundation of Hunan Province

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
683
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)7

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Puhachov IMartens CKry PBessmeltsev M(2023)Reconstruction of Machine-Made Shapes from Bitmap SketchesACM Transactions on Graphics10.1145/361836142:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618361
Xiao CSu WLiao JLian ZSong YFu H(2022)DifferSketchingACM Transactions on Graphics10.1145/3550454.355549341:6(1-16)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555493
Hähnlein FLi CMitra NBousseau A(2022)CAD2SketchACM Transactions on Graphics10.1145/3550454.355548841:6(1-18)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555488
Kong DWang QQi Y(2022)A Diffusion-ReFinement Model for Sketch-to-Point ModelingComputer Vision – ACCV 202210.1007/978-3-031-26293-7_4(54-70)Online publication date: 4-Dec-2022
https://dl.acm.org/doi/10.1007/978-3-031-26293-7_4
Fan KLiu SLu W(2022)SemiPainter: Learning to Draw Semi-realistic Paintings from the Manga Line Drawings and Flat ShadowAdvances in Computer Graphics10.1007/978-3-031-23473-6_24(305-317)Online publication date: 12-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-23473-6_24
Wang ZQiu SFeng NRushmeier HMcMillan LDorsey J(2021)Tracing versus freehand for evaluating computer-generated drawingsACM Transactions on Graphics10.1145/3450626.345981940:4(1-12)Online publication date: 19-Jul-2021
https://dl.acm.org/doi/10.1145/3450626.3459819
Hughes REdmond CWells LGlencross MZhu LBednarz T(2020)eXplainable AI (XAI)SIGGRAPH Asia 2020 Courses10.1145/3415263.3419166(1-62)Online publication date: 17-Nov-2020
https://dl.acm.org/doi/10.1145/3415263.3419166
Bhunia ADas AMuhammad UYang YHospedales TXiang TGryaditskaya YSong Y(2020)PixelorACM Transactions on Graphics10.1145/3414685.341784039:6(1-15)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417840
Li CPan HBousseau AMitra N(2020)Sketch2CADACM Transactions on Graphics10.1145/3414685.341780739:6(1-14)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417807
Hashimoto MFukusato TIgarashi T(2020)Neurally-Guided Texturing for Garment Line DrawingsSIGGRAPH Asia 2020 Technical Communications10.1145/3410700.3425428(1-4)Online publication date: 1-Dec-2020
https://dl.acm.org/doi/10.1145/3410700.3425428
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents