research-article

Perceptual-Aware Sketch Simplification Based on Integrated VGG Layers

Authors:

Huaidong Zhang,

Tien-Tsin WongAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 27, Issue 1

Pages 178 - 189

https://doi.org/10.1109/TVCG.2019.2930512

Published: 01 January 2021 Publication History

Abstract

Deep learning has been recently demonstrated as an effective tool for raster-based sketch simplification. Nevertheless, it remains challenging to simplify extremely rough sketches. We found that a simplification network trained with a simple loss, such as pixel loss or discriminator loss, may fail to retain the semantically meaningful details when simplifying a very sketchy and complicated drawing. In this paper, we show that, with a well-designed multi-layer perceptual loss, we are able to obtain aesthetic and neat simplification results preserving semantically important global structures as well as fine details without blurriness and excessive emphasis on local structures. To do so, we design a multi-layer discriminator by fusing all VGG feature layers to differentiate sketches and clean lines. The weights used in layer fusing are automatically learned via an intelligent adjustment mechanism. Furthermore, to evaluate our method, we compare our method to state-of-the-art methods through multiple experiments, including visual comparison and intensive user study.

References

[1]

B. Wilson and K.-L. Ma, “Rendering complexity in computer- generated pen-and-ink illustrations,” in Proc. 3rd Int. Symp. Non-Photorealistic Animation Rendering, 2004, pp. 129–137.

[2]

Y. Qi, Y. Z. Song, T. Xiang, H. Zhang, T. Hospedales, Y. Li, and J. Guo, “Making better use of edges via perceptual grouping,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1856–1865.

[3]

X. Liu, T.-T. Wong, and P.-A. Heng, “Closure-aware sketch simplification,” ACM Trans. Graph., vol. 34, no. 6, 2015, Art. no.

[4]

J.-D. Favreau, F. Lafarge, and A. Bousseau, “Fidelity vs. simplicity: A global approach to line drawing vectorization,” ACM Trans. Graph., vol. 35, no. 4, 2016, Art. no.

Digital Library

[5]

E. Simo-Serra, S. Iizuka, K. Sasaki, and H. Ishikawa, “Learning to simplify: Fully convolutional networks for rough sketch cleanup,” ACM Trans. Graph., vol. 35, no. 4, 2016, Art. no.

[6]

E. Simo-Serra, S. Iizuka, and H. Ishikawa, “Mastering sketching: Adversarial augmentation for structured prediction,” ACM Trans. Graph., vol. 37, no. 1, 2018, Art. no.

[7]

J. Johnson, A. Alahi, and F. F. Li, “Perceptual losses for real-time style transfer and super-resolution,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 694–711.

[8]

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. 3rd Int. Conf. Learn. Representations, San Diego, CA, USA, May 7–9, 2015. [Online]. Available: http://arxiv.org/abs/1409.1556

[9]

C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 4681–4690.

[10]

R. A. Yeh, C. Chen, T. Yian Lim, A. G. Schwing, M. Hasegawa-Johnson, and M. N. Do, “Semantic image inpainting with deep generative models,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 6882–6890.

[11]

Q. Yang, P. Yan, M. K. Kalra, and G. Wang, “CT image denoising with perceptive deep neural networks,” CoRR, vol. abs/1702.07019, 2017. [Online]. Available: http://arxiv.org/abs/1702.07019

[12]

P. Sangkloy, J. Lu, C. Fang, F. Yu, and J. Hays, “Scribbler: Controlling deep image synthesis with sketch and color,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 6836–6845.

[13]

P. Zhang, D. Wang, H. Lu, H. Wang, and R. Xiang, “Amulet: Aggregating multi-level convolutional features for salient object detection,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 202–211.

[14]

L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–848, Apr. 2018.

[15]

J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 640–651, Apr. 2017.

Digital Library

[16]

B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik, “Hypercolumns for object segmentation and fine-grained localization,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 447–456.

[17]

G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 577–593.

[18]

S. H. Bae, R. Balakrishnan, and K. Singh, “ILoveSketch: As-natural-as-possible sketching system for creating 3D curve models,” in Proc. ACM Symp. User Interface Softw. Technol., 2008, pp. 151–160.

[19]

J. Arvo and K. Novins, “Fluid sketches: Continuous recognition and morphing of simple hand-drawn shapes,” in Proc. ACM Symp. User Interface Softw. Technol., 2000, pp. 73–80.

[20]

B. Preim and T. Strothotte, “Tuning rendered line-drawings,” in Proc. Winter School Comput. Graph., 1995, pp. 228–238.

[21]

S. Grabli, F. Durand, and F. X. Sillion, “Density measure for line-drawing simplification,” in Proc. 12th Pacific Conf. Comput. Graph. Appl., 2004, pp. 309–318.

[22]

A. Shesh and B. Chen, “Efficient and dynamic simplification of line drawings,” Comput. Graph. Forum, vol. 27, pp. 537–545, 2008.

[23]

X. Hilaire and K. Tombre, “Robust and accurate vectorization of line drawings,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 6, pp. 890–904, Jun. 2006.

Digital Library

[24]

G. Noris, A. Hornung, R. W. Sumner, M. Simmons, and M. Gross, “Topology-driven vectorization of clean line drawings,” ACM Trans. Graph., vol. 32, no. 1, 2013, Art. no.

[25]

C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang, and H. Li, “High-resolution image inpainting using multi-scale neural patch synthesis,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 4076–4084.

[26]

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc. Int. Conf. Med. Image Comput. Comput.-Assisted Intervention, 2015, pp. 234–241.

[27]

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 586–595.

[28]

P. Sangkloy, N. Burnell, C. Ham, and J. Hays, “The sketchy database: Learning to retrieve badly drawn bunnies,” ACM Trans. Graph., vol. 35, no. 4, 2016, Art. no.

[29]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. 27th Int. Conf. Neural Inf. Process. Syst., 2014, pp. 2672–2680.

[30]

R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 649–666.

[31]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.

Cited By

Seo CAshtari ANoh J(2023)Semi-supervised reference-based sketch extraction using a contrastive learning frameworkACM Transactions on Graphics10.1145/359239242:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592392
Liu CAoki TBessmeltsev MSheffer A(2023)StripMaker: Perception-driven Learned Vector Sketch ConsolidationACM Transactions on Graphics10.1145/359213042:4(1-15)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592130
Zhang YLiang WYuan XZhang SYang GZeng Z(2023)Deep Learning-Based Abnormal Behavior Detection for Elderly Healthcare Using Consumer Network CamerasIEEE Transactions on Consumer Electronics10.1109/TCE.2023.330985270:1(2414-2422)Online publication date: 29-Aug-2023
https://dl.acm.org/doi/10.1109/TCE.2023.3309852
Show More Cited By

Index Terms

Perceptual-Aware Sketch Simplification Based on Integrated VGG Layers
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Closure-aware sketch simplification

In this paper, we propose a novel approach to simplify sketch drawings. The core problem is how to group sketchy strokes meaningfully, and this depends on how humans understand the sketches. The existing methods mainly rely on thresholding low-level ...
A new sketch based interface using the gray-level co-occurrence matrix for perceptual simplification of paper based scribbles
SBM'06: Proceedings of the Third Eurographics conference on Sketch-Based Interfaces and Modeling

The sketching activity has an important role in conceptual design and a variety of tools exist which help designers to facilitate the generation of 3D models form sketched drawings. This paper describes a new sketch-to-3D tool, which uses annotations to ...
Universal Sketch Perceptual Grouping
Computer Vision – ECCV 2018
Abstract
In this work we aim to develop a universal sketch grouper. That is, a grouper that can be applied to sketches of any category in any domain to group constituent strokes/segments into semantically meaningful object parts. The first obstacle to this ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 27, Issue 1

Jan. 2021

260 pages

ISSN:1077-2626

Issue’s Table of Contents

1077-2626 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 January 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Seo CAshtari ANoh J(2023)Semi-supervised reference-based sketch extraction using a contrastive learning frameworkACM Transactions on Graphics10.1145/359239242:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592392
Liu CAoki TBessmeltsev MSheffer A(2023)StripMaker: Perception-driven Learned Vector Sketch ConsolidationACM Transactions on Graphics10.1145/359213042:4(1-15)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592130
Zhang YLiang WYuan XZhang SYang GZeng Z(2023)Deep Learning-Based Abnormal Behavior Detection for Elderly Healthcare Using Consumer Network CamerasIEEE Transactions on Consumer Electronics10.1109/TCE.2023.330985270:1(2414-2422)Online publication date: 29-Aug-2023
https://dl.acm.org/doi/10.1109/TCE.2023.3309852
Reisfeld ESharf A(2023)OneSketch: learning high-level shape features from simple sketchesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02494-239:7(2811-2822)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1007/s00371-022-02494-2
De CDong YXiong ZXian D(2022)Accurate Spatial Positioning of Target Images in Visual Servo System under Multiple ConstraintsProceedings of the 2022 5th International Conference on Machine Learning and Natural Language Processing10.1145/3578741.3578818(366-371)Online publication date: 23-Dec-2022
https://dl.acm.org/doi/10.1145/3578741.3578818
Ashtari ASeo CKang CCha SNoh J(2022)Reference Based Sketch Extraction via Attention MechanismACM Transactions on Graphics10.1145/3550454.355550441:6(1-16)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555504
Li WZeng YZhang QWu YChen G(2021)Human Motion Capture Based on Incremental Dimension Reduction and Projection Position OptimizationWireless Communications & Mobile Computing10.1155/2021/55891002021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/5589100
Puhachov INeveu WChien EBessmeltsev M(2021)Keypoint-driven line drawing vectorization via PolyVector flowACM Transactions on Graphics10.1145/3478513.348052940:6(1-17)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480529
Mo HSimo-Serra EGao CZou CWang R(2021)General virtual sketching framework for vector line artACM Transactions on Graphics10.1145/3450626.345983340:4(1-14)Online publication date: 19-Jul-2021
https://dl.acm.org/doi/10.1145/3450626.3459833
Zhou YShimada N(2021)Generative Adversarial Network for Text-to-Face Synthesis and Manipulation with Pretrained BERT Model2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)10.1109/FG52635.2021.9666791(01-08)Online publication date: 15-Dec-2021
https://dl.acm.org/doi/10.1109/FG52635.2021.9666791
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents