research-article

Painterly Style Transfer With Learned Brush Strokes

Authors:

Xiao-Chang Liu,

Peter HallAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 30, Issue 9

Pages 6309 - 6320

https://doi.org/10.1109/TVCG.2023.3332950

Published: 01 September 2024 Publication History

Abstract

Real-world paintings are made, by artists, using brush strokes as the rendering primitive to depict semantic content. The bulk of the Neural Style Transfer (NST) is known transferring style using texture patches, not strokes. The output looks like the content image, but some are traced over using the style texture: it does not look painterly. We adopt a very different approach that uses strokes. Our contribution is to analyse paintings to learn <italic>stroke families</italic>—that is, distributions of strokes based on their shape (a dot, straight lines, curved arcs, etc.). When synthesising a new output, these distributions are sampled to ensure the output is painted with the correct style of stroke. Consequently, our output looks more “painterly” than NST output based on texture. Furthermore, where strokes are placed is an important contributing factor in determining output quality, and we have also addressed this aspect. Humans place strokes to emphasize salient semantically meaningful image content. Conventional NST uses a content loss premised on filter responses that is agnostic to salience. We show that replacing that loss with one based on the language-image model benefits the output through greater emphasis of salient content.

References

[1]

L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2016, pp. 2414–2423.

[2]

D. Kotovenko, M. Wright, A. Heimbrecht, and B. Ommer, “Rethinking style transfer: From pixels to parameterized brushstrokes,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2021, pp. 12191–12200.

[3]

G. Winkenbach and D. H. Salesin, “Computer-generated pen-and-ink illustration,” in Proc. 21st Annu. Conf. Comput. Graph. Interactive Techn., 1994, pp. 91–100.

[4]

G. D. Blasi and G. Gallo, “Artificial mosaics,” Vis. Comput., vol. 21, pp. 373–383, 2005.

[5]

D. Mould, “A stained glass image filter,” in Proc. 14th Eurographics Workshop Rendering, 2003, pp. 20–25.

[6]

Y.-Z. Song, D. Pickup, C. Li, P. Rosin, and P. Hall, “Abstract art by shape classification,” IEEE Trans. Vis. Comput. Graph., vol. 19, no. 8, pp. 1252–1263, Aug. 2013.

Digital Library

[7]

S. Brooks, “Mixed media painting and portraiture,” IEEE Trans. Vis. Comput. Graph., vol. 13, no. 5, pp. 1041–1054, Sep./Oct. 2007.

Digital Library

[8]

H. Kang, S. Lee, and C. K. Chui, “Flow-based image abstraction,” IEEE Trans. Vis. Comput. Graph., vol. 15, no. 1, pp. 62–76, Jan./Feb. 2009.

Digital Library

[9]

A. Hertzmann, “Non-photorealistic rendering and the science of art,” in Proc. 8th Int. Symp. Non-Photorealistic Animation Rendering, 2010, pp. 147–157.

[10]

A. Hertzmann and K. Perlin, “Painterly rendering for video and interaction,” in Proc. 1st Int. Symp. Non-Photorealistic Animation Rendering, 2000, pp. 7–12.

[11]

C.-R. Yan, M.-T. Chi, T.-Y. Lee, and W.-C. Lin, “Stylized rendering using samples of a painted image,” IEEE Trans. Vis. Comput. Graph., vol. 14, no. 2, pp. 468–480, Mar./Apr. 2008.

Digital Library

[12]

J. E. Kyprianidis, J. Collomosse, T. Wang, and T. Isenberg, “State of the “art”: A taxonomy of artistic stylization techniques for images and video,” IEEE Trans. Visual. Comput. Graph., vol. 19, no. 5, pp. 866–885, May 2013.

Digital Library

[13]

A. Finkelstein and D. H. Salesin, “Multiresolution curves,” in Proc. SIGGRAPH, 1994, pp. 261–268.

[14]

W. T. Freeman, J. B. Tenenbaum, and E. C. Pasztor, “Learning style translation for the lines of a drawing,” ACM Trans. Graph., vol. 22, no. 1, pp. 33–46, 2003.

Digital Library

[15]

J. Lu, F. Yu, A. Finkelstein, and S. DiVerdi, “HelpingHand: Example-based stroke stylization,” ACM Trans. Graph., vol. 31, no. 4, pp. 46:1–46:10, 2012.

[16]

K. Lang and M. Alexa, “The Markov pen: Online synthesis of free-hand drawing styles,” in Proc. Workshop Non-Photorealistic Animation Rendering, 2015, pp. 203–215.

[17]

Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song, “Neural style transfer: A review,” IEEE Trans. Vis. Comput. Graph., vol. 26, no. 11, pp. 3365–3385, Nov. 2020.

[18]

Y. Yao, J. Ren, X. Xie, W. Liu, Y.-J. Liu, and J. Wang, “Attention-aware multi-stroke style transfer,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 1467–1475.

[19]

A. Radford et al., “Learning transferable visual models from natural language supervision,” in Proc. Int. Conf. Mach. Learn., 2021, pp. 8748–8763.

[20]

B. Gooch and A. Gooch, Non-Photorealistic Rendering. Boca Raton, FL, USA: AK Peters/CRC Press, 2001.

[21]

J. Lansdown and S. Schofield, “Expressive rendering: A review of nonphotorealistic techniques,” IEEE Comput. Graph. Appl., vol. 15, no. 3, pp. 29–37, May 1995.

Digital Library

[22]

P. Haeberli, “Paint by numbers: Abstract image representations,” in Proc. 17th Annu. Conf. Comput. Graph. Interactive Techn., 1990, pp. 207–214.

[23]

A. Hertzmann, “Painterly rendering with curved brush strokes of multiple sizes,” in Proc. 25th Annu. Conf. Comput. Graph. Interactive Techn., 1998, pp. 453–460.

[24]

A. Hertzmann, “A survey of stroke-based rendering,” IEEE Comput. Graph. Appl., vol. 23, no. 4, pp. 70–81, Jul./Aug. 2003.

Digital Library

[25]

M. P. Salisbury, S. E. Anderson, R. Barzel, and D. H. Salesin, “Interactive pen-and-ink illustration,” in Proc. 21st Annu. Conf. Comput. Graph. Interactive Techn., 1994, pp. 393–400.

[26]

P. Litwinowicz, “Processing images and video for an impressionist effect,” in Proc. 24th Annu. Conf. Comput. Graph. Interactive Techn., 1997, pp. 407–414.

[27]

B. Gooch, G. Coombe, and P. Shirley, “Artistic vision: Painterly rendering using computer vision techniques,” in Proc. 2nd Int. Symp. Non-Photorealistic Animation Rendering, 2002, pp. 83–ff.

[28]

A. Hausner, “Simulating decorative mosaics,” in Proc. 28th Annu. Conf. Comput. Graph. Interactive Techn., 2001, pp. 573–580.

[29]

A. Hertzmann, “Paint by relaxation,” in Proc. IEEE Comput. Graph. Int., 2001, pp. 47–54.

[30]

A. Secord, “Weighted voronoi stippling,” in Proc. 2nd Int. Symp. Non-Photorealistic Animation Rendering, 2002, pp. 37–43.

[31]

D. Ha and D. Eck, “A neural representation of sketch drawings,” in Proc. Int. Conf. Learn. Representations, 2018, pp. 1–16.

[32]

Z. Huang, W. Heng, and S. Zhou, “Learning to paint with model-based deep reinforcement learning,” in Proc. Int. Conf. Comput. Vis., 2019, pp. 8709–8718.

[33]

S. Liu et al., “Paint transformer: Feed forward neural painting with stroke prediction,” in Proc. Int. Conf. Comput. Vis., 2021, pp. 6598–6607.

[34]

P. Schaldenbrand and J. Oh, “Content masked loss: Human-like brush stroke planning in a reinforcement learning painting agent,” in Proc. AAAI Conf. Artif. Intell., 2021, pp. 505–512.

[35]

Z. Zou, T. Shi, S. Qiu, Y. Yuan, and Z. Shi, “Stylized neural painting,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2021, pp. 15684–15693.

[36]

Z. Tong, X. Wang, S. Yuan, X. Chen, J. Wang, and X. Fang, “Im2Oil: Stroke-based oil painting rendering with linearly controllable fineness via adaptive sampling,” in Proc. 30th ACM Int. Conf. Multimedia, 2022, pp. 1035–1046.

[37]

J. Singh and L. Zheng, “Combining semantic guidance and deep reinforcement learning for generating human level paintings,” IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2021, pp. 16387–16396.

[38]

P. Schaldenbrand, J. McCann, and J. Oh, “FRIDA: A collaborative robot painter with a differentiable, Real2Sim2Real planning environment,” in Proc. IEEE Int. Conf. Robot. Automat., 2023, pp. 11712–11718.

[39]

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. Int. Conf. Learn. Representations, 2015, pp. 1–14.

[40]

L. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2015, pp. 262–270.

[41]

J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 694–711.

[42]

X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proc. Int. Conf. Comput. Vis., 2017, pp. 1501–1510.

[43]

D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2017, pp. 4105–4113.

[44]

X.-C. Liu, M.-M. Cheng, Y.-K. Lai, and P. L. Rosin, “Depth-aware neural style transfer,” in Proc. Symp. Non-Photorealistic Animation Rendering, 2017, pp. 1–10.

[45]

Y. Jing et al., “Stroke controllable fast style transfer with adaptive receptive fields,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 244–260.

[46]

S. S. Y. Kim, N. Kolkin, J. Salavon, and G. Shakhnarovich, “Deformable style transfer,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 246–261.

[47]

X.-C. Liu, Y.-L. Yang, and P. Hall, “Learning to warp for style transfer,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2021, pp. 3701–3710.

[48]

L. Höllein, J. Johnson, and M. Nießner, “StyleMesh: Style transfer for indoor 3D scene reconstructions,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 6188–6198.

[49]

F. Han, S. Ye, M. He, M. Chai, and J. Liao, “Exemplar-based 3D portrait stylization,” IEEE Trans. Vis. Comput. Graph., vol. 29, no. 2, pp. 1371–1383, Feb. 2023.

[50]

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. Int. Conf. Comput. Vis., 2017, pp. 2242–2251.

[51]

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2017, pp. 5967–5976.

[52]

L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman, “Controlling perceptual factors in neural style transfer,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2017, pp. 3730–3738.

[53]

N. Kolkin, J. Salavon, and G. Shakhnarovich, “Style transfer by relaxed optimal transport and self-similarity,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2019, pp. 10043–10052.

[54]

V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” in Proc. Int. Conf. Learn. Representations, 2017, pp. 1–11.

[55]

D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua, “Stylebank: An explicit representation for neural image style transfer,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2017, pp. 2770–2779.

[56]

A. Sanakoyeu, D. Kotovenko, S. Lang, and B. Ommer, “A style-aware content loss for real-time HD style transfer,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 698–714.

[57]

M.-M. Cheng, X.-C. Liu, J. Wang, S.-P. Lu, Y.-K. Lai, and P. L. Rosin, “Structure-preserving neural style transfer,” IEEE Trans. Image Process., vol. 29, pp. 909–920, 2019.

Digital Library

[58]

I. E. Berezhnoy, E. O. Postma, and H. J. van den Herik, “Automatic extraction of brushstroke orientation from paintings: Poet: Prevailing orientation extraction technique,” Mach. Vis. Appl., vol. 20, pp. 1–9, 2009.

Digital Library

[59]

J. Li, L. Yao, E. Hendriks, and J. Z. Wang, “Rhythmic brushstrokes distinguish van gogh from his contemporaries: Findings via automated brushstroke extraction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 6, pp. 1159–1176, Jun. 2012.

Digital Library

[60]

Y. Fu, H. Yu, C.-K. Yeh, T.-Y. Lee, and J. J. Zhang, “Fast accurate and automatic brushstroke extraction,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 17, no. 2, pp. 1–24, 2021.

Digital Library

[61]

T. Putri, R. Mukundan, and K. Neshatian, “Artistic style characterization and brush stroke modelling for non-photorealistic rendering,” in Proc. IEEE Int. Conf. Image Vis. Comput., 2017, pp. 1–7.

[62]

S. Shaheen, A. Rockwood, and B. Ghanem, “SAR: Stroke authorship recognition,” in Computer Graphics Forum, vol. 35. Hoboken, NJ, USA: Wiley, 2016, pp. 73–86.

[63]

A. Lecoutre, B. Negrevergne, and F. Yger, “Recognizing art style automatically in painting with deep learning,” in Proc. Asian Conf. Mach. Learn., 2017, pp. 327–342.

[64]

Y. Zang, H. Huang, and C.-F. Li, “Stroke style analysis for painterly rendering,” J. Comput. Sci. Technol., vol. 28, no. 5, pp. 762–775, 2013.

[65]

J. Collomosse and P. Hall, “Painterly rendering using image salience,” in Proc. 20th Eurographics U.K. Conf., 2002, pp. 122–128.

[66]

J. P. Collomosse and P. M. Hall, “Salience-adaptive painterly rendering using genetic search,” Int. J. Artif. Intell. Tools, vol. 15, no. 4, pp. 551–575, 2006.

[67]

T. Lindemeier, S. Pirk, and O. Deussen, “Image stylization with a painting machine using semantic hints,” Comput. Graph., vol. 37, no. 5, pp. 293–301, 2013.

Digital Library

[68]

M. Wang et al., “Towards photo watercolorization with artistic verisimilitude,” IEEE Trans. Visual. Comput. Graph., vol. 20, no. 10, pp. 1451–1460, Oct. 2014.

[69]

L. Kurzman, D. Vazquez, and I. Laradji, “Class-based styling: Real-time localized style transfer with semantic segmentation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshops, 2019, pp. 3189–3192.

[70]

Y.-S. Liao and C.-R. Huang, “Semantic context-aware image style transfer,” IEEE Trans. Image Process., vol. 31, pp. 1911–1923, 2022.

[71]

A. J. Champandard, “Semantic style transfer and turning two-bit doodles into fine artworks,” 2016,.

[72]

H.-H. Zhao, P. L. Rosin, Y.-K. Lai, and Y.-N. Wang, “Automatic semantic style transfer using deep convolutional neural networks and soft masks,” Vis. Comput., vol. 36, pp. 1307–1324, 2020.

Digital Library

[73]

Y.-L. Chen and C.-T. Hsu, “Towards deep style transfer: A content-aware perspective,” in Proc. Brit. Mach. Vis. Conf., 2016, pp. 1–11.

[74]

J. Singh, C. Smith, J. Echevarria, and L. Zheng, “Intelli-paint: Towards developing human-like painting agents,” in Proc. Eur. Conf. Comput. Vis., 2022, pp. 685–701.

[75]

C. Chan, F. Durand, and P. Isola, “Learning to generate line drawings that convey geometry and semantics,” in Proc. IEEE/CVF Comput. Vis. Pattern Recognit. Conf., 2022, pp. 7905–7915.

[76]

H. Mo, E. Simo-Serra, C. Gao, C. Zou, and R. Wang, “General virtual sketching framework for vector line art,” ACM Trans. Graph., vol. 40, no. 4, pp. 1–14, 2021.

Digital Library

[77]

J. Gravesen, “Adaptive subdivision and the length and energy of bézier curves,” Comput. Geometry, vol. 8, no. 1, pp. 13–31, 1997.

Digital Library

[78]

Y. Vinker et al., “CLIPasso: Semantically-aware object sketching,” ACM Trans. Graph., vol. 41, no. 4, pp. 1–11, 2022.

Digital Library

[79]

K. Frans, L. B. Soros, and O. Witkowski, “CLIPDraw: Exploring text-to-drawing synthesis through language-image encoders,” in Proc. Conf. Neural Inf. Process. Syst., 2022, pp. 5207–5218.

[80]

Y. Deng et al., “StyTr2: Image style transfer with transformers,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 11326–11336.

[81]

X. Qin, Z. Zhang, C. Huang, M. Dehghan, O. R. Zaiane, and M. Jagersand, “U2-Net: Going deeper with nested u-structure for salient object detection,” Pattern Recognit., vol. 106, 2020, Art. no.

[82]

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.

Digital Library

[83]

T. Lüddecke and A. Ecker, “Image segmentation using text and image prompts,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 7076–7086.

[84]

T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, pp. 18392–18402.

[85]

G. Kwon and J. C. Ye, “CLIPstyler: Image style transfer with a single text condition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 18041–18050.

Cited By

Wang QDai HYang JGuo CChilds PKleinsmann MGuo YWang P(2024)Learning-based Artificial Intelligence Artwork: Methodology Taxonomy and Quality EvaluationACM Computing Surveys10.1145/369810557:3(1-37)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3698105

Index Terms

Painterly Style Transfer With Learned Brush Strokes
1. Applied computing
  1. Arts and humanities
    1. Fine arts
    2. Performing arts
2. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Brush Writing Style Classification from Individual Chinese Characters
ICPR '06: Proceedings of the 18th International Conference on Pattern Recognition - Volume 01

Chinese calligraphic artwork usually contains a few or even a single character. In order to perform automatic brush writing style classification on these kinds of Chinese calligraphic images, existing approaches need to be modified because they assume ...
Stroke-Based Few-Shot Chinese Character Style Transfer
Advanced Intelligent Computing Technology and Applications
Abstract
Few-shot Chinese character style transfer aims to learn the style of Chinese characters with limited references and then generate a complete set of Chinese character fonts. This not only solves the repetitive and cumbersome manual effort in ...
Painterly rendering with content-dependent natural paint strokes

We present a new painterly rendering method that simulates artists’ content-dependent painting process and the natural variation of hand-painted strokes. First, a new stroke layout strategy is proposed to enhance the contrast between large and small ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 30, Issue 9

Sept. 2024

704 pages

Issue’s Table of Contents

1077-2626 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 September 2024

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang QDai HYang JGuo CChilds PKleinsmann MGuo YWang P(2024)Learning-based Artificial Intelligence Artwork: Methodology Taxonomy and Quality EvaluationACM Computing Surveys10.1145/369810557:3(1-37)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3698105

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents