Article

PalGAN: Image Colorization with Palette Generative Adversarial Networks

Authors:

Yu QiaoAuthors Info & Claims

Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV

Pages 271 - 288

https://doi.org/10.1007/978-3-031-19784-0_16

Published: 23 October 2022 Publication History

Abstract

Multimodal ambiguity and color bleeding remain challenging in colorization. To tackle these problems, we propose a new GAN-based colorization approach PalGAN, integrated with palette estimation and chromatic attention. To circumvent the multimodality issue, we present a new colorization formulation that estimates a probabilistic palette from the input gray image first, then conducts color assignment conditioned on the palette through a generative model. Further, we handle color bleeding with chromatic attention. It studies color affinities by considering both semantic and intensity correlation. In extensive experiments, PalGAN outperforms state-of-the-arts in quantitative evaluation and visual comparison, delivering notable diverse, contrastive, and edge-preserving appearances. With the palette design, our method enables color transfer between images even with irrelevant contexts.

References

[1]

Afifi, M., Brubaker, M.A., Brown, M.S.: HistoGAN: controlling colors of GAN-generated and real images via color histograms. In: CVPR, pp. 7941–7950 (2021)

[2]

Antic, J.: A deep learning based project for colorizing and restoring old images (and video!). https://github.com/jantic/DeOldify

[3]

Bahng H et al. Ferrari V, Hebert M, Sminchisescu C, Weiss Y, et al. Coloring with words: guiding image colorization through text-based palette generation Computer Vision – ECCV 2018 2018 Cham Springer 443-459

Digital Library

[4]

Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)

[5]

Caesar, H., Uijlings, J., Ferrari, V.: COCO-stuff: thing and stuff classes in context. In: CVPR, pp. 1209–1218 (2018)

[6]

Chang H, Fried O, Liu Y, DiVerdi S, and Finkelstein A Palette-based photo recoloring TOG 2015 34 4 139

Digital Library

[7]

Charpiat G, Hofmann M, and Schölkopf B Forsyth D, Torr P, and Zisserman A Automatic image colorization via multimodal predictions Computer Vision – ECCV 2008 2008 Heidelberg Springer 126-139

Digital Library

[8]

Chia AYS et al. Semantic colorization with internet images TOG 2011 30 6 1-8

Digital Library

[9]

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)

[10]

Deshpande, A., Lu, J., Yeh, M.C., Jin Chong, M., Forsyth, D.: Learning diverse image colorization. In: CVPR, pp. 6837–6845 (2017)

[11]

Goodfellow, I., et al.: Generative adversarial nets. In: NeurIPS, pp. 2672–2680 (2014)

[12]

Guadarrama, S., Dahl, R., Bieber, D., Norouzi, M., Shlens, J., Murphy, K.: Pixcolor: pixel recursive colorization. arXiv preprint arXiv:1705.07208 (2017)

[13]

He K, Sun J, and Tang X Guided image filtering TPAMI 2012 35 6 1397-1409

Digital Library

[14]

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

[15]

He M, Chen D, Liao J, Sander PV, and Yuan L Deep exemplar-based colorization TOG 2018 37 4 1-16

[16]

Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS, pp. 6626–6637 (2017)

[17]

Iizuka S, Simo-Serra E, and Ishikawa H Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification TOG 2016 35 4 1-11

Digital Library

[18]

Ironi R, Cohen-Or D, and Lischinski D Colorization by example Render. Tech. 2005 29 201-210

[19]

Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 1125–1134 (2017)

[20]

Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04948 (2018)

[21]

Kim, E., Lee, S., Park, J., Choi, S., Seo, C., Choo, J.: Deep edge-aware interactive colorization against color-bleeding effects. In: ICCV, pp. 14667–14676 (2021)

[22]

Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

[23]

Kumar, M., Weissenborn, D., Kalchbrenner, N.: Colorization transformer. arXiv preprint arXiv:2102.04432 (2021)

[24]

Larsson G, Maire M, and Shakhnarovich G Leibe B, Matas J, Sebe N, and Welling M Learning representations for automatic colorization Computer Vision – ECCV 2016 2016 Cham Springer 577-593

[25]

Larsson, G., Maire, M., Shakhnarovich, G.: Colorization as a proxy task for visual understanding. In: CVPR (2017)

[26]

Lei, C., Chen, Q.: Fully automatic video colorization with self-regularization and diversity. In: CVPR, pp. 3753–3761 (2019)

[27]

Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: SIGGRAPH, pp. 689–694 (2004)

[28]

Li, W., Lin, Z., Zhou, K., Qi, L., Wang, Y., Jia, J.: MAT: mask-aware transformer for large hole image inpainting. In: CVPR, pp. 10758–10768 (2022)

[29]

Liu, X., et al.: Intrinsic colorization. In: SIGGRAPH Asia, pp. 1–9 (2008)

[30]

Liu, X., Yin, G., Shao, J., Wang, X., et al.: Learning to predict layout-to-image conditional convolutions for semantic image synthesis. In: NeurIPS, pp. 570–580 (2019)

[31]

Liu, Z., Wang, Y., Qi, X., Fu, C.W.: Towards implicit text-guided 3d shape generation. In: CVPR, pp. 17896–17906, June 2022

[32]

Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)

[33]

Miyato, T., Koyama, M.: cGANs with projection discriminator. arXiv preprint arXiv:1802.05637 (2018)

[34]

Nguyen, R.M., Price, B., Cohen, S., Brown, M.S.: Group-theme recoloring for multi-image color consistency. In: Computer Graphics Forum, vol. 36, pp. 83–92. Wiley Online Library (2017)

[35]

Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR, pp. 2337–2346 (2019)

[36]

Qi, L., et al.: Open-world entity segmentation. arXiv preprint arXiv:2107.14228 (2021)

[37]

Qu Y, Wong TT, and Heng PA Manga colorization TOG 2006 25 3 1214-1220

Digital Library

[38]

Su, J.W., Chu, H.K., Huang, J.B.: Instance-aware image colorization. In: CVPR, pp. 7968–7977 (2020)

[39]

Tai, Y.W., Jia, J., Tang, C.K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: CVPR, vol. 1, pp. 747–754. IEEE (2005)

[40]

Torralba, A., Freeman, W.T.: Properties and applications of shape recipes (2002)

[41]

Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: CVPR, pp. 8798–8807 (2018)

[42]

Wang Y, Chen Y-C, Tao X, and Jia J Vedaldi A, Bischof H, Brox T, and Frahm J-M VCNet: a robust approach to blind image inpainting Computer Vision – ECCV 2020 2020 Cham Springer 752-768

Digital Library

[43]

Wang, Y., Chen, Y.C., Zhang, X., Sun, J., Jia, J.: Attentive normalization for conditional image generation. In: CVPR, pp. 5094–5103 (2020)

[44]

Wang, Y., Qi, L., Chen, Y.C., Zhang, X., Jia, J.: Image synthesis via semantic composition. In: ICCV, pp. 13749–13758 (2021)

[45]

Wang, Y., Tao, X., Qi, X., Shen, X., Jia, J.: Image inpainting via generative multi-column convolutional neural networks. In: NeurIPS (2018)

[46]

Wang, Y., Tao, X., Shen, X., Jia, J.: Wide-context semantic image extrapolation. In: CVPR (2019)

[47]

Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 277–280 (2002)

[48]

Wu, H., Zheng, S., Zhang, J., Huang, K.: Fast end-to-end trainable guided filter. In: CVPR, pp. 1838–1847 (2018)

[49]

Wu, Y., Wang, X., Li, Y., Zhang, H., Zhao, X., Shan, Y.: Towards vivid and diverse image colorization with generative color prior. In: ICCV, pp. 14377–14386 (2021)

[50]

Xia, M., Wang, Y., Han, C., Wong, T.T.: Enhance convolutional neural networks with noise incentive block. arXiv preprint arXiv:2012.12109 (2020)

[51]

Xu, X., Wang, Y., Wang, L., Yu, B., Jia, J.: Conditional temporal variational autoencoder for action video prediction. arXiv preprint arXiv:2108.05658 (2021)

[52]

Yatziv L and Sapiro G Fast image and video colorization using chrominance blending TIP 2006 15 5 1120-1129

Digital Library

[53]

Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318 (2018)

[54]

Zhang R, Isola P, and Efros AA Leibe B, Matas J, Sebe N, and Welling M Colorful image colorization Computer Vision – ECCV 2016 2016 Cham Springer 649-666

[55]

Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)

[56]

Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. arXiv preprint arXiv:1705.02999 (2017)

[57]

Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2223–2232 (2017)

[58]

Zomet, A., Peleg, S.: Multi-sensor super-resolution. In: Sixth IEEE Workshop on Applications of Computer Vision (WACV), pp. 27–31. IEEE (2002)

Cited By

Li YFu YDong FLiang D(2024)DiffGIC: Diffusion Prior Based Null-Space Correction for High Resolution Grayscale Image ColorizationAdvanced Intelligent Computing Technology and Applications10.1007/978-981-97-5600-1_20(234-245)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1007/978-981-97-5600-1_20
Yan SXu SYang WZhang S(2023)Image recoloring based on fast and flexible palette extractionMultimedia Tools and Applications10.1007/s11042-023-15114-582:30(47793-47810)Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/s11042-023-15114-5

Recommendations

Using assorted color spaces and pixel window sizes for colorization of grayscale images
ICWET '10: Proceedings of the International Conference and Workshop on Emerging Trends in Technology

There is no exact solution for colorization of grayscale images. In the initial work done [23], color traits transfer techniques to color grayscale images are proposed. The main focus of the techniques [23] is to minimise the human efforts needed in ...
Efficient image colorization based on seed pixel selection

Colorization is a technique to automatically produce color components for monochrome images and videos based on a few input colors. Generally, image colorization is initialized from a number of seed pixels whose colors are specified by users, and then ...
Research on Colorization of Qinghai Farmer Painting Image Based on Generative Adversarial Networks
CACML '23: Proceedings of the 2023 2nd Asia Conference on Algorithms, Computing and Machine Learning

At present, deep learning method is widely used in the field of gray image colorization. Qinghai farmer painting has distinct national characteristics. The farmer painting has bright colors, high saturation, chaotic color distribution and low color ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV

Oct 2022

798 pages

ISBN:978-3-031-19783-3

DOI:10.1007/978-3-031-19784-0

Editors:
Shai Avidan
Tel Aviv University, Tel Aviv, Israel
,
Gabriel Brostow
University College London, London, UK
,
Moustapha Cissé
Google AI, Accra, Ghana
,
Giovanni Maria Farinella
University of Catania, Catania, Italy
,
Tal Hassner
Facebook (United States), Menlo Park, CA, USA

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 October 2022

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li YFu YDong FLiang D(2024)DiffGIC: Diffusion Prior Based Null-Space Correction for High Resolution Grayscale Image ColorizationAdvanced Intelligent Computing Technology and Applications10.1007/978-981-97-5600-1_20(234-245)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1007/978-981-97-5600-1_20
Yan SXu SYang WZhang S(2023)Image recoloring based on fast and flexible palette extractionMultimedia Tools and Applications10.1007/s11042-023-15114-582:30(47793-47810)Online publication date: 12-May-2023
https://dl.acm.org/doi/10.1007/s11042-023-15114-5

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents