skip to main content
10.1145/3449365.3449379acmotherconferencesArticle/Chapter ViewAbstractPublication PagesapitConference Proceedingsconference-collections
research-article

Generative Image Inpainting for Large-Scale Edge Area

Published: 17 May 2021 Publication History

Abstract

In recent years, applying deep learning to computer vision is a very popular research direction, and a number of models with amazing effects have appeared. Deep learning-based approaches for end-to-end image inpainting have shown promise results. Recent research has made great progress in repairing rectangular and free-form areas but there are still many problems and room for improvement. For example, artifacts, blur and color missing still exist among the completion results of the large-scale border area. In this paper, we propose an end-to-end GAN-based image inpainting method, which has a better effect on the large boundary area. Our model is a two-stage adversarial network. The first stage completes the corresponding edge image, and the second stage uses the edge image generated in the first stage as a prior to complete the color image. We added parallel residual blocks to the edge completion network, and for the image completion network we replace the original residual blocks with multi-scale dilated convolution fusion blocks. Besides, a content loss based on DenseNet is added to the second stage. Experiments on multiple publicly available datasets show that our results have better effects on larger edge areas and can increase the average PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index).

References

[1]
C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera. Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing, 10(8):1200–1211, 2001. 2
[2]
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pages 417–424, 2000. 2
[3]
S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman, and P. Sen. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on graphics (TOG), 31(4):82–1, 2012. 1, 2
[4]
J.-B. Huang, S. B. Kang,N. Ahuja, and J. Kopf. Image completion using planar structure guidance. ACM Transactions on graphics (TOG), 33(4):129, 2014. 1, 2
[5]
C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. Patch match: A randomized correspondence algorithm for structural image editing. ACM Transactions on graphics (TOG), 28(3):24, 2009. 2
[6]
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR),pages2536–2544,2016. 2
[7]
S. Iizuka, E. Simo-Serra, and H. Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 36(4):107, 2017. 2, 5, 6, 13
[8]
F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. International Conference on Learning Representations (ICLR), 2016. 2
[9]
C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang, and H. Li. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2
[10]
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 2
[11]
J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3, 5, 6, 13
[12]
G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro. Image inpainting for irregular holes using partial convolutions. In European Conference on Computer Vision (ECCV), September 2018. 2, 5, 6
[13]
J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv:1806.03589, 2018.
[14]
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edge connect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212, 2019. 2, 3, 7, 14
[15]
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3, 11
[16]
J. -Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In The IEEE International Conference on Computer Vision (ICCV), 2017. 3, 11
[17]
G. Huang, Z. Liu, L. v. d. Maaten and K. Q. Weinberger, "Densely Connected Convolutional Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017,pp.2261-2269.
[18]
L. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems, pages 262–270, 2015. 4
[19]
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein Gan. arXiv preprint arXiv:1701.07875, 2017. 4
[20]
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. 4
[21]
A. Odena, J. Buckman, C. Olsson, T. B. Brown, C. Olah, C. Raffel, and I. Goodfellow. Is generator conditioning causally related to gan performance? In Proceedings of the 35th International Conference on Machine Learning, 2018. 4
[22]
H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Selfattention generative adversarial networks. arXiv preprint arXiv:1805.08318, 2018. 4, 6
[23]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. 5
[24]
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015. 4
[25]
L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2414–2423, 2016. 4
[26]
J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (ECCV), pages 694–711. Springer, 2016. 3, 4, 11
[27]
M. S. M. Sajjadi, B. Scholkopf, and M. Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In The IEEE International Conference on Computer Vision (ICCV), pages 4501–4510. IEEE, 2017. 3, 4
[28]
A. Odena, V. Dumoulin, and C. Olah. Deconvolution and checkerboard artifacts. Distill, 1(10):e3, 2016. 4

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
APIT '21: Proceedings of the 2021 3rd Asia Pacific Information Technology Conference
January 2021
140 pages
ISBN:9781450388108
DOI:10.1145/3449365
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 May 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Edge
  2. Generative Adversarial Networks
  3. Inpainting

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

APIT 2021

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 56
    Total Downloads
  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)0
Reflects downloads up to 21 Dec 2024

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media