Nothing Special   »   [go: up one dir, main page]
Skip to main content
SpringerLink
Log in

SFI-Swin: symmetric face inpainting with swin transformer by distinctly learning face components distributions

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Image inpainting consists of filling holes or missing parts of an image. Inpainting face images with symmetric characteristics is more challenging than inpainting a natural scene. None of the powerful existing models can fill out the missing parts of an image while considering the symmetry and homogeneity of the picture. Moreover, the metrics that assess a repaired face image quality cannot measure the preservation of symmetry between the rebuilt and existing parts of a face. In this paper, we intend to solve the symmetry problem in the face inpainting task by using multiple discriminators that check each face organ’s reality separately and a transformer-based network. We also propose "symmetry concentration score" as a new metric for measuring the symmetry of a repaired face image. The quantitative and qualitative results show the superiority of our proposed method compared to some of the recently proposed algorithms in terms of the reality, symmetry, and homogeneity of the inpainted parts. The code for the proposed method is available at https://github.com/mohammadrezanaderi4/SFI-Swin.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The dataset used during the current study is available in the GitHub repository, https://github.com/advimman/lama

References

  1. Zhao S et al (2021) Large scale image completion via co-modulated generative adversarial networks. ArXiv Prepr:ArXiv2103.10428

  2. Zeng Y, Fu J, Chao H, Guo B (2022) Aggregated contextual transformations for high-resolution image inpainting. IEEE Trans Vis Comput Graph 29(7):3266–3280

  3. Ma Y et al (2022) Region-wise generative adversarial image inpainting for large missing areas. IEEE Trans Cybern 53(8):5226–5239

  4. Nazeri K, Ng E, Joseph T, Qureshi FZ, Ebrahimi M (2019) Edgeconnect:generative image inpainting with adversarial edge learning. ArXiv Prepr:ArXiv1901.00212

  5. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2019) Free-form image inpainting with gated convolution. In: Proceedings of the IEEE Conference on Computer Vision, pp 4471–4480

  6. Suvorov R et al (2022) Resolution-robust large mask inpainting with Fourier convolutions. In: Proceedings IEEE/CVF Winter Conference on Applications of Computer Vision, pp 2149–2159

  7. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1125–1134

  8. Liu Z et al (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 10012–10022

  9. Zhang X, Zhai D, Li T, Zhou Y, Lin Y (2022) Image inpainting based on deep learning: a review. Inf Fusion 90:74–94

  10. Qin Z, Zeng Q, Zong Y, Xu F (2021) Image inpainting based on deep learning: A review. Displays 69:102028

    Article  Google Scholar 

  11. Jam J, Kendrick C, Walker K, Drouard V, Hsu JG-S, Yap MH (2021) A comprehensive review of past and present image inpainting methods. Comput Vis image Underst 203:103147

    Article  Google Scholar 

  12. Su S, Yang M, He L, Shao X, Zuo Y, Qiang Z (2022) A survey of face image inpainting based on deep learning. In: Proceedings of Cloud Computing, pp 72–87

  13. Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):24

    Article  Google Scholar 

  14. Criminisi A, Pérez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans image Process 13(9):1200–1212

    Article  Google Scholar 

  15. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5505–5514

  16. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2536–2544

  17. Zhao Z et al (2021) Prior based human completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7951–7961

  18. Hui Z, Li J, Wang X, Gao X (2020) Image fine-grained inpainting. ArXiv Prepr:ArXiv2002.02609

  19. Zhou Y, Barnes C, Shechtman E, Amirghodsi S (2021) Transfill: reference-guided image inpainting by merging multiple color and spatial transformations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2266–2276

  20. Dosovitskiy A et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. ArXiv Prepr:ArXiv2010.11929

  21. Yuan L et al (2021) Tokens-to-token vit: training vision transformers from scratch on imagenet. In: Proceedings of the IEEE Conference on Computer Vision, pp 558–567

  22. Yin H, Vahdat A, Alvarez JM, Mallya A, Kautz J, Molchanov P (2022) A-ViT: adaptive tokens for efficient vision transformer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 10809–10818

  23. Han K et al (2022) A survey on vision transformer. IEEE Trans Pattern Anal Mach Intell, 45(1): 87–110

  24. Wan Z, Zhang J, Chen D, Liao J (2021) High-fidelity pluralistic image completion with transformers. In: Proceedings IEEE/CVF Conferene Computer Vision, pp 4692–4701

  25. Zheng C, Cham TJ, Cai J (2021) Tfill: image completion via a transformer-based architecture. ArXiv Prepr:ArXiv2104.00845

  26. Li W, Lin Z, Zhou K, Qi L,Wang Y, Jia J (2022) MAT: mask-aware transformer for large hole image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10758–10768

  27. Dong Q, Cao C, Fu Y (2022) Incremental transformer structure enhanced image inpainting with masking positional encoding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 11358–11368

  28. Wang J, Chen S, Wu Z, Jiang YG (2022) FT-TDR: frequency-guided transformer and top-down refinement network for blind face inpainting. IEEE Trans Multimed 25:2382–2392

  29. Chi L, Jiang B, Mu Y (2020) Fast Fourier convolution. Adv Neural Inf Process Syst 33:4479–4488

    Google Scholar 

  30. Cao H et al (2021) Swin-unet: unet-like pure transformer for medical image segmentation. ArXiv Prepr:ArXiv2105.05537

  31. Wang P, Li Y, Vasconcelos N (2021) Rethinking and improving the robustness of image style transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 124–133

  32. Yu C, Gao C, Wang J, Yu G, Shen C, Sang N (2021) Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. Int J Comput Vis 129(11):3051–3068

    Article  Google Scholar 

  33. Karras T, Aila T, Laine S, Lehtinen J (2017) Progressive growing of gans for improved quality, stability, and variation. ArXiv Prepr:ArXiv1710.10196

  34. Zhang R, Isola P, Efros AA, Shechtman E,Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 586–595

  35. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Proceedings of Advances in Neural Information Processing Systems, pp 6629–6640

  36. Paszke A et al(2019) Pytorch: an imperative style, high-performance deep learning library. In: Proceedings of the Advances in Neural Information Processing Systems, pp 1–12

  37. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. ArXiv Prepr:ArXiv1412.6980

  38. Jeevan P, Kumar DS, Sethi A (2023) WavePaint: resource-efficient token-mixer for self-supervised inpainting. ArXiv Prepr:ArXiv2307.00407

Download references

Author information

Author notes

  1. MohammadHossein Givkashi and MohammadReza Naderi contributed equally.

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Isfahan University of Technology, Isfahan, 84156-83111, Iran

    MohammadHossein Givkashi, MohammadReza Naderi & Nader Karimi

  2. Department of Electrical and Computer Engineering, McMaster University, Hamilton, L8S 4L8, Canada

    Shahram Shirani & Shadrokh Samavi

  3. Computer Science Department, Seattle University, Seattle, 98122, USA

    Shadrokh Samavi

Authors
  1. MohammadHossein Givkashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. MohammadReza Naderi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nader Karimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shahram Shirani
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shadrokh Samavi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nader Karimi.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Givkashi, M., Naderi, M., Karimi, N. et al. SFI-Swin: symmetric face inpainting with swin transformer by distinctly learning face components distributions. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19365-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19365-8

Keywords

Search

Navigation