Nothing Special   »   [go: up one dir, main page]
Skip to main content
SpringerLink
Log in

A high-fidelity face swapping algorithm based on mutual information-guided feature decoupling

  • Research
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

A large number of high-quality face swapping images are often required to improve the performance of forgery detection. High-quality face swapping images require fewer face swapping traces and fewer face swapping artificialities in the image while ensuring the same identity consistency with the source face and the same attribute consistency with the target. It is a challenging task today to properly separate identity and non-identity related attribute information. In this paper, we propose a novel framework that called high-fidelity face swapping algorithm (HFSA) which consists of two part networks, a GAN-based mutual information swapping network, MuIn-swap, for face swapping and an MAE-based Detail Repair Net, DRN, for detail repair. We introduce mutual information into feature compression, explicitly computing the mutual information of identity and attribute information obtained by compression of latent features. Therefore, the learning of the network is explicitly guided by formulating the minimum mutual information as the optimization goal, so that we can obtain pure identity and attribute information. In addition to overcome the problem of information loss during face swapping, we additionally design an DRN to repair the details of the face swapping to achieve more realistic and fidelity face swapping images. Through extensive experiments, it has been demonstrated that the forgery samples generated by HFSA guarantee smaller forgery traces and fewer artifacts while guaranteeing identity consistency with the source and attribute preservation. The code is already available on GitHub: https://github.com/karasuma123/HFSA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Liu, D., Gao, X., Peng, C., Wang, N., Li, J.: Heterogeneous face interpretable disentangled representation for joint face recognition and synthesis. IEEE Trans. Neural Netw. Learn. Syst. 33(10), 5611–5625 (2021)

    Article  Google Scholar 

  2. Xie, Z., Zhang, W., Sheng, B., Li, P., Chen, C.L.P.: BaGFN: Broad attentive graph fusion network for high-order feature interactions. IEEE Trans. Neural Netw. Learn Syst. 34(8), 4499–4513 (2021)

    Article  Google Scholar 

  3. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face swapping: automatically replacing faces in photographs, In: ACM SIGGRAPH, 2008, pp. 1–8

  4. Wang, H.X., Pan, C., Gong, H., Wu, H.Y.: Facial image composition based on active appearance model, In: IEEE international conference on acoustics, speech and signal processing, (2008), pp. 893–896

  5. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter T.: A 3D face model for pose and illumination invariant face recognition, In: IEEE international conference on advanced video and signal based surveillance, (2009), pp. 296–301.

  6. Nirkin, Y., Masi, I., Tran Tuan, A., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception, In: IEEE International conference on automatic face & gesture recognition, (2018), pp. 98–105

  7. Liu, J., Li, W., Pei, H., Wang, Y., Qu, F., Qu, Y., Chen, Y.: Identity preserving generative adversarial network for cross-domain person re-identification, In: IEEE Access, (2019), pp. 114021–114032

  8. Xu, Z., Yu, X., Hong, Z., Zhu, Z., Han, J., Liu, J., et al.: Facecontroller: controllable attribute editing for face in the wild. In: Proceedings of the AAAI Conference on Artificial Intelligence, (2021), pp. 3083–3091

  9. Li, L., Bao, J., Yang, H., Chen, D., Wen, F.: Advancing high fidelity identity swapping for forgery detection, In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (2020), pp. 5074–5083

  10. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: IEEE International conference on computer vision, (2017), pp. 1510–1519

  11. Chen, R., Chen, X., Ni, B., Ge, Y.: Simswap: an efficient framework for high fidelity face swapping. In: Proceedings of the 28th ACM international conference on multimedia, (2020), pp. 2003–2011

  12. Gao, G., Huang, H., Fu, C., Li, Z., He, R.: Information bottleneck disentanglement for identity swapping, In: IEEE conference on computer vision and pattern recognition, (2021), pp. 3403–3412

  13. Tishby, N., Zaslavsky, N.: Deep learning and the information bottleneck principle, In: IEEE Information Theory Workshop (ITW), (2015), pp. 1–5

  14. Shwartz-Ziv, R., Tishby, N.: Opening the black box of deep neural networks via information. (2017) arXiv preprint arXiv:1703.00810

  15. Achille, A., Soatto, S.: On the emergence of invariance and disentangling in deep representations. (2017) arXiv preprint arXiv:1706.01350,

  16. Lin, X., Sun, S., Huang, W., Sheng, B., Li, P., Feng, D.D.: EAPT: efficient attention pyramid transformer for image processing. IEEE Trans. Multim. 25, 50–61 (2021)

    Article  Google Scholar 

  17. Achille, A., Soatto S.: On the emergence of invariance and disentangling in deep representations. (2017) arXiv preprint arXiv:1706.01350

  18. Hjelm, R.D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., Bengio, Y.: Learning deep representations by mutual information estimation and maximization. 2018 arXiv preprint arXiv:1808.06670

  19. Liu, D., Gao, X., Peng, C., Wang, N., Li, J.: Composite components-based face sketch recognition. Neurocomputing 302, 46–54 (2018)

    Article  Google Scholar 

  20. Sheng, B., Li, P., Ali, R., Chen, C.L.P.: Improving video temporal consistency via broad learning system. In IEEE Trans Cybern 52(7), 6662–6675 (2021)

    Article  Google Scholar 

  21. Li, J., et al.: Automatic detection and classification system of domestic waste via multimodel cascaded convolutional neural network. IEEE Trans. Ind. Inform. 18(1), 163–173 (2021)

    Article  Google Scholar 

  22. Chen, Z., et al.: GPSD: generative parking spot detection using multi-clue recovery model. Visual Comput. 37, 2657–2669 (2021)

    Article  Google Scholar 

  23. Liu, D., Zheng, Z., Peng, C., Wang, Y., Wang, N., Gao, X.: Hierarchical forgery classifier on multi-modality face forgery clues. IEEE Trans. Multim. (2023). https://doi.org/10.1109/TMM.2023.3304913

    Article  Google Scholar 

  24. Liu, D., et al.: FedForgery: generalized face forgery detection with residual federated learning. IEEE Trans. Inform. Forens. Secur 18, 4272–4284 (2023)

    Article  Google Scholar 

  25. Korshunova, I., Shi, W., Dambre, J., Theis, L.: Fast face-swap using convolutional neural networks, In: IEEE international conference on computer vision, (2016), pp. 3677–3685

  26. Natsume, R., Yatagawa, T., Morishima, S.: Rsgan: face swapping and editing using face and hair representation in latent spaces. (2018) arXiv preprint arXiv:1804.03447

  27. Natsume, R., Yatagawa, T., Morishima, S.: Fsnet: an identity-aware generative model for image-based face swapping, In: Asian conference on computer vision, (2018), pp. 117–132

  28. Xu, Z., Zhou, H., Hong, Z., Liu, Z., Liu, J., Guo, Z., et al.: Style swap: style-based generator empowers robust face swapping, In: Computer vision–ECCV, 2022, pp. 661–677

  29. Hoque, N., Bhattacharyya, D.K., Kalita, J.K., et al.: MIFS-ND: a mutual information-based feature selection method. Expert Syst. Appl. 41(14), 6371–6385 (2014)

    Article  Google Scholar 

  30. Bennasar, M., Hicks, Y., Setchi, R.: Feature selection using joint mutual information maximization. Expert Syst. Appl. 42(22), 8520–8532 (2015)

    Article  Google Scholar 

  31. Doquire, G., Verleysen, M.: Mutual information-based feature selection for multilabel classification. Neurocomputing 122, 148–155 (2013)

    Article  Google Scholar 

  32. Xu, J.L., Zhou, Y.M., Chen, L., Xu, B.: An unsupervised feature selection approach based on mutual information. J. Comput. Res. Develop. 49(2), 372–382 (2012)

    Google Scholar 

  33. Rivaz, H., Karimaghaloo, Z., Collins, D.L.: Self-similarity weighted mutual information: a new nonrigid image registration metric. Med. Image Anal. 18(2), 343–358 (2014)

    Article  PubMed  Google Scholar 

  34. Plattard, D., Soret, M., Troccaz, J., Vassal, P., Giraud, J.Y., G. ampleboux, et, al.: Patient set-up using portal images: 2D/2D image registration using mutual information. Comput Aid Surg 5(4), 246–262 (2015)

    Article  Google Scholar 

  35. Masse, N.Y., Cachero, S., Ostrovsky, A.D., Jefferis, G.S.: A mutual information approach to automate identification of neuronal clusters in Drosophila brain images. Front. Neuroinform. (2012). https://doi.org/10.3389/fninf.2012.00021

    Article  PubMed  PubMed Central  Google Scholar 

  36. Wang, J., Hou, B., Jiao, L., Wang, S.: Representative learning via span-based mutual information for PolSAR image classification. Remote Sens. 13(9), 1609 (2021)

    Article  ADS  Google Scholar 

  37. Holden, M., Marsfield, S., Griffin, D.L., Hill, L.D.: Multi-dimensional mutual information image similarity metrics based on derivatives of linear scale-space, In: Proc APRS workshop on digital image computing, (2005)

  38. Ružić, T., Pižurica, A.: Context-aware patch-based image inpainting using Markov random field modeling. IEEE Trans. Image Process. 24(1), 444–456 (2014)

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  39. Kawai, N., Sato, T., Yokoya, N., Yokoya, N.: Diminished reality based on image inpainting considering background geometry. IEEE Trans. Visualiz. Comput. Graphics 22(3), 1236–1247 (2015)

    Article  Google Scholar 

  40. Guo, Q., Gao, S., Zhang, X., Yin, Y., Zhang, C.: Patch-based image inpainting via two-stage low rank approximation. IEEE Trans. Visual Comput. Graphics 24(6), 2023–2036 (2017)

    Article  Google Scholar 

  41. Yao, F.: Damaged region filling by improved criminisi image inpainting algorithm for thangka. Clust. Comput. 22, 13683–13691 (2019)

    Article  Google Scholar 

  42. Li, H., Luo, W., Huang, J.: Localization of diffusion-based inpainting in digital images. IEEE Trans. Inf. Forensics Secur. 12(12), 3050–3064 (2017)

    Article  Google Scholar 

  43. Li, K., Wei, Y., Yang, Z., Wei, W.: Image inpainting algorithm based on TV model and evolutionary algorithm. Soft Comput. 20, 885–893 (2016)

    Article  Google Scholar 

  44. Sridevi, G., Srinivas-Kumar, S.: Image inpainting based on fractional-order nonlinear diffusion for image reconstruction. Circ. Syst. Signal Process. 38, 3802–3817 (2019)

    Article  Google Scholar 

  45. Jin, X., Su, Y., Zou, L., Wang, Y., Jing, P., Wang, Z.J.: Sparsity-based image inpainting detection via canonical correlation analysis with low-rank constraints, In: IEEE Access, (2018), pp. 49967–49978

  46. Mo, J., Zhou, Y.: The research of image inpainting algorithm using self-adaptive group structure and sparse representation. Clust. Comput. 22, 7593–7601 (2019)

    Article  Google Scholar 

  47. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting, In: Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 2536–2544

  48. Yang, C., X., Lu, Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis, In: Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), pp. 6721–6729

  49. Zhang, H., Hu, Z., Luo, C., Zuo, W., Wang, M.: Semantic image inpainting with progressive generative networks, In: Proceedings of the 26th ACM international conference on Multimedia, (2018), pp. 1939–1947

  50. Zheng, C., Cham, T. J., Cai, J.: Pluralistic image completion, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2019), pp. 1438–1447

  51. Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., Ebrahimi, M.: Edgeconnect: generative image inpainting with adversarial edge learning. (2019), arXiv preprint arXiv:1901.00212

  52. Jo, Y., Park, J.: Sc-fegan: face editing generative adversarial network with user's sketch and color, In: Proceedings of the IEEE international conference on computer vision, (2019), pp. 1745–1753

  53. Liu, H., Jiang, B., Xiao, Y., Yang, C.: Coherent semantic attention for image inpainting, In: Proceedings of the IEEE international conference on computer vision, (2019), pp. 4170–4179

  54. Kingma, P.D., Welling, M.: Auto-encoding variational bayes. (2013) arXiv preprint arXiv:1312.6114

  55. Smilkov, D., Thorat, N., Kim, B., Viégas, F., Wattenberg, M.: Smoothgrad: removing noise by adding noise.” (2017) arXiv preprint arXiv:1706.03825

  56. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan, In: Proceedings of the IEEE conference on computer vision and pattern recognition, (2020), pp. 8110–8119

  57. Wang, K., Zhao, B., Peng, X., Zhu, Z., Deng, J., Wang, X., et al.: FaceMAE: privacy-preserving face recognition via masked autoencoders, (2022) arXiv preprint arXiv:2205.11090

  58. Luo, Y., Chen, Z., Gao, X.: Self-distillation augmented masked autoencoders for histopathological image classification, (2022), arXiv preprint arXiv:2203.16983

  59. Mechrez, R., Talmi, I., Zelnik-Manor, L.: The contextual loss for image transformation with non-aligned data, In: Proceedings of the European conference on computer vision, (2018), pp. 768–783

  60. Jiang, L., Dai, B., Wu, W., Loy, C.C.: Focal frequency loss for image reconstruction and synthesis, In: Proceedings of the IEEE International conference on computer vision, (2021), pp. 13919–13929

  61. Wang, M., Deng, W.: Deep face recognition: a survey. Neurocomputing 429, 215–244 (2021)

    Article  Google Scholar 

  62. Yang, T., Ren, P., Xie, X., Zhang L.: Gan prior embedded network for blind face restoration in the wild, In: Proceedings of the IEEE conference on computer vision and pattern recognition, (2021), pp. 672–681

  63. DeepFakes. 2020. faceswap. https://github.com/deepfakes/faceswap

  64. Zhu, Y., Li, Q., Wang, J., Xu, C., & Sun, Z. (2021). One shot face swapping on megapixels. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4832-4842

  65. Li, Q., Wang, W., Xu, C., Sun, Z.: Learning disentangled representation for one-shot progressive face swapping. (2022). ArXiv abs/2203.12985

  66. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niessner, M.: Faceforensics++: Learning to detect manipulated facial images, in: Proceedings of the IEEE international conference on computer vision, (2019), pp. 1–11

  67. Wang, H., Wang, Y., Zhou, Z., Ji, X., Li, Z., Gong, D., Zhou, J., Liu, W.: Cosface: large margin cosine loss for deep face recognition, In: Proceedings of the IEEE conference on computer vision and pattern recognition, (2018), pp. 5265–5274

  68. Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., Li, S.: Towards fast, accurate and stable 3d dense face alignment, In: Computer Vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX. Cham: Springer international publishing, 2020, pp. 152–168

  69. Li, L., Bao, J., Yang, H., Chen, D., Wen, F.: Advancing high fidelity identity swapping for forgery detection, In: Proceedings of the IEEE conference on computer vision and pattern recognition, (2020), pp. 5074–5083

  70. Li, X., Chen, C., Zhou, S., Lin, X., Zuo, W., Zhang, L.: Blind face restoration via deep multi-scale component dictionaries, In: Computer Vision – ECCV 2020, (2020) pp 12354

Download references

Acknowledgements

Foundation Item: The National Natural Science Foundation of China (62101414, 62201423), Beijing Municipal Natural Science Foundation (4232034), The China Postdoctoral Science Foundation (2021M702546, 2021M702548), The China Postdoctoral Science Foundation (2022T150508), The Young Talent Fund Xi’an Association for Science and Technology(095920221320) and The Guangdong Basic and Applied Basic Research Foundation (2020A1515110856).

Author information

Authors and Affiliations

  1. Department of Electronic and Communication Engineering, Beijing Electronic Science and Technology Institute, Beijing, 100070, China

    Song Xiao

  2. School of Telecommunications Engineering, Xidian University, Xi’an, 710071, China

    Song Xiao, ZhiGuo Liu, Jian Gao & ChangXin Wang

Authors
  1. Song Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. ZhiGuo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. ChangXin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Song Xiao and ZhiGug Liu wrote the main manuscript text. Song Xiao provided critical guidance and financial support for this study. ZhiGuo Liu was responsible for the writing of the core code and the preliminary algorithm ideas. Jian Gao and Changxin Wang prepared the experimental data. All authors reviewed the manuscript.

Corresponding author

Correspondence to Song Xiao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiao, S., Liu, Z., Gao, J. et al. A high-fidelity face swapping algorithm based on mutual information-guided feature decoupling. Vis Comput (2024). https://doi.org/10.1007/s00371-024-03288-4

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-024-03288-4

Keywords

Search

Navigation