Abstract
Image-based virtual try-on aims to fit an in-shop garment into a reference person image. To achieve this, a key step is garment warping, which aligns the target garment with the corresponding parts of the reference person and warps it reasonably. Previous methods typically adopt unweighted appearance flow estimation, which inherently makes it difficult to learn meaningful positions and generates unrealistic warping when the reference and the target have a large spatial difference. To overcome this limitation, a novel weighted appearance flow estimation strategy is proposed in this work. First, we extract the fusion latent vector of the reference and the target via Dual Branch Bottleneck Transformer. This enables us to take advantage of a latent vector to encode the global context. Then, we enhance the realism of appearance flow by performing sparse spatial sampling. This strengthens the communication of local information and applies constraints to warping. Experiment results on a popular virtual try-on benchmark show that our method outperforms the current state-of-the-art method in both quantitative and qualitative evaluations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Obtained By Openpose, https://github.com/CMU-Perceptual-Computing-Lab/openpose.
References
Zhao, F., Xie, Z., Kampffmeyer, M., et al.: M3D-VTON: a monocular-to-3D virtual try-on network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13239–13249 (2021)
Santesteban, I., Otaduy, M.A., Casas, D.: SNUG: self-supervised neural dynamic garments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8140–8150 (2022)
Han, X., Wu, Z., Wu, Z., et al.: VITON: an image-based virtual try-on network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7543–7552 (2018)
Yang, H., Zhang, R., Guo, X., et al.: Towards photo-realistic virtual try-on by adaptively generating-preserving image content. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7850–7859 (2020)
Ge, Y., Song, Y., Zhang, R., et al.: Parser-free virtual try-on via distilling appearance flows. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8485–8493 (2021)
He, S., Song, Y.Z., Xiang, T.: Style-based global appearance flow for virtual try-on. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3470–3479 (2022)
Issenhuth, T., Mary, J., Calauzènes, C.: Do not mask what you do not need to mask: a parser-free virtual try-on. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020, Part XX. LNCS, vol. 12365, pp. 619–635. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_37
Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part IV. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_18
Liu, Y., Li, S., Wu, Y., et al.: UMT: unified multi-modal transformers for joint video moment retrieval and highlight detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3042–3051 (2022)
Srinivas, A., Lin, T.Y., Parmar, N., et al.: Bottleneck transformers for visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16519–16529 (2021)
Zhu, X., Su, W., Lu, L., et al.: Deformable DETR: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020)
Minar, M.R., Tuan, T.T., Ahn, H., et al.: CP-VTON+: clothing shape and texture preserving image-based virtual try-on. In: CVPR Workshops, vol. 3, pp. 10–14 (2020)
Yu, R., Wang, X., Xie, X.: VTNFP: an image-based virtual try-on network with body and clothing feature preservation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10511–10520 (2019)
Minar, M.R., Ahn, H.: CloTH-VTON: clothing three-dimensional reconstruction for hybrid image-based virtual try-on. In: Proceedings of the Asian Conference on Computer Vision (2020)
Chopra, A., Jain, R., Hemani, M., et al.: ZFlow: gated appearance flow-based virtual try-on with 3d priors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5433–5442 (2021)
Bai, S., Zhou, H., Li, Z.: Single stage virtual try-on via deformable attention flows. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022, Part XV. LNCS, vol. 13675, pp. 409–425. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19784-0_24
Han, X., Hu, X., Huang, W., et al.: ClothFlow: a flow-based model for clothed person generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10471–10480 (2019)
AlBahar, B., Lu, J., Yang, J., et al.: Pose with Style: detail-preserving pose-guided image synthesis with conditional styleGAN. ACM Trans. Graph. (TOG) 40(6), 1–11 (2021)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020, Part I. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Lin, T.Y., Dollár, P., Girshick, R., et al.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)
Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Heusel, M., Ramsauer, H., Unterthiner, T., et al.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Acknowledgment
This research is supported by the National Key R &D Program of China (No. 2021YFF0900900).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, X., Zheng, G., Zhou, F., Su, Z., Lin, G. (2023). High Fidelity Virtual Try-On via Dual Branch Bottleneck Transformer. In: Lu, H., et al. Image and Graphics. ICIG 2023. Lecture Notes in Computer Science, vol 14355. Springer, Cham. https://doi.org/10.1007/978-3-031-46305-1_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-46305-1_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46304-4
Online ISBN: 978-3-031-46305-1
eBook Packages: Computer ScienceComputer Science (R0)