Spatial and Surface Correspondence Field for Interaction Transfer
Authors: 
Zeyu Huang, Honghao Xu, Haibin Huang, Chongyang Ma, 
Hui Huang, Ruizhen HuAuthors Info & Claims
Article No.: 83, Pages 1 - 12
Abstract
In this paper, we introduce a new method for the task of interaction transfer. Given an example interaction between a source object and an agent, our method can automatically infer both surface and spatial relationships for the agent and target objects within the same category, yielding more accurate and valid transfers. Specifically, our method characterizes the example interaction using a combined spatial and surface representation. We correspond the agent points and object points related to the representation to the target object space using a learned spatial and surface correspondence field, which represents objects as deformed and rotated signed distance fields. With the corresponded points, an optimization is performed under the constraints of our spatial and surface interaction representation and additional regularization. Experiments conducted on human-chair and hand-mug interaction transfer tasks show that our approach can handle larger geometry and topology variations between source and target shapes, significantly outperforming state-of-the-art methods.
Supplementary Material
supplemental
- Download
- 44.21 MB
References
[1]
Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. 2020. Skeleton-aware networks for deep motion retargeting. ACM Transactions on Graphics (TOG) 39, 4 (2020), 62--1.
[2]
Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. 2019. Learning character-agnostic motion for motion retargeting in 2d. arXiv preprint arXiv:1905.01680 (2019).
[3]
Rami Ali Al-Asqhar, Taku Komura, and Myung Geol Choi. 2013. Relationship descriptors for interactive motion adaptation. In Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 45--53.
[4]
Jean Basset, Stefanie Wuhrer, Edmond Boyer, and Franck Multon. 2019. Contact preserving shape transfer for rigging-free motion retargeting. In Proceedings of the 12th ACM SIGGRAPH Conference on Motion, Interaction and Games. 1--10.
[5]
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR]. Stanford University --- Princeton University --- Toyota Technological Institute at Chicago.
[6]
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.
[7]
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. 2021a. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12200--12209.
[8]
Yu Deng, Jiaolong Yang, and Xin Tong. 2021b. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10286--10296.
[9]
Edmond SL Ho, Taku Komura, and Chiew-Lan Tai. 2010. Spatial relationship preserving character motion adaptation. In ACM SIGGRAPH 2010 papers. 1--8.
[10]
Zeyu Huang, Juzhan Xu, Sisi Dai, Kai Xu, Hao Zhang, Hui Huang, and Ruizhen Hu. 2023. Nift: Neural interaction field and template for object manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1875--1881.
[11]
Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Zhiyuan Zhang, Yixin Chen, He Wang, Yixin Zhu, and Siyuan Huang. 2023. Full-Body Articulated Human-Object Interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 9365--9376.
[12]
Taeil Jin, Meekyoung Kim, and Sung-Hee Lee. 2018. Aura mesh: Motion retargeting to preserve the spatial relationships between skinned characters. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 311--320.
[13]
Jongmin Kim, Yeongho Seol, and Taesoo Kwon. 2021. Interactive multi-character motion retargeting. Computer Animation and Virtual Worlds 32, 3--4 (2021), e2015.
[14]
Sihyeon Kim, Minseok Joo, Jaewon Lee, Juyeon Ko, Juhan Cha, and Hyunwoo J Kim. 2023. Semantic-Aware Implicit Template Learning via Part Deformation Consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 593--603.
[15]
Yeonjoon Kim, Hangil Park, Seungbae Bang, and Sung-Hee Lee. 2016. Retargeting human-object interaction to virtual avatars. IEEE transactions on visualization and computer graphics 22, 11 (2016), 2405--2412.
[16]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[17]
Hao Li, Robert W Sumner, and Mark Pauly. 2008. Global correspondence optimization for non-rigid registration of depth scans. In Computer graphics forum, Vol. 27. Wiley Online Library, 1421--1430.
[18]
Jiaman Li, Jiajun Wu, and C Karen Liu. 2023. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) 42, 6 (2023), 1--11.
[19]
Zhiguang Liu, Antonio Mucherino, Ludovic Hoyet, and Franck Multon. 2018. Surface based motion retargeting by preserving spatial relationship. In Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games. 1--11.
[20]
Robin Magnet, Jing Ren, Olga Sorkine-Hornung, and Maks Ovsjanikov. 2022. Smooth non-rigid shape matching via effective Dirichlet energy optimization. arXiv preprint arXiv:2210.02870 (2022).
[21]
Andriy Myronenko and Xubo Song. 2010. Point set registration: Coherent point drift. IEEE transactions on pattern analysis and machine intelligence 32, 12 (2010), 2262--2275.
[22]
Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. 2012. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (ToG) 31, 4 (2012), 1--11.
[23]
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. 2019. Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 10975--10985.
[24]
Sören Pirk, Vojtech Krs, Kaimo Hu, Suren Deepak Rajasekaran, Hao Kang, Yusuke Yoshiyasu, Bedrich Benes, and Leonidas J Guibas. 2017. Understanding and exploiting object interaction landscapes. ACM Transactions on Graphics (TOG) 36, 3 (2017), 1--14.
[25]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 652--660.
[26]
Jing Ren, Simone Melzi, Peter Wonka, and Maks Ovsjanikov. 2021. Discrete optimization for shape matching. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 81--96.
[27]
Diego Rodriguez and Sven Behnke. 2018. Transferring category-based functional grasping skills by latent space non-rigid registration. IEEE Robotics and Automation Letters 3, 3 (2018), 2662--2669.
[28]
Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36, 6 (Nov. 2017).
[29]
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. 2022. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA). IEEE, 6394--6400.
[30]
Shanlin Sun, Kun Han, Deying Kong, Hao Tang, Xiangyi Yan, and Xiaohui Xie. 2022. Topology-preserving shape reconstruction and registration via neural diffeomorphic flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20845--20855.
[31]
Rina Wu, Tianqiang Zhu, Wanli Peng, Jinglue Hang, and Yi Sun. 2023. Functional grasp transfer across a category of objects from only one labeled instance. IEEE Robotics and Automation Letters 8, 5 (2023), 2748--2755.
[32]
Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. 2022. OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20953--20962.
[33]
Yunbo Zhang, Deepak Gopinath, Yuting Ye, Jessica Hodgins, Greg Turk, and Jungdam Won. 2023. Simulation and retargeting of complex multi-character interactions. In ACM SIGGRAPH 2023 Conference Proceedings. 1--11.
[34]
Xi Zhao, Myung Geol Choi, and Taku Komura. 2017. Character-object interaction retrieval using the interaction bisector surface. 36, 2 (2017), 119--129.
[35]
Xi Zhao, Ruizhen Hu, Haisong Liu, Taku Komura, and Xinyu Yang. 2019. Localization and completion for 3D object interactions. IEEE transactions on visualization and computer graphics 26, 8 (2019), 2634--2644.
[36]
Xi Zhao, He Wang, and Taku Komura. 2014. Indexing 3d scenes using the interaction bisector surface. ACM Transactions on Graphics 33, 3 (2014), 1--14.
[37]
Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu. 2021. Deep implicit templates for 3d shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1429--1439.
[38]
Kun Zhou, Jin Huang, John Snyder, Xinguo Liu, Hujun Bao, Baining Guo, and Heung-Yeung Shum. 2005. Large mesh deformation using the volumetric graph laplacian. In ACM SIGGRAPH 2005 Papers. 496--503.
Index Terms
- Spatial and Surface Correspondence Field for Interaction Transfer
Recommendations
Hierarchical Spherical Deformation for Shape Correspondence
Medical Image Computing and Computer Assisted Intervention – MICCAI 2018AbstractWe present novel spherical deformation for a landmark-free shape correspondence in a group-wise manner. In this work, we aim at both addressing template selection bias and minimizing registration distortion in a single framework. The proposed ...
Deformation-driven shape correspondence via shape recognition
Many approaches to shape comparison and recognition start by establishing a shape correspondence. We "turn the table" and show that quality shape correspondences can be obtained by performing many shape recognition tasks. What is more, the method we ...
Deformation-driven topology-varying 3D shape correspondence
We present a deformation-driven approach to topology-varying 3D shape correspondence. In this paradigm, the best correspondence between two shapes is the one that results in a minimal-energy, possibly topology-varying, deformation that transforms one ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 July 2024
Published in TOG Volume 43, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
- Guangdong Natural Science Foundation
- Shenzhen Science and Technology Program
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 116Total Downloads
- Downloads (Last 12 months)116
- Downloads (Last 6 weeks)11
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in