Fine-Grained Recognition of Manipulation Activities on Objects via Multi-Modal Sensing
Authors: 
Xiulong Liu, Bojun Zhang, Lizhang Wang, Sheng Chen, Xin Xie, Xinyu Tong, Tao Gu, Keqiu LiAuthors Info & Claims
Pages 9614 - 9628
Abstract
Fine-grained recognition of human manipulation activities on objects is crucial in the era of human-computer-object integration. However, there is a lack of solutions for simultaneous recognition of human identity, manipulation activities (including drawing and rotation), and manipulated objects. Therefore, we propose an RF-Camera system that combines RFID and computer vision techniques to address this challenge in multi-person and multi-object scenarios. In RF-Camera, we employ a skeleton-assisted method to extract facial images of target individuals, enabling precise recognition of their identities. To identify manipulation activities, we analyze the 3D hand trajectory and fingertip vector angle, differentiating drawing and rotation manipulation activities. Additionally, we model target person's hand movements to predict phase data of the target tag, enabling the determination of person-object relationships. Implementing RF-Camera using COTS RFID and Kinect devices involves overcoming challenges such as extracting effective data from noisy streams, predicting virtual phase data considering hand-tag offset, and ensuring high tag reading rates in tag-dense scenarios. We conducted experiments involving six participants performing object manipulation activities, including drawing letters/symbols and rotating movements. Extensive experimental results show that RF-Camera achieves over 90% accuracy in recognizing person identity, manipulation activities, and person-object matching in most conditions.
References
[1]
X. Liu, D. Liu, J. Zhang, T. Gu, and K. Li, “RFID and camera fusion for recognition of human-object interactions,” in Proc. Annu. Int. Conf. Mobile Comput. Netw., 2021, pp. 296–308.
[2]
P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz, “Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 4207–4215.
[3]
P. Narayana, R. Beveridge, and B. A. Draper, “Gesture recognition: Focus on the hands,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 5235–5244.
[4]
X. S. Nguyen, L. Brun, O. Lézoray, and S. Bougleux, “A neural network based on SPD manifold learning for skeleton-based hand gesture recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 12 036–12 045.
[5]
W. Ruan, Q. Z. Sheng, L. Yang, T. Gu, P. Xu, and L. Shangguan, “AudioGest: Enabling fine-grained hand gesture detection by decoding echo signal,” in Proc. ACM Int. Joint Conf. Pervasive Ubiquitous Comput., 2016, pp. 474–485.
[6]
Y. Wang, J. Shen, and Y. Zheng, “Push the limit of acoustic gesture recognition,” in Proc. IEEE Conf. Comput. Commun., 2020, pp. 566–575.
[7]
Q. Pu, S. Gupta, S. Gollakota, and S. Patel, “Whole-home gesture recognition using wireless signals,” in Proc. Annu. Int. Conf. Mobile Comput. Netw., 2013, pp. 15–18.
[8]
Y. Zheng et al., “Zero-effort cross-domain gesture recognition with Wi-Fi,” in Proc. Annu. Int. Conf. Mobile Syst. Appl. Serv., 2019, pp. 313–325.
[9]
C. Li, M. Liu, and Z. Cao, “WiHF: Enable user identified gesture recognition with WiFi,” in Proc. IEEE Conf. Comput. Commun., 2020, pp. 586–595.
[10]
Z. Zhou, L. Shangguan, X. Zheng, L. Yang, and Y. Liu, “Design and implementation of an RFID-based customer shopping behavior mining system,” IEEE/ACM Trans. Netw., vol. 25, no. 4, pp. 2405–2418, Aug. 2017.
[11]
L. Shangguan, Z. Zhou, and K. Jamieson, “Enabling gesture-based interactions with objects,” in Proc. Annu. Int. Conf. Mobile Syst. Appl. Serv., 2017, pp. 239–251.
[12]
C. Wang et al., “Multi-touch in the air: Device-free finger tracking and gesture recognition via COTS RFID,” in Proc. IEEE Conf. Comput. Commun., 2018, pp. 1691–1699.
[13]
Azure Kinect DK, Official Website, 2020. [Online]. Available: https://azure.microsoft.com/en-us/services/kinect-dk/
[14]
D. F. Silva, G. E. A. P. A. Batista, and E. Keogh, “Prefix and suffix invariant dynamic time warping,” in Proc. IEEE Int. Conf. Data Mining, 2016, pp. 1209–1214.
[15]
L. Shangguan, Z. Yang, A. X. Liu, Z. Zhou, and Y. Liu, “Relative localization of RFID tags using spatial-temporal phase profiling,” in Proc. USENIX Conf. Netw. Syst. Des. Implementation, 2015, pp. 251–263.
[16]
K. G. Derpanis, “Overview of the RANSAC algorithm,” Image Rochester NY, vol. 4, no. 1, pp. 2–3, 2010.
[17]
X. Liu, K. Meina, W. Wu, S. Shan, and X. Chen, “VIPLFACENET: An open source deep face recognition SDK,” Front. Comput. Sci., vol. 11, no. 2, pp. 208–218, 2017.
[18]
Description of unwrap, 2006. [Online]. Available: https://www.mathworks.com/help/matlab/ref/unwrap.html
[19]
M. Lenehan, “Octane SDK,” 2017. Accessed: Jan. 09, 2021. [Online]. Available: https://support.impinj.com/hc/en-us/articles/202755268-Octane-SDK
[20]
Y. Zou, J. Xiao, J. Han, K. Wu, Y. Li, and L. M. Ni, “GRfid: A device-free RFID-based gesture recognition system,” IEEE Trans. Mobile Comput., vol. 16, no. 2, pp. 381–393, Feb. 2017.
[21]
S. Zhang, C. Yang, X. Kui, J. Wang, X. Liu, and S. Guo, “ReActor: Real-time and accurate contactless gesture recognition with RFID,” in Proc. IEEE 16th Annu. Int. Conf. Sens. Commun. Netw., 2019, pp. 1–9.
[22]
Y. Yu, D. Wang, R. Zhao, and Q. Zhang, “RFID based real-time recognition of ongoing gesture with adversarial learning,” in Proc. Conf. Embedded Netw. Sensor Syst., 2019, pp. 298–310.
[23]
C. Wang, L. Xie, K. Zhang, W. Wang, Y. Bu, and S. Lu, “Spin-antenna: 3D motion tracking for tag array labeled objects via spinning antenna,” in Proc. IEEE Conf. Comput. Commun., 2019, pp. 1–9.
[24]
S. Pradhan, E. Chai, K. Sundaresan, L. Qiu, M. A. Khojastepour, and S. Rangarajan, “RIO: A pervasive RFID-based touch gesture interface,” in Proc. Annu. Int. Conf. Mobile Comput. Netw., 2017, pp. 261–274.
[25]
K. Ali, A. X. Liu, E. Chai, and K. Sundaresan, “Monitoring browsing behavior of customers in retail stores via RFID imaging,” IEEE Trans. Mobile Comput., vol. 21, no. 3, pp. 1034–1048, Mar. 2022.
[26]
J. Han et al., “CBID: A customer behavior identification system using passive tags,” IEEE/ACM Trans. Netw., vol. 24, no. 5, pp. 2885–2898, Oct. 2016.
[27]
T. Liu, L. Yang, X.-Y. Li, H. Huang, and Y. Liu, “TagBooth: Deep shopping data acquisition powered by RFID tags,” in Proc. IEEE Conf. Comput. Commun., 2015, pp. 1670–1678.
[28]
L. Shangguan, Z. Zhou, X. Zheng, L. Yang, Y. Liu, and J. Han, “ShopMiner: Mining customer shopping behavior in physical clothing stores with COTS RFID devices,” in Proc. ACM Conf. Embedded Netw. Sensor Syst., 2015, pp. 113–125.
[29]
Z. Wang, M. Xu, N. Ye, R. Wang, and H. Huang, “RF-Focus: Computer vision-assisted region-of-interest RFID tag recognition and localization in multipath-prevalent environments,” in Proc. ACM Interactive Mobile Wearable Ubiquitous Technol., vol. 3, no. 1, pp. 1–30, 2019.
[30]
J. Wu, A. Osuntogun, T. Choudhury, M. Philipose, and J. M. Rehg, “A scalable approach to activity recognition based on object use,” in Proc. IEEE Int. Conf. Comput. Vis., 2007, pp. 1–8.
[31]
Z. Liu, X. Liu, and K. Li, “Deeper exercise monitoring for smart gym using fused RFID and CV data,” in Proc. IEEE Conf. Comput. Commun., 2020, pp. 11–19.
[32]
T. Boroushaki, J. Leng, I. Clester, A. Rodriguez, and F. Adib, “Robotic grasping of fully-occluded objects using RF perception,” in Proc. IEEE Int. Conf. Robot. Automat., 2021, pp. 923–929.
Index Terms
- Fine-Grained Recognition of Manipulation Activities on Objects via Multi-Modal Sensing
Index terms have been assigned to the content through auto-classification.
Recommendations
Multimodal Wearable Sensing for Fine-Grained Activity Recognition in Healthcare
State-of-the-art in-home activity recognition schemes with wearable devices are mostly capable of detecting coarse-grained activities (sitting, standing, walking, or lying down), but can't distinguish complex activities (sitting on the floor versus the ...
AO-Finger: Hands-free Fine-grained Finger Gesture Recognition via Acoustic-Optic Sensor Fusing
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsFinger gesture recognition is gaining great research interest for wearable device interactions such as smartwatches and AR/VR headsets. In this paper, we propose a hands-free fine-grained finger gesture recognition system AO-Finger based on acoustic-...
Activity Recognition Based on Intra and Extra Manipulation of Everyday Objects
Ubiquitous Computing SystemsAbstractRecognizing activities based on an actor’s interaction with everyday objects is an important research approach within ubiquitous computing. We present a recognition approach which complement objects grabbed or released information with the object’...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
1536-1233 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 09 February 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in