Human-Robot Interactive System for Warehouses using Speech SLAM and Deep Learning-based Barcode Recognition
Authors: 
Harish Ram Nambiappan, Sama Nikanfar, Ayon Roy, Joey Hussain, Deep Shinglot, Sneh Acharya, Nicholas Gans, Fillia MakedonAuthors Info & Claims
Pages 38 - 44
Abstract
This paper presents an initial investigation of a speech-based human-robot interaction system for locating items in a warehouse environment. The system uses a 2D SLAM map and visual servoing with deep learning-based barcode recognition to identify and locate items based on user speech commands. The system was tested with and without item location in the SLAM map and achieved a 100% success rate in identifying and localizing items. The average speech processing time was recorded at 9.28 seconds, and the system demonstrated a best-case timing of 15.46 seconds and a worst-case timing of 3 minutes for identifying items on different tables. The proposed system has the potential to improve the employment opportunities and experiences of blind or visually impaired workers in the warehouse industry. Future work will focus on testing the system in real-world environments and improving its performance in cluttered and dynamic settings.
References
[1]
Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019. Character Region Awareness for Text Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9365–9374.
[2]
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems 33 (2020), 12449–12460.
[3]
Steve Banker. 2021. Warehouse Labor Woes Are Worse Than Ever. Forbes (2021).
[4]
Darwin Bautista and Rowel Atienza. 2022. Scene Text Recognition with Permuted Autoregressive Sequence Models. In Proceedings of the European Conference on Computer Vision.
[5]
Daniela Cernega and Razvan Solea. 2020. Hybrid Control Application Using Mobile Visual Servoing for Flexible Manufacturing Mechatronics Line. In International Conference on System Theory, Control and Computing. 636–641.
[6]
Hyeon Cho, Dongyi Kim, Junho Park, Kyungshik Roh, and Wonjun Hwang. 2018. 2D barcode detection using images for drone-assisted inventory management. In International Conference on Ubiquitous Robots. 461–465.
[7]
Daniel Kold Hansen, Kamal Nasrollahi, Christoffer Bøgelund Rasmussen, and Thomas B Moeslund. 2017. Real-time barcode detection and classification using deep learning. In International Joint Conference on Computational Intelligence. 321–327.
[8]
James Jeffs. 2022. Mobile robotics in logistics, warehousing and delivery 2022-2042. https://www.idtechex.com/en/research-report/mobile-robotics-in-logistics-warehousing-and-delivery-2022-2042/855
[9]
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, 2020. Libri-light: A benchmark for asr with limited or no supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing. 7669–7673.
[10]
Ivan Kalinov, Alexander Petrovsky, Valeriy Ilin, Egor Pristanskiy, Mikhail Kurenkov, Vladimir Ramzhaev, Ildar Idrisov, and Dzmitry Tsetserukou. 2020. Warevision: CNN barcode detection-based UAV trajectory optimization for autonomous warehouse stocktaking. IEEE Robotics and Automation Letters 5, 4 (2020), 6647–6653.
[11]
Woong Kwon, Jun Ho Park, Minsu Lee, Jongbeom Her, Sang-Hyeon Kim, and Ja-Won Seo. 2019. Robust autonomous navigation of unmanned aerial vehicles (UAVs) for warehouses’ inventory application. IEEE Robotics and Automation Letters 5, 1 (2019), 243–249.
[12]
Michele C McDonnall and Zhen Sui. 2019. Employment and unemployment rates of people who are blind or visually impaired: Estimates from multiple sources. Journal of Visual Impairment & Blindness 113, 6 (2019), 481–492.
[13]
Rafael Munoz-Salinas and Rafael Medina-Carnicer. 2020. UcoSLAM: Simultaneous localization and mapping by fusion of keypoints and squared planar markers. Pattern Recognition 101 (2020), 107193.
[14]
Harish Ram Nambiappan, Stephanie Arevalo Arboleda, Cody Lee Lundberg, Maria Kyrarini, Fillia Makedon, and Nicholas Gans. 2022. Mina: A robotic assistant for hospital fetching tasks. Technologies 10, 2 (2022), 41.
[15]
Harish Ram Nambiappan, Krishna Chaitanya Kodur, Maria Kyrarini, Fillia Makedon, and Nicholas Gans. 2021. MINA: A Multitasking Intelligent Nurse Aid Robot. In PErvasive Technologies Related to Assistive Environments Conference. 266–267.
[16]
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing. 5206–5210.
[17]
Ismot Sadik Peyas, Zahid Hasan, Md Rafat Rahman Tushar, Al Musabbir, Raisa Mehjabin Azni, and Shahnewaz Siddique. 2021. Autonomous Warehouse Robot using Deep Q-Learning. In IEEE Region 10 Conference (TENCON). 857–862.
[18]
Dan Popescu, Viorel Mihai, Cristian Drăgana, Loretta Ichim, 2020. Visual Servoing System for Local Robot Control in a Flexible Assembly Line. In Mediterranean Conference on Control and Automation. 927–932.
[19]
Rafael Rey, Marco Corzetto, Jose Antonio Cobano, Luis Merino, and Fernando Caballero. 2019. Human-robot co-working system for warehouse automation. In IEEE International Conference on Emerging Technologies and Factory Automation. 578–585.
[20]
Georgian Simion, Adrian Filipescu, Dan Ionescu, Răzvan Șolea, Daniela Cernega, Eugenia Mincă, and Adriana Filipescu. 2022. Mobile Visual Servoing Based Control of a Complex Autonomous System Assisting a Manufacturing Technology on a Mechatronics Line. Inventions 7, 3 (2022), 47.
[21]
Nan Tian, Ajay Kummar Tanwani, Jinfa Chen, Mas Ma, Robert Zhang, Bill Huang, Ken Goldberg, and Somayeh Sojoudi. 2019. A fog robotic system for dynamic visual servoing. In International Conference on Robotics and Automation. 1982–1988.
[22]
Paul Wellener, Victor Reyes, Heather Ashton, and Chad Moutray. 2021. Creating pathways for tomorrow’s workforce today. Deloitte Insights (2021).
[23]
Yunzhe Xiao and Zhong Ming. 2019. 1D barcode detection via integrated deep-learning and geometric approach. Applied Sciences 9, 16 (2019), 3268.
[24]
Qifan Yang, Yibin Yang, Yanru Liu, Yindong Lian, Wei Xie, Langwen Zhang, and Mi Pan. 2022. Visual Servoing of AGV Based on Nonlinear Model Predictive Control. In Chinese Control and Decision Conference. 5048–5053.
[25]
Hui Zhang, Guoliang Shi, Li Liu, Miao Zhao, and Zhicong Liang. 2018. Detection and identification method of medical label barcode based on deep learning. In International Conference on Image Processing Theory, Tools and Applications. 1–6.
[26]
Huijuan Zhang, Chengning Zhang, Wei Yang, and Chin-Yin Chen. 2015. Localization and navigation using QR code for mobile robot in indoor environment. In IEEE international conference on robotics and biomimetics. 2501–2506.
[27]
Xia Zhu. 2021. Design of Barcode Recognition System Based on YOLOV5. In Journal of Physics: Conference Series, Vol. 1995. IOP Publishing, 012052.
Index Terms
- Human-Robot Interactive System for Warehouses using Speech SLAM and Deep Learning-based Barcode Recognition
Recommendations
ROS-Based SLAM for a Gazebo-Simulated Mobile Robot in Image-Based 3D Model of Indoor Environment
ACIVS 2015: Proceedings of the 16th International Conference on Advanced Concepts for Intelligent Vision Systems - Volume 9386Nowadays robot simulators have robust physics engines, high-quality graphics, and convenient interfaces, affording researchers to substitute physical systems with their simulation models in order to pre-estimate the performance of theoretical findings ...
The visual SLAM system for a hexapod robot
ICCVG'10: Proceedings of the 2010 international conference on Computer vision and graphics: Part IIThe precise localization of the mobile robot plays a vital role in the autonomous operation of the mobile robot. The vision based simultaneous localization and mapping (SLAM) is a widely known technique for tracking the movement of the camera in the ...
SLAM and navigation in indoor environments
PSIVT'11: Proceedings of the 5th Pacific Rim conference on Advances in Image and Video Technology - Volume Part IIn this paper, we propose a system for wheeled robot SLAM and navigation in indoor environments. An omni-directional camera and a laser range finder are the sensors to extract the point features and the line features as the landmarks. In SLAM and self-...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
708 pages
ISBN:9798400717604
DOI:10.1145/3652037
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
PETRA '24
PETRA '24: The PErvasive Technologies Related to Assistive Environments Conference
June 26 - 28, 2024
Crete, Greece
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 203Total Downloads
- Downloads (Last 12 months)203
- Downloads (Last 6 weeks)78
Reflects downloads up to 26 Nov 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in