GO-Finder: A Registration-free Wearable System for Assisting Users in Finding Lost Hand-held Objects
Authors: 
Takuma Yagi, Takumi Nishiyasu, Kunimasa Kawasaki, Moe Matsuki, Yoichi SatoAuthors Info & Claims
Article No.: 35, Pages 1 - 29
Abstract
People spend an enormous amount of time and effort looking for lost objects. To help remind people of the location of lost objects, various computational systems that provide information on their locations have been developed. However, prior systems for assisting people in finding objects require users to register the target objects in advance. This requirement imposes a cumbersome burden on the users, and the system cannot help remind them of unexpectedly lost objects. We propose GO-Finder (“Generic Object Finder”), a registration-free wearable camera-based system for assisting people in finding an arbitrary number of objects based on two key features: automatic discovery of hand-held objects and image-based candidate selection. Given a video taken from a wearable camera, GO-Finder automatically detects and groups hand-held objects to form a visual timeline of the objects. Users can retrieve the last appearance of the object by browsing the timeline through a smartphone app. We conducted user studies to investigate how users benefit from using GO-Finder. In the first study, we asked participants to perform an object retrieval task and confirmed improved accuracy and reduced mental load in the object search task by providing clear visual cues on object locations. In the second study, the system’s usability on a longer and more realistic scenario was verified, accompanied by an additional feature of context-based candidate filtering. Participant feedback suggested the usefulness of GO-Finder also in realistic scenarios where more than one hundred objects appear.
References
[1]
Irshad Abibouraguimane, Kakeru Hagihara, Keita Higuchi, Yuta Itoh, Yoichi Sato, Tetsu Hayashida, and Maki Sugimoto. 2019. CoSummary: Adaptive fast-forwarding for surgical videos by detecting collaborative scenes using hand regions and gaze positions. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 580–590.
[2]
Relja Arandjelovic and Andrew Zisserman. 2013. All about VLAD. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1578–1585.
[3]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. J. Usab. Stud. 4, 3, 114–123.
[4]
Gedas Bertasius, Hyun Soo Park, Stella X. Yu, and Jianbo Shi. 2017. Unsupervised learning of important objects from first-person videos. In Proceedings of the IEEE International Conference on Computer Vision. 1956–1964.
[5]
Luca Bertinetto, Jack Valmadre, Joao F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. 2016. Fully-convolutional siamese networks for object tracking. In Proceedings of the European Conference on Computer Vision Workshops. 850–865.
[6]
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. 2016. Simple online and realtime tracking. In Proceedings of the IEEE International Conference on Image Processing. 3464–3468.
[7]
Marc Bolaños and Petia Radeva. 2015. Ego-object discovery. arXiv preprint arXiv:1504.01639.
[8]
Gaetano Borriello, Waylon Brunette, Matthew Hall, Carl Hartung, and Cameron Tangney. 2004. Reminding about tagged objects using passive RFIDs. In Proceedings of the ACM International Conference on Ubiquitous Computing. 36–53.
[9]
William F. Brewer. 1988. Qualitative analysis of the recalls of randomly sampled autobiographical events. In Practical Aspects of Memory: Current Research and Issues, M. M. Gruneberg, P. E. Morris, and R. N. Sykes (Eds.). Vol. 1. Wiley, 263–268.
[10]
John Brooke. 1996. SUS: A “quick and dirty” usability. Usab. Eval. Industr. Taylor & Francis, 189–194.
[11]
Andreas Butz, Michael Schneider, and Mira Spassova. 2004. Searchlight—A lightweight search function for pervasive environments. In Proceedings of the IEEE International Conference on Pervasive Computing. 351–356.
[12]
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2020. Rescaling egocentric vision. Comput. Res. Reposit. abs/2006.13256 (2020).
[13]
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2018. Scaling egocentric vision: The EPIC-KITCHENS dataset. In Proceedings of the European Conference on Computer Vision. 720–736.
[14]
Tamara Denning, Zakariya Dehlawi, and Tadayoshi Kohno. 2014. In situ with bystanders of augmented reality glasses: Perspectives on recording and privacy-mediating technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2377–2386.
[15]
Margery Eldridge, Abigail Sellen, and Debra Bekerian. 1992. Memory Problems at Work: Their Range, Frequency and Severity. Technical Report EPC–92–129. Rank Xerox EUROPARC.
[16]
David Elsweiler, Ian Ruthven, and Christopher Jones. 2007. Towards memory supporting personal information management tools. J. Amer. Societ. Inf. Sci. Technol. 58, 7 (2007), 924–946.
[17]
Markus Funk, Robin Boldt, Bastian Pfleging, Max Pfeiffer, Niels Henze, and Albrecht Schmidt. 2014. Representing indoor location of objects on wearable computers with head-mounted displays. In Proceedings of the 5th Augmented Human International Conference. 1–4.
[18]
Markus Funk, Albrecht Schmidt, and Lars Erik Holmquist. 2013. Antonius: A mobile search engine for the physical world. In Proceedings of the ACM Conference on Pervasive and Ubiquitous Computing Adjunct. 179–182.
[19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.
[20]
Keita Higuchi, Ryo Yonetani, and Yoichi Sato. 2017. EgoScanning: Quickly scanning first-person videos with egocentric elastic timelines. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 6536–6546.
[21]
Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. SenseCam: A retrospective memory aid. In Proceedings of the ACM International Conference on Ubiquitous Computing. 177–193.
[22]
Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014. Privacy behaviors of lifeloggers using wearable cameras. In Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing. 571–582.
[23]
Apple Inc.2021. AirTag. Retrieved from https://www.apple.com/airtag/.
[24]
Tile Inc.2017. Find your keys, Wallet & phone with Tile’s app and Bluetooth tracker device | Tile. Retrieved from https://www.thetileapp.com/en-eu/.
[25]
J. Indratmo and Julita Vassileva. 2008. A review of organizational structures of personal information management. J. Dig. Inf. 9, 1 (2008).
[26]
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning. PMLR, 448–456.
[27]
Amalie Enshelm Jensen, Caroline Møller Jægerfelt, Sanne Francis, Birger Larsen, and Toine Bogers. 2018. I just scroll through my stuff until I find it or give up: A contextual inquiry of PIM on private handheld devices. In Proceedings of the Conference on Human Information Interaction & Retrieval. 140–149.
[28]
William Jones. 2007. Personal information management. Ann. Rev. Inf. Sci. Technol. 41, 1 (2007), 453–504.
[29]
Tatsuyuki Kawamura, Tomohiro Fukuhara, Hideaki Takeda, Yasuyuki Kono, and Masatsugu Kidode. 2007. Ubiquitous memories: A memory externalization system using physical objects. Person. Ubiq. Comput. 11, 4, 287–298.
[30]
Liadh Kelly, Yi Chen, Marguerite Fuller, and Gareth J. F. Jones. 2008. A study of remembered context for information access from personal digital archives. In Proceedings of the International Symposium on Information Interaction in Context. 44–50.
[31]
Julie A. Kientz, Shwetak N. Patel, Arwa Z. Tyebkhan, Brian Gane, Jennifer Wiley, and Gregory D. Abowd. 2006. Where’s my stuff? Design and evaluation of a mobile system for locating lost items for the visually impaired. In Proceedings of the 8th ACM Conference on Computers and Accessibility. 103–110.
[32]
Harold W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Res. Logist. Quart. 2, 1–2 (1955), 83–97.
[33]
Kyungjun Lee and Hernisa Kacorri. 2019. Hands holding clues for object recognition in teachable machines. In Proceedings of the ACM CHI Conference of Human Factors in Computing Systems. 1–12.
[34]
Kyungjun Lee, Abhinav Shrivastava, and Hernisa Kacorri. 2020. Hand-priming in object localization for assistive egocentric vision. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 3422–3432.
[35]
Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. 2012. Discovering important people and objects for egocentric video summarization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1346–1353.
[36]
Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. 2019. SiamRPN++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4282–4291.
[37]
Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, and Khai N. Truong. 2019. FMT: A wearable camera-based object tracking memory aid for older adults. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 3, 3, 1–25.
[38]
Yin Li, Miao Liu, and James M. Rehg. 2018. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European Conference on Computer Vision. 619–635.
[39]
Xiaotao Liu, Mark D. Corner, and Prashant Shenoy. 2006. Ferret: RFID localization for pervasive multimedia. In Proceedings of the ACM International Conference on Ubiquitous Computing. 422–440.
[40]
David G. Lowe. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 2 (2004), 91–110.
[41]
Cewu Lu, Renjie Liao, and Jiaya Jia. 2015. Personal object discovery in first-person videos. IEEE Trans. Image Process. 24, 12, 5789–5799.
[42]
Robert J. Orr, Ronald Raymond, Joshua Berman, and A. Fleming Seay. 1999. A System for Finding Frequently Lost Objects in the Home. Technical Report GIT-GVU-99-24. Georgia Institute of Technology.
[43]
Ling Pei, Ruizhi Chen, Jingbin Liu, Tomi Tenhunen, Heidi Kuusniemi, and Yuwei Chen. 2010. Inquiry-based Bluetooth indoor positioning via RSSI probability distributions. In Proceedings of the International Conference on Advances in Satellite and Space Communications. 151–156.
[44]
Rodney E. Peters, Richard Pak, Gregory D. Abowd, Arthur D. Fisk, and Wendy A. Rogers. 2004. Finding Lost Objects: Informing the Design of Ubiquitous Computing Services for the Home. Technical Report GIT-GVU-04-01. Georgia Institute of Technology.
[45]
Cristian Reyes, Eva Mohedano, Kevin McGuinness, Noel E. O’Connor, and Xavier Giro-i Nieto. 2016. Where is my phone? Personal object retrieval from egocentric images. In Proceedings of the 1st Workshop on Lifelogging Tools and Applications. 55–62.
[46]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 3 (2015), 211–252.
[47]
Bernt Schiele, Nuria Oliver, Tony Jebara, and Alex Pentland. 1999. DyPERS: Dynamic personal enhanced reality system. In Proceedings of the International Conference on Computer Vision Systems.
[48]
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815–823.
[49]
David Schwarz, Max Schwarz, Jörg Stückler, and Sven Behnke. 2014. Cosero, find my keys! Object localization and retrieval using Bluetooth low energy tags. In Robot World Cup XVIII. Springer. 195–206.
[50]
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey. 2020. Understanding human hands in contact at Internet scale. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9869–9878.
[51]
Jianbo Shi and Jitendra Malik. 2000. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22, 8 (2000), 888–905.
[52]
Makoto Shinnishi. 1999. Hide and seek: Physical real artifacts which responds to the user. In Proceedings of the World Multiconference on Systemics, Cybernetics and Informatics. 84–88.
[53]
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. 2018. Actor and observer: Joint modeling of first and third-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7396–7404.
[54]
Masaya Tanbo, Ryoma Nojiri, Yuusuke Kawakita, and Haruhisa Ichikawa. 2017. Active RFID attached object clustering method with new evaluation criterion for finding lost objects. Mob. Inf. Syst. 2017 (2017), 3637814.
[55]
Pixie Technology. 2017. The nation’s biggest lost and found survey, by Pixie. Retrieved from https://tinyurl.com/yxrzbsnp.
[56]
Quan T. Tran, Gina Calcaterra, and Elizabeth D. Mynatt. 2005. Cook’s Collage. In Proceedings of the International Conference on Home-oriented Informatics and Telematics. 15–32.
[57]
Takahiro Ueoka, Tatsuyuki Kawamura, Yasuyuki Kono, and Masatsugu Kidode. 2003. I’m here!: A wearable object remembrance support system. In Proceedings of the ACM International Conference on Mobile Human-computer Interaction. 422–427.
[58]
Paul Wilson, Daniel Prashanth, and Hamid Aghajan. 2007. Utilizing RFID signaling scheme for localization of stationary objects and speed estimation of mobile objects. In Proceedings of the IEEE International Conference on RFID. 94–99.
[59]
Dan Xie, Tingxin Yan, Deepak Ganesan, and Allen Hanson. 2008. Design and implementation of a dual-camera wireless sensor network for object retrieval. In Proceedings of the IEEE International Conference on Information Processing in Sensor Networks. 469–480.
Index Terms
- GO-Finder: A Registration-free Wearable System for Assisting Users in Finding Lost Hand-held Objects
Recommendations
GO-Finder: A Registration-Free Wearable System for Assisting Users in Finding Lost Objects via Hand-Held Object Discovery
IUI '21: Proceedings of the 26th International Conference on Intelligent User InterfacesPeople spend an enormous amount of time and effort looking for lost objects. To help remind people of the location of lost objects, various computational systems that provide information on their locations have been developed. However, prior systems for ...
LocatAR: An AR Object Search Assistance System for a Shared Space
AHs '23: Proceedings of the Augmented Humans International Conference 2023Item-finding tasks due to memory lapse are costly activities commonly experienced by many people. However, conventional systems are not suitable for use in a collaborative environment. Therefore, we propose a multi-functional, pre-registration-free, ...
CamFi: An AI-driven and Camera-based System for Assisting Users in Finding Lost Objects in Multi-Person Scenarios
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing SystemsIt is important to study how to help people quickly find misplaced objects. However, previous studies have focused on single-person scenarios without considering the influence of other people in public places. Based on the technology of object detection ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2022
321 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 November 2022
Online AM: 27 April 2022
Accepted: 15 February 2022
Revised: 29 November 2021
Received: 13 August 2021
Published in TIIS Volume 12, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
- Refereed
Funding Sources
- JST AIP Acceleration Research
- Masason Foundation, and The University of Tokyo Toyota-Dwango Scholarship
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 278Total Downloads
- Downloads (Last 12 months)87
- Downloads (Last 6 weeks)10
Reflects downloads up to 03 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format