StructureSense: Inferring Constructive Assembly Structures from User Behaviors
Authors: 
Xincheng Huang, 
Keylonnie L. Miller, Alanson P. Sample, 
Nikola BanovicAuthors Info & Claims
Article No.: 204, Pages 1 - 25
Abstract
Recent advancements in object-tracking technologies can turn mundane constructive assemblies into Tangible User Interfaces (TUI) media. Users rely on instructions or their own creativity to build both permanent and temporary structures out of such objects. However, most existing object-tracking technologies focus on tracking structures as monoliths, making it impossible to infer and track the user's assembly process and the resulting structures. Technologies that can track the assembly process often rely on specially fabricated assemblies, limiting the types of objects and structures they can track. Here, we present StructureSense, a tracking system based on passive UHF-RFID sensing that infers constructive assembly structures from object motion. We illustrated StructureSense in two use cases (as guided instructions and authoring tool) on two different constructive sets (wooden lamp and Jumbo Blocks), and evaluated system performance and usability. Our results showed the feasibility of using StructureSense to track mundane constructive assembly structures.
Supplementary Material
Supplemental movie, appendix, image and software files for, StructureSense: Inferring Constructive Assembly Structures from User Behaviors
- Download
- 106.31 MB
References
[1]
David Anderson, James L. Frankel, Joe Marks, Aseem Agarwala, Paul Beardsley, Jessica Hodgins, Darren Leigh, Kathy Ryall, Eddie Sullivan, and Jonathan S. Yedidia. 2000. Tangible Interaction + Graphical Interpretation: A New Approach to 3D Modeling. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). ACM Press/Addison-Wesley Publishing Co., USA, 393--402. https://doi.org/10.1145/344779.344960
[2]
David Anderson, James L. Frankel, Joe Marks, Darren Leigh, Eddie Sullivan, Jonathan Yedidia, and Kathy Ryall. 1999. Building Virtual Structures with Physical Blocks. In Proceedings of the 12th Annual ACM Symposium on User Interface Software and Technology (Asheville, North Carolina, USA) (UIST '99). Association for Computing Machinery, New York, NY, USA, 71--72. https://doi.org/10.1145/320719.322587
[3]
Masahiro Ando, Yuichi Itoh, Toshiki Hosoi, Kazuki Takashima, Kosuke Nakajima, and Yoshifumi Kitamura. 2014. StackBlock: Block-Shaped Interface for Flexible Stacking. In Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST'14 Adjunct). Association for Computing Machinery, New York, NY, USA, 41--42. https://doi.org/10.1145/2658779.2659104
[4]
Patrick Baudisch, Torsten Becker, and Frederik Rudeck. 2010. Lumino: Tangible blocks for tabletop computers based on glass fiber bundles, Vol. 2. 1165--1174. https://doi.org/10.1145/1753326.1753500
[5]
James O Berger. 1985. Statistical Decision Theory and Bayesian Analysis. Springer New York.
[6]
Jose Bernardo and Adrian Smith. 2000. Bayesian Theory. Vol. 15. https://doi.org/10.2307/2983298
[7]
Kévin Bouchard, Dany Fortin-Simard, Sebastien Gaboury, Bruno Bouchard, and Abdenour Bouzouane. 2014. Accurate Trilateration for Passive RFID Localization in Smart Homes. International Journal of Wireless Information Networks 21 (03 2014), 1--18. https://doi.org/10.1007/s10776-013-0234-4
[8]
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. 2011. Handbook of Markov Chain Monte Carlo. CRC press.
[9]
Michael Buettner, Richa Prasad, Matthai Philipose, and David Wetherall. 2009. Recognizing Daily Activities with RFID-Based Sensors. In Proceedings of the 11th International Conference on Ubiquitous Computing (Orlando, Florida, USA) (UbiComp '09). Association for Computing Machinery, New York, NY, USA, 51--60. https://doi.org/10.1145/1620545.1620553
[10]
Liwei Chan, Stefanie Mueller, Anne Roudaut, and Patrick Baudisch. 2012. CapStones and ZebraWidgets: Sensing Stacks of Building Blocks, Dials and Sliders on Capacitive Touch Screens. https://doi.org/10.1145/2207676.2208371
[11]
Kirti Chawla and Gabriel Robins. 2011. An RFID-based object localisation framework. IJRFITA 3 (2011), 2--30.
[12]
Markus Funk, Thomas Kosch, and Albrecht Schmidt. 2016. Interactive Worker Assistance: Comparing the Effects of in-Situ Projection, Head-Mounted Displays, Tablet, and Paper Instructions. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp '16). Association for Computing Machinery, New York, NY, USA, 934--939. https://doi.org/10.1145/2971648.2971706
[13]
W.R. Gilks, S. Richardson, and D. Spiegelhalter. 1995. Markov Chain Monte Carlo in Practice. Taylor & Francis.
[14]
Matthew Gorbet, Maggie Orth, and Hiroshi Ishii. 1998. Triangles: Tangible Interface for Manipulation and Exploration of Digital Information Topography. 49--56.
[15]
Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen. 2012. DuploTrack: A real-time system for authoring and guiding duplo block assembly. UIST'12 - Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, 389--402. https://doi.org/10.1145/2380116.2380167
[16]
Toshiki Hosoi, Kazuki Takashima, Tomoaki Adachi, Yuichi Itoh, and Yoshifumi Kitamura. 2014. A-Blocks: Recognizing and Assessing Child Building Processes during Play with Toy Blocks. In SIGGRAPH Asia 2014 Emerging Technologies (Shenzhen, China) (SA '14). Association for Computing Machinery, New York, NY, USA, Article 1, 2 pages. https://doi.org/10.1145/2669047.2669061
[17]
Meng-Ju Hsieh, Rong-Hao Liang, Jr-Ling Guo, and Bing-Yu Chen. 2018. RFIDesk: an interactive surface for multi-touch and rich-ID stackable tangible interactions. In SIGGRAPH Asia 2018 Emerging Technologies (Tokyo, Japan) (SA '18, Article 11). Association for Computing Machinery, New York, NY, USA, 1--2.
[18]
Meng-Ju Hsieh, Rong-Hao Liang, Da-Yuan Huang, Jheng-You Ke, and Bing-Yu Chen. 2018. RFIBricks: Interactive Building Blocks Based on RFID. 1--10. https://doi.org/10.1145/3173574.3173763
[19]
Koshi Ikegawa, Masaya Tsuruta, Tetsuya Abe, Arika Yoshida, Buntarou Shizuki, and Shin Takahashi. 2016. Lightweight Capacitance-based Block System for 3D Space Interaction. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS '16). Association for Computing Machinery, New York, NY, USA, 307--312.
[20]
Hiroshi Ishii and Brygg Ullmer. 1997. Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '97). Association for Computing Machinery, New York, NY, USA, 234--241. https://doi.org/10.1145/258549.258715
[21]
C. Jiang, Y. He, X. Zheng, and Y. Liu. 2018. Orientation-Aware RFID Tracking with Centimeter-Level Accuracy. In 2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). 290--301. https://doi.org/10.1109/IPSN.2018.00057
[22]
Ricardo Jota and Hrvoje Benko. 2011. Constructing Virtual 3D Models with Physical Building Blocks. In CHI '11 Extended Abstracts on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI EA '11). Association for Computing Machinery, New York, NY, USA, 2173--2178. https://doi.org/10.1145/1979742.1979915
[23]
Mandeep Kaur, Manjeet Sandhu, Neeraj Mohan, and Parvinder Sandhu. 2011. RFID Technology Principles, Advantages,Limitations and Its Applications. International Journal of Computer and Electrical Engineering 3(01 2011), 151--157. https://doi.org/10.7763/IJCEE.2011.V3.306
[24]
Bui Minh Khuong, Kiyoshi Kiyokawa, Andrew Miller, Joseph J. La Viola, Tomohiro Mashita, and Haruo Takemura. 2014. The effectiveness of an AR-based context-aware assembly support system in object assembly. In 2014 IEEE Virtual Reality (VR). 57--62. https://doi.org/10.1109/VR.2014.6802051
[25]
Benjamin Lafreniere, Tovi Grossman, Fraser Anderson, Justin Matejka, Heather Kerrick, Danil Nagy, Lauren Vasey, Evan Atherton, Nicholas Beirne, Marcelo H. Coelho, Nicholas Cote, Steven Li, Andy Nogueira, Long Nguyen, Tobias Schwinn, James Stoddart, David Thomasson, Ray Wang, Thomas White, David Benjamin, Maurice Conti, Achim Menges, and George Fitzmaurice. 2016. Crowdsourced Fabrication. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST '16). Association for Computing Machinery, New York, NY, USA, 15--28. https://doi.org/10.1145/2984511.2984553
[26]
Mannu Lambrichts, Raf Ramakers, Steve Hodges, Sven Coppers, and James Devine. 2021. A Survey and Taxonomy of Electronics Toolkits for Interactive and Ubiquitous Device Prototyping. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 2, Article 70 (jun 2021), 24 pages. https://doi.org/10.1145/3463523
[27]
Danny Leen, Raf Ramakers, and Kris Luyten. 2017. StrutModeling: A Low-Fidelity Construction Kit to Iteratively Model, Test, and Adapt 3D Objects. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST '17). Association for Computing Machinery, New York, NY, USA, 471--479.
[28]
Hanchuan Li, Eric Brockmeyer, Elizabeth J. Carter, Josh Fromm, Scott E. Hudson, Shwetak N. Patel, and Alanson Sample. 2016. PaperID: A Technique for Drawing Functional Battery-Free Wireless Interfaces on Paper. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 5885--5896. https://doi.org/10.1145/2858036.2858249
[29]
Hanchuan Li, Can Ye, and Alanson P. Sample. 2015. IDSense: A Human Object Interaction Detection System Based on Passive UHF RFID. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 2555--2564. https://doi.org/10.1145/2702123.2702178
[30]
Rong-Hao Liang, Liwei Chan, Hung-Yu Tseng, Han-Chih Kuo, Da-Yuan Huang, De-Nian Yang, and Bing-Yu Chen. 2014. GaussBricks. 587--590. https://doi.org/10.1145/2559206.2574776
[31]
Haishu Ma and Kesheng Wang. 2017. Fusion of RSS and Phase Shift Using the Kalman Filter for RFID Tracking. IEEE Sensors Journal 17, 11 (2017), 3551--3558. https://doi.org/10.1109/JSEN.2017.2696054
[32]
Timothy McNerney. 2004. From turtles to Tangible Programming Bricks: Explorations in physical language design. Personal and Ubiquitous Computing 8 (09 2004), 326--337. https://doi.org/10.1007/s00779-004-0295-6
[33]
Andrew Miller, Brandyn White, Emiko Charbonneau, Zach Kanzler, and Joseph J LaViola, Jr. 2012. Interactive 3D model acquisition and tracking of building block structures. IEEE Trans. Vis. Comput. Graph. 18, 4 (April 2012), 651--659.
[34]
Andrea Motroni, Paolo Nepa, Alice Buffi, and Bernardo Tellini. 2019. A Phase-Based Method for Mobile Node Localization through UHF-RFID Passive Tags. In 2019 IEEE International Conference on RFID Technology and Applications (RFID-TA). 470--475. https://doi.org/10.1109/RFID-TA.2019.8892264
[35]
Brad A. Myers. 1985. The Importance of Percent-Done Progress Indicators for Computer-Human Interfaces. SIGCHI Bull. 16, 4 (April 1985), 11--17. https://doi.org/10.1145/1165385.317459
[36]
Jakob Nielsen. 1989. Usability Engineering at a Discount. In Proceedings of the Third International Conference on Human-Computer Interaction on Designing and Using Human-Computer Interfaces and Knowledge Based Systems (2nd Ed.) (Boston, Massachusetts, USA). Elsevier Science Inc., USA, 394--401.
[37]
Jakob Nielsen. 1994. Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[38]
Hayes Solos Raffle, Amanda J Parkes, and Hiroshi Ishii. 2004. Topobo: a constructive assembly system with kinetic memory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI '04). Association for Computing Machinery, New York, NY, USA, 647--654.
[39]
Andrew Spielberg, Alanson Sample, Scott E. Hudson, Jennifer Mankoff, and James McCann. 2016. RapID: A Framework for Fabricating Low-Latency Interactive Objects with RFID Tags. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 5897--5908. https://doi.org/10.1145/2858036.2858243
[40]
Anna Syberfeldt, Oscar Danielsson, Magnus Holm, and Lihui Wang. 2015. Visual Assembling Guidance Using Augmented Reality. Procedia Manufacturing 1 (2015), 98--109. https://doi.org/10.1016/j.promfg.2015.09.068 43rd North American Manufacturing Research Conference, NAMRC 43, 8--12 June 2015, UNC Charlotte, North Carolina, United States.
[41]
Jeppe U. Walther, J. Andreas Bærentzen, and Henrik Aanæs. 2016. Tangible 3D Modeling of Coherent and Themed Structures. Comput. Graph. 58, C (Aug. 2016), 53--65. https://doi.org/10.1016/j.cag.2016.05.004
[42]
Jue Wang, Fadel Adib, Ross Knepper, Dina Katabi, and Daniela Rus. 2013. RF-Compass: Robot Object Manipulation Using RFIDs. In Proceedings of the 19th Annual International Conference on Mobile Computing; Networking (Miami, Florida, USA) (MobiCom '13). Association for Computing Machinery, New York, NY, USA, 3--14. https://doi.org/10.1145/2500423.2500451
[43]
John Williamson, Antti Oulasvirta, Per Ola Kristensson, and Nikola Banovic (Eds.). 2022. Bayesian Methods for Interaction and Design. Cambridge University Press. https://doi.org/10.1017/9781108874830
[44]
Nesra Yannier, Scott Hudson, and Kenneth Koedinger. 2020. Active Learning is About More Than Hands-On: A Mixed-Reality AI System to Support STEM Education. International Journal of Artificial Intelligence in Education (02 2020). https://doi.org/10.1007/s40593-020-00194-3
[45]
Hui-Shyong Yeo, Ryosuke Minami, Kirill Rodriguez, George Shaker, and Aaron Quigley. 2018. Exploring Tangible Interactions with Radar Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 200 (dec 2018), 25 pages. https://doi.org/10.1145/3287078
[46]
Hui-Shyong Yeo and Aaron Quigley. 2017. Radar Sensing in Human-Computer Interaction. Interactions 25, 1 (dec 2017), 70--73. https://doi.org/10.1145/3159651
Index Terms
- StructureSense: Inferring Constructive Assembly Structures from User Behaviors
Recommendations
Novel interaction techniques using touch-sensitive tangibles in tabletop environments
ITS '12: Proceedings of the 2012 ACM international conference on Interactive tabletops and surfacesIn this work, we propose techniques for interaction that use a touch-sensitive tangible to assist 3D manipulation in tabletop applications. The objective of this research is to investigate the effectiveness and user satisfaction with this combination ...
Vision-Driven: Beyond Tangible Bits, Towards Radical Atoms
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing SystemsOur vision-driven design research is carried out through an artistic approach. Whereas much of today's mainstream Human Computer Interaction (HCI) research addresses functional concerns the needs of users, practical applications, and usability ...
Installation to teach energy conservation to kids through tangible objects in an outdoor environment
IITM '10: Proceedings of the First International Conference on Intelligent Interactive Technologies and MultimediaThis paper presents a design case of an interactive installation in a contextual outdoor environment for helping kids learn about how to save energy in their day to day life activities. Playful Interaction, physical movement and manipulation of tangible ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 January 2023
Published in IMWUT Volume 6, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 200Total Downloads
- Downloads (Last 12 months)93
- Downloads (Last 6 weeks)2
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in