Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3563657.3596022acmconferencesArticle/Chapter ViewAbstractPublication PagesdisConference Proceedingsconference-collections
research-article

Beyond Audio: Towards a Design Space of Headphones as a Site for Interaction and Sensing

Published: 10 July 2023 Publication History

Abstract

Via Research through Design (RtD), we explore the potential of headphones as a general-purpose input device for both foreground motion-gestures as well as background sensing of user activity. As a familiar wearable device, headphones offer a compelling site for head-situated interaction and sensing. Using emerging sensing modalities such as inertial motion, capacitive touch sensing, and depth cameras, our implemented prototypes explore sensing and interaction techniques that offer a range of compelling capabilities.
User scenarios include context-aware privacy, gestural audio-visual control, and co-opting natural body language as context to drive animated avatars for "camera-off" scenarios in remote work–or to co-opt (oft-subconscious) head movements such as dodging attacks in video games to enhance the gameplay experience.
Drawing from literature and other frameworks, we situate our prototypes and related techniques in a design space across the dual dimensions of (1) type of input (touch, mid-air, or head orientation); and (2) the context of user action (application, body, or environment). In particular, interactions that combine multiple inputs and contexts at the same time offer a rich design space of headphone-situated wearable interactions and sensing techniques.

References

[1]
2012. Exploring peripheral interaction design for primary school teachers. In Proc. of Int. Conf. on Tangible, Embedded and Embodied Interaction, TEI 2012. 245–252. https://doi.org/10.1145/2148131.2148184
[2]
2020. Earphone and Headphone Market Demand, Size, Share| Forecast 2030. Technical Report. Market Research Future. https://www.marketresearchfuture.com/reports/earphone-headphone-market-7628
[3]
2020. Headset Statistics for 2021 - UC Today. Technical Report. UC Today. https://www.uctoday.com/endpoints/headset-statistics/
[4]
Kaho Abe and Katherine Isbister. 2016. Hotaru: the lightning bug game. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 277–280.
[5]
Apple. 2020. Apple introduces AirPods Max, the magic of AirPods in a stunning over-ear design. https://www.apple.com/ng/newsroom/2020/12/apple-introduces-airpods-max-the-magic-of-airpods-in-a-stunning-over-ear-design/
[6]
S. Bakker, E. Van Den Hoven, and B. Eggen. 2015. Peripheral interaction: characteristics and considerations. Personal and Ubiquitous Computing 19, 1 (may 2015), 239–254. https://doi.org/10.1007/s00779-014-0775-2
[7]
S. Bakker, E. Van Den Hoven, D. Hausen, A. Butz, T. Selker, and B. Eggen. 2014. Peripheral interaction: Shaping the research and design space. In Conference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, 99–102. https://doi.org/10.1145/2559206.2560470
[8]
Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic interaction: designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces. 121–130.
[9]
S. M Boker, J.F. Cohn, B.-J. Theobald, I. Matthews, T. R Brick, and J.R. Spies. 2009. Effects of damping head movement and facial expression in dyadic conversation using real–time facial expression tracking and synthesized avatars. Philosophical Trans. of the Royal Society B: Biological Sciences 364, 1535 (2009), 3485–3495.
[10]
Bose. 2019. Bose Frames original version. https://www.bose.com/en_us/products/frames.html
[11]
John Bowers. 2012. The logic of annotated portfolios: Communicating the value of ’research through design’. Proceedings of the Designing Interactive Systems Conference, DIS ’12 (2012), 68–77. https://doi.org/10.1145/2317956.2317968
[12]
J. R. Brubaker, G. Venolia, and J. C. Tang. 2012. Focusing on Shared Experiences: Moving beyond the Camera in Video Communication. In Proc. of the Designing Interactive Sys. Conf.(DIS ’12). ACM, 96–105. https://doi.org/10.1145/2317956.2317973
[13]
F. Brudy, C. Holz, R. Rädle, C.-J. Wu, S. Houben, C.N. Klokmose, and N. Marquardt. 2019. Cross-device taxonomy: Survey, opportunities and challenges of interactions spanning across multiple devices. In CHI 2019. 1–28.
[14]
V. Buil, G. Hollemans, and S. van de Wijdeven. 2005. Headphones with Touch Control. In MobileHCI ’05. ACM, 377–378. https://doi.org/10.1145/1085777.1085877
[15]
W. Buxton. 1995. Integrating the periphery and context: A new taxonomy of telematics. In Proc. of graphics interface, Vol. 95. 239–246.
[16]
F.R. Castelli and M.A. Sarvary. [n.d.]. Why students do not turn on their video cameras during online classes and an equitable and inclusive plan to encourage them to do so. Ecology and Evolution n/a, n/a ([n. d.]). https://doi.org/10.1002/ece3.7123 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ece3.7123.
[17]
Y.-c. Chen, D.-y. Huang, B.-y. Chen, Y.-C. Chen, C.-Y. Liao, S.-w. Hsu, D.-Y. Huang, B.-Y. Chen, and Exploring User. 2020. Exploring User Defined Gestures for Ear-Based Interactions. https://doi.org/10.1145/1122445.1122456
[18]
E. Dagan, Márquez S.E., Altarriba B. F., M. Flores, R. Mitchell, and K.e Isbister. 2019. Design framework for social wearables. In Proc. 2019 Designing Interactive Sys. Conf.1001–1015.
[19]
L. Devendorf, J. Lo, N. Howell, J.L. Lee, N.-W. Gong, M.E. Karagozler, S. Fukuhara, I. Poupyrev, E. Paulos, and K. Ryokai. 2016. " I don’t Want to Wear a Screen" Probing Perceptions of and Possibilities for Dynamic Displays on Clothing. In CHI 16. 6028–6039.
[20]
Matjaz Divjak and Horst Bischof. 2009. Eye blink based fatigue detection for prevention of computer vision syndrome. In Proceedings of the 11th IAPR Conference on Machine Vision Applications, MVA 2009. 350–353.
[21]
T. Erickson and W.A. Kellogg. 2000. Social translucence: an approach to designing systems that support social processes. ACM TOCHI 7, 1 (2000), 59–83.
[22]
Y. Fernaeus and P. Sundström. 2012. The Material Move How Materials Matter in Interaction Design Research. In DIS ’12. ACM, 486–495. https://doi.org/10.1145/2317956.2318029
[23]
C. Frayling. 1993. Research in Art and Design. Royal College of Art Research Papers 1, 1 (1993).
[24]
Bill Gaver and John Bowers. 2012. Annotated portfolios. Interactions 19, 4 (2012), 40–49. https://doi.org/10.1145/2212877.2212889
[25]
W. Gaver. 2012. What should we expect from research through design?. In Proceedings of CHI ’12. 937–946.
[26]
S. Greenberg and B. Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proceedings CHI ’08. 111–120.
[27]
S. Greenberg, N. Marquardt, T. Ballendat, R. Diaz-Marino, and M. Wang. 2011. Proxemic interactions: the new ubicomp?interactions 18, 1 (2011), 42–50.
[28]
A. Greenfield. 2004. Everyware: The Dawning Age of Ubiquitous Computing. New Riders Publishing.
[29]
J. Grudin. 2001. Partitioning digital worlds: focal and peripheral awareness in multiple monitor use. In CHI 01. 458–465.
[30]
E.T. Hall. 1966. The hidden dimension. Vol. 609. Garden City, NY: Doubleday.
[31]
P. Hamilton and D.J. Wigdor. 2014. Conductor: enabling and understanding cross-device interaction. In CHI 14. 2773–2782.
[32]
T. Hashimoto and H. Kobayashi. 2009. Study on natural head motion in waiting state with receptionist robot SAYA that has human-like appearance. In 2009 IEEE Workshop on Robotic Intelligence in Informationally Structured Space. 93–98. https://doi.org/10.1109/RIISS.2009.4937912
[33]
M. Heijboer, E. van den Hoven, B. Bongers, and S. Bakker. 2016. Facilitating peripheral interaction: design and evaluation of peripheral interaction for a gesture-based lighting control with multimodal feedback. Personal and Ubiquitous Computing 20, 1 (feb 2016), 1–22. https://doi.org/10.1007/s00779-015-0893-5
[34]
K. Hinckley, J. Pierce, E. Horvitz, and M. Sinclair. 2005. Foreground and background interaction with sensor-enhanced mobile devices. ACM TOCHI 12, 1 (2005), 31–52.
[35]
Wang H.L. and K. Sengupta. 2001. Manipulation of remote 3D Avatar through facial feature detection and real time tracking. In IEEE ICME 2001.857–860. https://doi.org/10.1109/ICME.2001.1237857
[36]
J. Hollan, E. Hutchins, and D. Kirsh. 2000. Distributed cognition: toward a new foundation for human-computer interaction research. ACM TOCHI 7, 2 (2000), 174–196.
[37]
D.R. Olsen Jr. 2007. Evaluating user interface systems research. In Proceedings of UIST ’07. 251–258.
[38]
W. Ju and L. Leifer. 2008. The design of implicit interactions: Making interactive systems less obnoxious. Design Issues 24, 3 (2008), 72–84. https://doi.org/10.1162/desi.2008.24.3.72
[39]
H. Jung and E. Stolterman. 2010. Material probe: exploring materiality of digital artifacts. In Proc. of int. conf. on Tangible, embedded, and embodied interaction. 153–156.
[40]
H. Jung and E. Stolterman. 2012. Digital Form and Materiality: Propositions for a New Approach to Interaction Design Research. In Nordic CHI ’12. ACM, 645–654. https://doi.org/10.1145/2399016.2399115
[41]
S.a Junuzovic, K. Inkpen, J. Tang, M. Sedlins, and K. Fisher. 2012. To see or not to see: A study comparing four-way avatar, video, and audio conferencing for work. In GROUP ’12. 31–34. https://doi.org/10.1145/2389176.2389181
[42]
A. Kendon. 2002. Some uses of the head shake. Gesture 2 (Dec. 2002), 147–182. https://doi.org/10.1075/gest.2.2.03ken
[43]
A. Kendon. 2004. Gesture: Visible action as utterance. Cambridge University Press.
[44]
T. Kikuchi, Y. Sugiura, K. Masai, M. Sugimoto, and B.H. Thomas. 2017. EarTouch: Turning the ear into an input surface. In MobileHCI 2017. https://doi.org/10.1145/3098279.3098538
[45]
S.R. Klemmer, B. Hartmann, and L. Takayama. 2006. How bodies matter: five themes for interaction design. In Proc. conf. on Designing Interactive systems. 140–149.
[46]
Y. Kuno, T. Ishiyama, S. Nakanishi, and Y. Shirai. 1999. Combining Observations of Intentional and Unintentional Behaviors for Human-Computer Interaction. In CHI 99. ACM, 238–245. https://doi.org/10.1145/302979.303051
[47]
D. Ledo, S. Houben, J. Vermeulen, N. Marquardt, L. Oehlberg, and S. Greenberg. 2018. Evaluation strategies for HCI toolkit research. In Proceedings of CHI ’18. 1–17.
[48]
Juyoung Lee, Hui Shyong Yeo, Murtaza Dhuliawala, Jedidiah Akano, Junichi Shimizu, Thad Starner, Aaron Quigley, Woontack Woo, and Kai Kunze. 2017. Itchy nose: Discreet gesture interaction using EOG sensors in smart eyewear. In Proceedings - International Symposium on Wearable Computers, ISWC, Vol. Part F1305. 94–97. https://doi.org/10.1145/3123021.3123060
[49]
R. Li, J. Lee, W. Woo, and T. Starner. 2020. Kissglass: Greeting gesture recognition using smart glasses. In Proc. of the Augmented Humans Conf.1–5.
[50]
R. Lissermann, J. Huber, A. Hadjakos, S. Nanayakkara, and M. Mühlhäuser. 2014. EarPut: Augmenting ear-worn devices for ear-based interaction. In OzCHI 2014. 300–307. https://doi.org/10.1145/2686612.2686655
[51]
P. Luff, C. Heath, H. Kuzuoka, J. Hindmarsh, K. Yamazaki, and S. Oyama. 2003. Fractured ecologies: creating environments for collaboration. Human-Computer Interaction 18, 1 (June 2003), 51–84. https://doi.org/10.1207/S15327051HCI1812_3
[52]
H. Manabe and M. Fukumoto. 2011. Tap Control for Headphones without Sensors. In UIST 11. ACM, 309–314. https://doi.org/10.1145/2047196.2047236
[53]
H. Manabe and M. Fukumoto. 2012. Headphone Taps: A Simple Technique to Add Input Function to Regular Headphones. In MobileHCI ’12. ACM, 177–180. https://doi.org/10.1145/2371664.2371703
[54]
N. Marquardt, R. Diaz-Marino, S. Boring, and S. Greenberg. 2011. The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In UIST ’11. 315–326.
[55]
N. Marquardt, K. Hinckley, and S. Greenberg. 2012. Cross-device interaction via micro-mobility and f-formations. In UIST 12. 13–22.
[56]
T. Matthews, T. Rattenbury, and S. Carter. 2007. Defining, designing and evaluating peripheral displays: An analysis using activity theory. Human-Computer Interaction 22, 1-2 (2007), 221–261. https://doi.org/10.1080/07370020701307997
[57]
V. Mehta, P. Gupta, R. Subramanian, and A. Dhall. 2021. FakeBuster: A DeepFakes Detection Tool for Video Conferencing Scenarios. arXiv preprint arXiv:2101.03321 (2021).
[58]
C. Metzger, M. Anderson, and T. Starner. 2004. FreeDigiter: A contact-free device for gesture control. In ISWC. 18–21. https://doi.org/10.1109/iswc.2004.23
[59]
A.L. Molinsky, M.A. Krabbenhoft, N. Ambady, and Y.S. Choi. 2005. Cracking the Nonverbal Code: Intercultural Competence and Gesture Recognition Across Cultures. Journal of Cross-Cultural Psychology 36, 3 (May 2005), 380–395. https://doi.org/10.1177/0022022104273658
[60]
C. Neustaedter, S. Greenberg, and M. Boyle. 2006. Blur Filtration Fails to Preserve Privacy for Home-Based Video Conferencing. ACM Trans. Comput.-Hum. Interact. 13, 1 (March 2006), 1–36. https://doi.org/10.1145/1143518.1143519
[61]
C. Neustaedter, J. Procyk, A. Chua, A. Forghani, and C. Pang. 2020. Mobile Video Conferencing for Sharing Outdoor Leisure Activities Over Distance. Human–Computer Interaction 35, 2 (2020), 103–142. https://doi.org/10.1080/07370024.2017.1314186 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/07370024.2017.1314186.
[62]
M. Nicas and D. Best. 2008. A study quantifying the hand-to-face contact rate and its potential application to predicting respiratory tract infection. Journal of Occup. Environ. Hyg. 5, 6 (2008), 347–352. https://doi.org/10.1080/15459620802003896
[63]
S.Y. Oh, J. Bailenson, N.e Krämer, and B. Li. 2016. Let the avatar brighten your smile: Effects of enhancing facial expressions in virtual environments. PloS one 11, 9 (2016), e0161794.
[64]
A. Olwal, J. Moeller, G. Priest-Dorman, T. Starner, and B. Carroll. 2018. I/O Braid: Scalable touch-sensitive lighted cords using spiraling, repeating sensing textiles and fiber optics. In UIST ’18. 485–497.
[65]
Payod Panda, Molly Jane Nicholas, Mar Gonzalez-franco, Kori Inkpen, Eyal Ofek, Ross Cutler, Ken Hinckley, and Jaron Lanier. 2022. AllTogether: Effect of Avatars in Mixed-Modality Conferencing Environments. In 2022 Symposium on Human-Computer Interaction for Work (CHIWORK 2022). ACM, Durham, New Hampshire, USA. https://doi.org/10.1145/3533406.3539658
[66]
Z. Pousman and J. Stasko. 2006. A taxonomy of ambient information systems: Four patterns of design. In Proc. of Workshop on Advanced Visual Interfaces, Vol. 2006. 67–74. https://doi.org/10.1145/1133265.1133277
[67]
B. Saatçi, R. Rädle, S. Rintel, K. O’Hara, and C.N. Klokmose. 2019. Hybrid Meetings in the Modern Workplace: Stories of Success and Failure. In Int. Conference on Collaboration and Technology. Springer, 45–61.
[68]
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of IEEE CVPR ’18. 4510–4520.
[69]
S. Santosa and D.J. Wigdor. 2013. A field study of multi-device workflows in distributed workspaces. In Proc. ACM int. joint conf. on Pervasive and ubiquitous computing. 63–72.
[70]
P. Schoessler, S.-w. Leigh, K. Jagannath, P. van Hoof, and H. Ishii. 2015. Cord UIs: Controlling Devices with Augmented Cables. In TEI ’15. ACM, 395–398. https://doi.org/10.1145/2677199.2680601
[71]
D.A. Schön. 1983. The reflective practitioner : how professionals think in action. 374 pages.
[72]
J. S.E. Zimmerman and J. Forlizzi. 2010. An analysis and critique of Research through Design: towards a formalization of a research approach. In proceedings of the Conference on Designing Interactive Systems. 310–319.
[73]
L.M. Seuren, J. Wherton, T. Greenhalgh, and S.E. Shaw. 2021. Whose turn is it anyway? Latency and the organization of turn-taking in video-mediated interaction. Journal of Pragmatics 172 (2021), 63 – 78. https://doi.org/10.1016/j.pragma.2020.11.005
[74]
P.J. Stappers and E. Giaccardi. 2017. Research through design. In The encyclopedia of human-computer interaction. The Interaction Design Foundation, 1–94.
[75]
J. Sweller. 2011. Human cognitive architecture: Why some instructional procedures work and others do not. In APA educational psychology handbook, Vol 1: Theories, constructs, and critical issues.https://doi.org/10.1037/13273-011
[76]
UC Today. 2022. UC Spending Will Continue to Rise Say Firms. Technical Report. UC Today. https://www.uctoday.com/unified-communications/ucaas/uc-spending-will-continue-to-rise-say-firms/
[77]
Susie W.2017. Using internet video calls in qualitative (longitudinal) interviews: some implications for rapport. Int. Journal of Social Research Methodology 20, 6 (2017), 613–625. https://doi.org/10.1080/13645579.2016.1269505 arXiv:https://doi.org/10.1080/13645579.2016.1269505
[78]
M. Weigel and J. Steimle. 2017. DeformWear: Deformation Input on Tiny Wearable Devices. Proc. of Interact. Mob. Wearable Ubiquitous Tech. 1, 2, Article 28 (June 2017), 23 pages. https://doi.org/10.1145/3090093
[79]
Mark Weiser. 1991. The Computer for the 21 st Century. Scientific american 265, 3 (1991), 94–105.
[80]
M. Weiser and John S. Brown. 1997. The Coming Age of Calm Technology. In Beyond Calculation. 75–85. https://doi.org/10.1007/978-1-4612-0685-9_6
[81]
M. Wiberg, H. Ishii, P. Dourish, A. Vallgårda, T. Kerridge, P. Sundström, D. Rosner, and M. Rolston. 2013. Materiality Matters—Experience Materials. Interactions 20, 2 (March 2013), 54–57. https://doi.org/10.1145/2427076.2427087
[82]
C. Zhang, Y. Rui, and L. He. 2006. Light Weight Background Blurring for Video Conferencing Applications. In 2006 International Conference on Image Processing. 481–484. https://doi.org/10.1109/ICIP.2006.312498
[83]
Forlizzi J.and Evenson S Zimmerman, J.2007. Research through design as a method for interaction design research in HCI. In CHI 2007. 493–502.
[84]
A. Zolyomi, A. Begel, J.F. Waldern, J. Tang, M. Barnett, E. Cutrell, D. McDuff, S. Andrist, and M. R. Morris. 2019. Managing Stress: The Needs of Autistic Adults in Video Calling. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 134 (Nov. 2019), 29 pages. https://doi.org/10.1145/3359236

Cited By

View all
  • (2024)Warmth on Demand: Designing Headphones for Enhanced Thermal Comfort in Work EnvironmentsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650884(1-7)Online publication date: 11-May-2024
  • (2024)Virtual Games, Real Interactions: A Look at Cross-reality Asymmetrical Co-located Social GamesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650824(1-9)Online publication date: 11-May-2024
  • (2024)XDTK: A Cross-Device Toolkit for Input & Interaction in XR2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00092(467-470)Online publication date: 16-Mar-2024

Index Terms

  1. Beyond Audio: Towards a Design Space of Headphones as a Site for Interaction and Sensing

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      DIS '23: Proceedings of the 2023 ACM Designing Interactive Systems Conference
      July 2023
      2717 pages
      ISBN:9781450398930
      DOI:10.1145/3563657
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 10 July 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Badges

      • Best Paper

      Author Tags

      1. design space
      2. headphones
      3. research through design
      4. sensing
      5. wearables

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      DIS '23
      Sponsor:
      DIS '23: Designing Interactive Systems Conference
      July 10 - 14, 2023
      PA, Pittsburgh, USA

      Acceptance Rates

      Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)270
      • Downloads (Last 6 weeks)27
      Reflects downloads up to 18 Nov 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Warmth on Demand: Designing Headphones for Enhanced Thermal Comfort in Work EnvironmentsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650884(1-7)Online publication date: 11-May-2024
      • (2024)Virtual Games, Real Interactions: A Look at Cross-reality Asymmetrical Co-located Social GamesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650824(1-9)Online publication date: 11-May-2024
      • (2024)XDTK: A Cross-Device Toolkit for Input & Interaction in XR2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00092(467-470)Online publication date: 16-Mar-2024

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media