research-article

Open access

The View from MARS: Empowering Game Stream Viewers with Metadata Augmented Real-time Streaming

Authors:

Erik Harpstead,

Jessica HammerAuthors Info & Claims

UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

Article No.: 62, Pages 1 - 13

https://doi.org/10.1145/3586183.3606753

Published: 29 October 2023 Publication History

All formats PDF

Abstract

We present MARS (Metadata Augmented Real-time Streaming), a system that enables game-aware streaming interfaces for Twitch. Current streaming interfaces provide a video stream of gameplay and a chat channel for conversation, but do not allow viewers to interact with game content independently from the steamer or other viewers. With MARS, a Unity game’s metadata is rendered in real-time onto a Twitch viewer’s interface. The metadata can then power viewer-side interfaces that are aware of the streamer’s game activity and provide new capacities for viewers. Use cases include providing contextual information (e.g. clicking on a unit to learn more), improving accessibility (e.g. slowing down text presentation speed), and supporting novel stream-based game designs (e.g. asymmetric designs where the viewers know more than the streamer). We share the details of MARS’ architecture and capabilities in this paper, and showcase a working prototype for each of our three proposed use cases.

Supplemental Material

ZIP File

Supplemental File

Download
13.27 MB

References

[1]

2011. Easily create apps for PC games on the Overwolf framework. https://overwolf.github.io/. Accessed: 2023-7-21.

[2]

2018. Accessible Player Experiences (APX). https://accessible.games/accessible-player-experiences/. Accessed: 2023-4-5.

[3]

2021. Open Broadcast Software. https://obsproject.com/

[4]

2021. Tweening to create animated action. https://www.adobe.com/creativecloud/video/discover/tweening.html. Accessed: 2023-7-24.

[5]

2021. TwitchTracker Statistics and Charts. https://twitchtracker.com/statistics

[6]

Amazon Web Services. 2021. Amazon EC2 Concepts. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html. Accessed: April 5, 2023.

[7]

Jen Beeston, Christopher Power, Paul Cairns, and Mark Barlet. 2018. Accessible Player Experiences (APX): The Players. In Computers Helping People with Special Needs. Springer International Publishing, 245–253.

[8]

Sven Charleer, Kathrin Gerling, Francisco Gutiérrez, Hans Cauwenbergh, Bram Luycx, and Katrien Verbert. 2018. Real-Time Dashboards to Support eSports Spectating. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (Melbourne, VIC, Australia) (CHI PLAY ’18). Association for Computing Machinery, New York, NY, USA, 59–71.

Digital Library

[9]

Zhilong Chen, Hancheng Cao, Yuting Deng, Xuan Gao, Jinghua Piao, Fengli Xu, Yu Zhang, and Yong Li. 2021. Learning from Home: A Mixed-Methods Analysis of Live Streaming Based Remote Education Experience in Chinese Colleges during the COVID-19 Pandemic. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445428

Digital Library

[10]

Gifford Cheung and Jeff Huang. 2011. Starcraft from the stands: understanding the game spectator. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 763–772.

Digital Library

[11]

John Joon Young Chung, Hijung Valentina Shin, Haijun Xia, Li-Yi Wei, and Rubaiat Habib Kazi. 2021. Beyond Show of Hands: Engaging Viewers via Expressive and Scalable Visual Communication in Live Streaming. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21, Article 109). Association for Computing Machinery, New York, NY, USA, 1–14.

Digital Library

[12]

Nick Couldry. 2004. Liveness, “Reality,” and the Mediated Habitus from Television to the Mobile Phone. The Communication Review 7, 4 (Oct. 2004), 353–361.

[13]

Vaibhav Diwanji, Abigail Reed, Arienne Ferchaud, Jonmichael Seibert, Victoria Weinbrecht, and Nicholas Sellers. 2020. Don’t just watch, join in: Exploring information behavior and copresence on Twitch. Comput. Human Behav. 105 (April 2020), 106221.

[14]

Travis Faas, Lynn Dombrowski, Alyson Young, and Andrew D Miller. 2018. Watch me code: Programming mentorship communities on twitch. tv. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1–18.

Digital Library

[15]

Claudia Flores-Saviaga, Jessica Hammer, Juan Pablo Flores, Joseph Seering, Stuart Reeves, and Saiph Savage. 2019. Audience and Streamer Participation at Scale on Twitch. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (Hof, Germany) (HT ’19). Association for Computing Machinery, New York, NY, USA, 277–278.

Digital Library

[16]

Colin Ford, Dan Gardner, Leah Elaine Horgan, Calvin Liu, A m Tsaasan, Bonnie Nardi, and Jordan Rickman. 2017. Chat Speed OP PogChamp: Practices of Coherence in Massive Twitch Chat. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 858–871.

Digital Library

[17]

C Ailie Fraser, Joy O Kim, Alison Thornsberry, Scott Klemmer, and Mira Dontcheva. 2019. Sharing the studio: How creative livestreaming can inspire, educate, and engage. In Proceedings of the 2019 on Creativity and Cognition. 144–155.

Digital Library

[18]

Matt Makes Games. 2018. Celeste. Windows, Nintendo Switch, PlayStation 4, Xbox One.

[19]

Aron Garst. 2020. Grounded Lets You Make The Spiders Less Terrifying If You Have Arachnophobia. https://www.gamespot.com/articles/grounded-lets-you-make-the-spiders-less-terrifying/1100-6480035/

[20]

Seth Glickman, Nathan McKenzie, Joseph Seering, Rachel Moeller, and Jessica Hammer. 2018. Design Challenges for Livestreamed Audience Participation Games. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (Melbourne, VIC, Australia) (CHI PLAY ’18). Association for Computing Machinery, New York, NY, USA, 187–199.

Digital Library

[21]

Google LLC. 2021. Google Translate API. https://cloud.google.com/translate. Accessed: April 5, 2023.

[22]

Daniel Gros, Brigitta Wanner, Anna Hackenholt, Piotr Zawadzki, and Kathrin Knautz. 2017. World of Streaming. Motivation and Gratification on Twitch. In Social Computing and Social Media. Human Behavior. Springer International Publishing, 44–57.

[23]

Emily Halpin. [n. d.]. Introducing Extensions: A Streaming Revolution. https://blog.twitch.tv/en/2017/08/31/introducing-extensions-a-streaming-revolution-c31762addcd9/

[24]

William A Hamilton, Oliver Garretson, and Andruid Kerne. 2014. Streaming on twitch: fostering participatory communities of play within live mixed media. In Proceedings of the SIGCHI conference on human factors in computing systems. 1315–1324.

Digital Library

[25]

John Harris and Mark Hancock. 2019. To Asymmetry and Beyond! Improving Social Connectedness by Increasing Designed Interdependence in Cooperative Play. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19, Paper 9). Association for Computing Machinery, New York, NY, USA, 1–12.

Digital Library

[26]

John Harris, Mark Hancock, and Stacey D Scott. 2016. Leveraging Asymmetries in Multiplayer Games: Investigating Design Elements of Interdependent Play. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (Austin, Texas, USA) (CHI PLAY ’16). Association for Computing Machinery, New York, NY, USA, 350–361.

Digital Library

[27]

John Joseph Harris. 2019. Leveraging asymmetry and interdependence to enhance social connectedness in cooperative digital games. Ph. D. Dissertation.

[28]

Zorah Hilvert-Bruce, James T. Neill, Max Sjöblom, and Juho Hamari. 2018. Social motivations of live-streaming viewer engagement on Twitch. Computers in Human Behavior 84 (2018), 58–67. https://doi.org/10.1016/j.chb.2018.02.013

Digital Library

[29]

Ian Hamilton. 2021. Game Accessibility Guidelines. https://gameaccessibilityguidelines.com/why-and-how/. Accessed: April 5, 2023.

[30]

Naughty Dog Inc.2020. The Last of Us Part II. PlayStation 4.

[31]

Pascal Lessel and Maximilian Altmeyer. 2019. Understanding and empowering interactions between streamer and audience in game live streams. Interactions 27, 1 (Dec. 2019), 40–45.

Digital Library

[32]

Pascal Lessel, Maximilian Altmeyer, and Antonio Krüger. 2018. Viewers’ Perception of Elements Used in Game Live-Streams. In Proceedings of the 22nd International Academic Mindtrek Conference (Tampere, Finland) (Mindtrek ’18). Association for Computing Machinery, New York, NY, USA, 59–68.

Digital Library

[33]

Pascal Lessel, Alexander Vielhauer, and Antonio Krüger. 2017. Expanding Video Game Live-Streams with Enhanced Communication Channels: A Case Study. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 1571–1576.

Digital Library

[34]

Zhicong Lu, Michelle Annett, Mingming Fan, and Daniel Wigdor. 2019. " I feel it is my responsibility to stream" Streaming and Engaging with Intangible Cultural Heritage through Livestreaming. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[35]

Zhicong Lu, Seongkook Heo, and Daniel J Wigdor. 2018. StreamWiki: Enabling Viewers of Knowledge Sharing Live Streams to Collaboratively Generate Archival Documentation for Effective In-Stream and Post Hoc Learning. Proc. ACM Hum.-Comput. Interact. 2, CSCW (Nov. 2018), 1–26.

Digital Library

[36]

Zhicong Lu, Rubaiat Habib Kazi, Li-Yi Wei, Mira Dontcheva, and Karrie Karahalios. 2021. StreamSketch: Exploring Multi-Modal Interactions in Creative Live Streams. Proc. ACM Hum.-Comput. Interact. 5, CSCW1 (April 2021), 1–26.

Digital Library

[37]

Chance Lytle, Parker Ramsey, Joey Yeo, Trace Dressen, Dong Hyun Kang, Brenda Bakker Harger, and Jessica Hammer. 2020. Toward Live Streamed Improvisational Game Experiences. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Virtual Event, Canada) (CHI PLAY ’20). Association for Computing Machinery, New York, NY, USA, 148–159.

Digital Library

[38]

Matthew K Miller, John C Tang, Gina Venolia, Gerard Wilkinson, and Kori Inkpen. 2017. Conversational Chat Circles: Being All Here Without Having to Hear It All. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2394–2404.

Digital Library

[39]

Rui Pan, Lyn Bartram, and Carman Neustaedter. 2016. TwitchViz: A Visualization Tool for Twitch Chatrooms. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 1959–1965.

Digital Library

[40]

Karine Pires and Gwendal Simon. 2015. YouTube live and Twitch: a tour of user-generated live streaming systems. In Proceedings of the 6th ACM multimedia systems conference. 225–230.

Digital Library

[41]

Dennis Ramirez, Jenny Saucerman, and Jeremy Dietmeier. 2014. Twitch plays pokemon: a case study in big g games. In Proceedings of DiGRA. 3–6.

[42]

RealityEscape. 2019. What Makes Tower Defence Games Good?

[43]

Redis Technologies. [n. d.]. Introduction to Redis. https://redis.io/docs/about/. Accessed: 2023-4-5.

[44]

Raquel Robinson, Zachary Rubin, Elena Márquez Segura, and Katherine Isbister. 2017. All the feels: designing a tool that reveals streamers’ biometrics to spectators. In Proceedings of the 12th International Conference on the Foundations of Digital Games (Hyannis, Massachusetts) (FDG ’17, Article 36). Association for Computing Machinery, New York, NY, USA, 1–6.

Digital Library

[45]

Patrick Schuck, Maximilian Altmeyer, Antonio Krüger, and Pascal Lessel. 2022. Viewer types in game live streams: questionnaire development and validation. User Model. User-adapt Interact. 32, 3 (July 2022), 417–467.

Digital Library

[46]

Joseph Seering, Saiph Savage, Michael Eagle, Joshua Churchin, Rachel Moeller, Jeffrey P Bigham, and Jessica Hammer. 2017. Audience participation games: Blurring the line between player and spectator. In Proceedings of the 2017 Conference on Designing Interactive Systems. 429–440.

Digital Library

[47]

Max Sjöblom and Juho Hamari. 2017. Why do people watch others play video games? An empirical study on the motivations of Twitch users. Computers in human behavior 75 (2017), 985–996.

[48]

Olli Sotamaa. 2010. When the game is not enough: Motivations and practices among computer game modding culture. Games and Culture 5, 3 (2010), 239–255.

[49]

Samantha Stahlke, James Robb, and Pejman Mirza-Babaei. 2018. The Fall of the Fourth Wall: Designing and Evaluating Interactive Spectator Experiences. International Journal of Gaming and Computer-Mediated Simulations (IJGCMS) 10, 1 (2018), 42–62.

Digital Library

[50]

Streamer Tactics. [n. d.]. List of Games with Twitch Integration - Streamer Tactics. https://streamertactics.com/games-with-twitch-integration/. Accessed: 2023-4-5.

[51]

The Oh Lab. 2021. Bloomwood Stories. http://theohlab.org/bloomwood-stories. Accessed: April 5, 2023.

[52]

Zachary O Toups, Jessica Hammer, William A Hamilton, Ahmad Jarrah, William Graves, and Oliver Garretson. 2014. A framework for cooperative communication game mechanics from grounded theory. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play (Toronto, Ontario, Canada) (CHI PLAY ’14). Association for Computing Machinery, New York, NY, USA, 257–266.

Digital Library

[53]

Twitch Tv. [n. d.]. Twitch.tv - Extensions. https://www.twitch.tv/p/en/extensions/. Accessed: 2023-4-5.

[54]

Unity Technologies. 2018. Tower Defense Template. https://assetstore.unity.com/packages/essentials/tutorial-projects/tower-defense-template-107692. Accessed: April 5, 2023.

[55]

Unity Technologies. 2021. Game Objects. https://docs.unity3d.com/Manual/GameObjects.html. Accessed: April 5, 2023.

[56]

Unity Technologies. 2021. Occlusion Culling. https://docs.unity3d.com/Manual/OcclusionCulling.html. Accessed: April 5, 2023.

[57]

Andrew M Webb, Chen Wang, Andruid Kerne, and Pablo Cesar. 2016. Distributed Liveness: Understanding How New Technologies Transform Performance Experiences. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (San Francisco, California, USA) (CSCW ’16). Association for Computing Machinery, New York, NY, USA, 432–437.

Digital Library

[58]

Donghee Yvette Wohn and Guo Freeman. 2020. Audience Management Practices of Live Streamers on Twitch. In ACM International Conference on Interactive Media Experiences (Cornella, Barcelona, Spain) (IMX ’20). Association for Computing Machinery, New York, NY, USA, 106–116. https://doi.org/10.1145/3391614.3393653

Digital Library

[59]

Jason Wuertz, Sultan A Alharthi, William A Hamilton, Scott Bateman, Carl Gutwin, Anthony Tang, Zachary Toups, and Jessica Hammer. 2018. A Design Framework for Awareness Cues in Distributed Multiplayer Games. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18, Paper 243). Association for Computing Machinery, New York, NY, USA, 1–14.

Digital Library

[60]

Saelyne Yang, Jisu Yim, Juho Kim, and Hijung Valentina Shin. 2022. CatchLive: Real-time Summarization of Live Streams with Stream Content and Interaction Data. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22, Article 500). Association for Computing Machinery, New York, NY, USA, 1–20.

Digital Library

Cited By

Hammad NHarpstead EHammer J(2024)Towards a Design Framework for Data-Driven Game Streaming: A Multi-Stakeholder ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36771078:CHI PLAY(1-28)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677107
Hammad NElavsky FMoharana SChen JLee SCarrington PMoritz DHammer JHarpstead E(2024)Exploring The Affordances of Game-Aware Streaming to Support Blind and Low Vision Viewers: A Design Probe StudyProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675665(1-13)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675665

Index Terms

The View from MARS: Empowering Game Stream Viewers with Metadata Augmented Real-time Streaming
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Web-based interaction
    2. Interactive systems and tools
      1. User interface toolkits
  2. Interaction design
    1. Interaction design process and methods
      1. Interface design prototyping

Recommendations

GameAware Streaming Interfaces
CHI PLAY Companion '23: Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play

We present three game demos built using our game streaming system, MARS (Metadata Augmented Real-time Streaming). Current streaming interfaces fall short of providing viewer interaction mechanisms beyond the chat channel. Additionally, personalized ...
Tell Me Before You Stream Me: Managing Information Disclosure in Video Game Live Streaming

Live streaming is a new media format that simultaneously records and broadcasts in real time, in multiple channels including video, audio, and text. A new application area of live streaming is in the video game community, where players stream their ...
Exploring The Affordances of Game-Aware Streaming to Support Blind and Low Vision Viewers: A Design Probe Study
ASSETS '24: Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility

This paper explores new ways to support blind and low vision (BLV) game stream participants. Prior work on game-aware streaming systems has focused on the potential for viewer interaction and personalization for sighted viewers, but how such systems ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

October 2023

1825 pages

ISBN:9798400701320

DOI:10.1145/3586183

Editors:
Sean Follmer
Stanford University, USA
,
Jeff Han,
Jürgen Steimle
Saarland University, Germany
,
Nathalie Henry Riche
Microsoft Research, USA

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

UIST '23

Sponsor:

UIST '23: The 36th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 1, 2023

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
459
Total Downloads

Downloads (Last 12 months)427
Downloads (Last 6 weeks)43

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hammad NHarpstead EHammer J(2024)Towards a Design Framework for Data-Driven Game Streaming: A Multi-Stakeholder ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36771078:CHI PLAY(1-28)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677107
Hammad NElavsky FMoharana SChen JLee SCarrington PMoritz DHammer JHarpstead E(2024)Exploring The Affordances of Game-Aware Streaming to Support Blind and Low Vision Viewers: A Design Probe StudyProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675665(1-13)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675665

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents