research-article

Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch

Authors:

Daniele Gravina,

Georgios N. YannakakisAuthors Info & Claims

FDG '20: Proceedings of the 15th International Conference on the Foundations of Digital Games

Article No.: 60, Pages 1 - 10

https://doi.org/10.1145/3402942.3402958

Published: 17 September 2020 Publication History

Abstract

Is it possible to predict moment-to-moment gameplay engagement based solely on game telemetry? Can we reveal engaging moments of gameplay by observing the way the viewers of the game behave? To address these questions in this paper, we reframe the way gameplay engagement is defined and we view it, instead, through the eyes of a game’s live audience. We build prediction models for viewers’ engagement based on data collected from the popular battle royale game PlayerUnknown’s Battlegrounds as obtained from the Twitch streaming service. In particular, we collect viewers’ chat logs and in-game telemetry data from several hundred matches of five popular streamers (containing over 100,000 game events) and machine learn the mapping between gameplay and viewer chat frequency during play, using small neural network architectures. Our key findings showcase that engagement models trained solely on 40 gameplay features can reach accuracies of up to 80% on average and 84% at best. Our models are scalable and generalisable as they perform equally well within- and across-streamers, as well as across streamer play styles.

References

[1]

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, 2016. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283.

Digital Library

[2]

Sander CJ Bakkes, Pieter HM Spronck, and Giel van Lankveld. 2012. Player behavioural modelling for video games. Entertainment Computing 3, 3 (2012), 71–79.

[3]

Francesco Barbieri, Luis Espinosa Anke, Miguel Ballesteros, Juan Soler, and Horacio Saggion. 2017. Towards the understanding of gaming audiences by modeling twitch emotes. In Proceedings of the 3rd Workshop on Noisy User-generated Text. 11–20.

[4]

Richard Bartle. 1996. Hearts, clubs, diamonds, spades: Players who suit MUDs. Journal of MUD research 1, 1 (1996), 19.

[5]

Christian Bauckhage, Anders Drachen, and Rafet Sifa. 2015. Clustering game behavior data. IEEE Transactions on Computational Intelligence and AI in Games 7, 3(2015), 266–278.

[6]

Rafael A Calvo, Sidney D’Mello, Jonathan Matthew Gratch, and Arvid Kappas. 2015. The Oxford handbook of affective computing. Oxford University Press, USA.

Digital Library

[7]

Alessandro Canossa, Sasha Makarovych, Julian Togelius, and Anders Drachenn. 2018. Like a DNA String: Sequence-Based Player Profiling in Tom Clancy’s The Division. In Proceedings of the 14th Artificial Intelligence and Interactive Digital Entertainment Conference.

[8]

Peter Carruthers and Peter K Smith. 1996. Theories of theories of mind. Cambridge University Press.

[9]

Peter Chapman, Sanjeebhan Selvarajah, and Jane Webster. 1999. Engagement in multimedia training systems. In Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers. IEEE, 9–pp.

[10]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289(2015).

[11]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. ACM, 191–198.

Digital Library

[12]

James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, 2010. The YouTube video recommendation system. In Proceedings of the fourth ACM conference on Recommender systems. ACM, 293–296.

Digital Library

[13]

Anders Drachen, Alessandro Canossa, and Georgios N Yannakakis. 2009. Player modeling using self-organization in Tomb Raider: Underworld. In 2009 IEEE symposium on computational intelligence and games. IEEE, 1–8.

[14]

Anders Drachen and Shawn Connor. 2018. Games Analytics for Games User Research. In Games User Research, Anders Drachen, Pejman Mirza-Babaei, and Lennart E Nacke (Eds.). Oxford University Press, 333–355.

[15]

Magy Seif El-Nasr, Anders Drachen, and Alessandro Canossa. 2016. Game analytics. Springer. 792 pages.

[16]

Colin Ford, Dan Gardner, Leah Elaine Horgan, Calvin Liu, AM Tsaasan, Bonnie Nardi, and Jordan Rickman. 2017. Chat speed op pogchamp: Practices of coherence in massive twitch chat. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 858–871.

Digital Library

[17]

William A Hamilton, Oliver Garretson, and Andruid Kerne. 2014. Streaming on twitch: fostering participatory communities of play within live mixed media. In Proceedings of the SIGCHI conference on human factors in computing systems. 1315–1324.

Digital Library

[18]

Morgan Jennings. 2000. Theory and models for creating engaging and immersive ecommerce websites. In Proceedings of the 2000 ACM SIGCPR conference on Computer personnel research. ACM, 77–85.

Digital Library

[19]

Nicole Lazarro. 2004. Why we play games: Four keys to more emotion without story. In Game developer’s conference, San Jose.

[20]

Antonios Liapis, Daniele Gravina, Emil Kastbjerg, and Georgios N Yannakakis. 2019. Modelling the Quality of Visual Creations in Iconoscope. In International Conference on Games and Learning Alliance. Springer, 129–138.

[21]

Lassi A Liikkanen. 2013. Three Metrics for Measuring User Engagement with Online Media and a YouTube Case Study. arXiv preprint arXiv:1312.5547(2013).

[22]

Tobias Mahlmann, Anders Drachen, Julian Togelius, Alessandro Canossa, and Georgios N Yannakakis. 2010. Predicting player behavior in Tomb Raider: Underworld. In Proceedings of the Symposium on Computational Intelligence and Games (CIG). IEEE, 178–185.

[23]

Konstantinos Makantasis, Antonios Liapis, and Georgios N Yannakakis. 2019. From Pixels to Affect: A Study on Games and Player Experience. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–7.

[24]

Hector P Martinez, Georgios N Yannakakis, and John Hallam. 2014. Don’t classify ratings of affect; rank them!IEEE transactions on affective computing 5, 3 (2014), 314–326.

[25]

Akhil Mathur, Nicholas D Lane, and Fahim Kawsar. 2016. Engagement-aware computing: Modelling user engagement from mobile contexts. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 622–633.

Digital Library

[26]

David Melhart, Ahmad Azadvar, Alessandro Canossa, Antonios Liapis, and Georgios N Yannakakis. 2019. Your Gameplay Says it All: Modelling Motivation in Tom Clancy’s The Division. Proceedings of IEEE Conference on Games (CoG) (2019).

Digital Library

[27]

David Melhart, Antonios Liapis, and Georgios N Yannakakis. 2019. PAGAN: Platform for Audiovisual General-purpose ANnotation. In 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 75–76.

[28]

David Melhart, Antonios Liapis, and Georgios N Yannakakis. 2019. PAGAN: Video Affect Annotation Made Easy. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 130–136.

[29]

Ilya Musabirov, Denis Bulygin, Paul Okopny, and Ksenia Konstantinova. 2018. Between an arena and a sports bar: Online chats of esports spectators. arXiv preprint arXiv:1801.02862(2018).

[30]

Heather L O’Brien and Elaine G Toms. 2008. What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American society for Information Science and Technology 59, 6 (2008), 938–955.

Digital Library

[31]

Rui Pan, Lyn Bartram, and Carman Neustaedter. 2016. TwitchViz: a visualization tool for twitch chatrooms. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1959–1965.

Digital Library

[32]

África Periáñez, Alain Saas, Anna Guitart, and Colin Magne. 2016. Churn prediction in mobile social games: towards a complete assessment using survival ensembles. In Proceedings of the International Conference on Data Science and Advanced Analytics (DSAA). 564–573.

[33]

Matthias Rauterberg. 1995. About a framework for information and information processing of learning systems. In Information System Concepts. Springer, 54–69.

[34]

Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics 20 (1987), 53–65.

Digital Library

[35]

Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell system technical journal 27, 3 (1948), 379–423.

[36]

Max Sjöblom and Juho Hamari. 2017. Why do people watch others play video games? An empirical study on the motivations of Twitch users. Computers in Human Behavior 75 (2017), 985–996.

Digital Library

[37]

Max Sjöblom, Maria Törhönen, Juho Hamari, and Joseph Macey. 2017. Content structure is king: An empirical study on gratifications, game genres and content type on Twitch. Computers in Human Behavior 73 (2017), 161–171.

Digital Library

[38]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15, 1 (2014), 1929–1958.

Digital Library

[39]

Ruck Thawonmas and Tomohiro Harada. 2017. AI for Game Spectators: Rise of PPG. In Workshops at the Thirty-First AAAI Conference on Artificial Intelligence.

[40]

Elaine G Toms. 2002. Information interaction: Providing a framework for information architecture. Journal of the American Society for Information Science and Technology 53, 10 (2002), 855–862.

Digital Library

[41]

Markus Viljanen, Antti Airola, Jukka Heikkonen, and Tapio Pahikkala. 2018. Playtime measurement with survival analysis. IEEE Transactions on Games 10, 2 (2018), 128–138.

[42]

Joe H Ward Jr. 1963. Hierarchical grouping to optimize an objective function. Journal of the American statistical association 58, 301(1963), 236–244.

[43]

Sarah Wassermann, Nikolas Wehner, and Pedro Casas. 2019. Machine Learning Models for YouTube QoE and User Engagement Prediction in Smartphones. ACM SIGMETRICS Performance Evaluation Review 46, 3 (2019), 155–158.

Digital Library

[44]

Tim Wulf, Frank M Schneider, and Stefan Beckert. 2018. Watching players: An exploration of media enjoyment on Twitch. Games and Culture (2018), 1555412018788161.

[45]

Georgios N Yannakakis, Roddy Cowie, and Carlos Busso. 2017. The Ordinal Nature of Emotions. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 248–255.

[46]

Georgios N Yannakakis, Roddy Cowie, and Carlos Busso. 2018. The Ordinal Nature of Emotions: An Emerging Approach. IEEE Transactions on Affective Computing(2018).

[47]

Georgios N Yannakakis, Pieter Spronck, Daniele Loiacono, and Elisabeth André. 2013. Player modeling. In Dagstuhl Follow-Ups, Vol. 6. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

[48]

Georgios N. Yannakakis and Julian Togelius. 2018. Artificial Intelligence and Games. Springer Nature. http://gameaibook.org.

[49]

Nick Yee. 2006. Motivations for play in online games. CyberPsychology & behavior 9, 6 (2006), 772–775.

Cited By

Kleinman ESeif El-Nasr MPfau JKriglstein SWallner GMelhart DYannakakis GZhu JWatson BHarteveld C(2024)Ethics and Transparency in Game DataCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678859(466-470)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678859
Nguyen DMelcer EAdams D(2024)Exploring Gender and Racial/Ethnic Bias Against Video Game Streamers: Comparing Perceived Gameplay Skill and Viewer EngagementProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3650009(1-11)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3650009
Mavromoustakos-Blom PMelhárt DLiapis AYannakakis GBakkes SSpronck P(2023)Multiplayer Tension In the Wild: A Hearthstone CaseProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3582440(1-9)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1145/3582437.3582440
Show More Cited By

Recommendations

Social motivations of live-streaming viewer engagement on Twitch
Abstract
Little is known about the motivations underlying viewer engagement in the rapidly growing live-streaming multimedia phenomenon. This study trialled an eight-factor socio-motivational model, based on Uses and Gratifications Theory, to ...
Highlights
- Live-stream viewer engagement was explored from a socio-motivational perspective.
Investigating Players' Engagement, Immersion, and Experiences in Playing Pokémon Go
C&C '17: Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition

In recent years, Augmented Reality (AR) based mobile games have become popular among players. The Pokémon Go is one of the well-known examples. Although Pokémon Go game has become a global phenomenon, there is a limited study about players' experiences ...
Enjoyment or engagement? role of social interaction in playing massively mulitplayer online role-playing games (MMORPGS)
ICEC'06: Proceedings of the 5th international conference on Entertainment Computing

Based on data collected through 40 in-depth interviews, it is found that (a) the balance between perceived challenges and skills, and (b) the types of in-game social interactions can both facilitate and impede the enjoyment of game playing. Through ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

FDG '20: Proceedings of the 15th International Conference on the Foundations of Digital Games

September 2020

804 pages

ISBN:9781450388078

DOI:10.1145/3402942

Editors:
Georgios N. Yannakakis
University of Malta
,
Antonios Liapis
University of Malta
,
Penny Kyburz
Australian National University
,
Vanessa Volz
Modl.AI
,
Foaad Khosmood
California Polytechnic State University
,
Phil Lopes
École Polytechnique Fédèrale de Lausanne

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

FDG '20

FDG '20: International Conference on the Foundations of Digital Games

September 15 - 18, 2020

Bugibba, Malta

Acceptance Rates

Overall Acceptance Rate 152 of 415 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
318
Total Downloads

Downloads (Last 12 months)49
Downloads (Last 6 weeks)4

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kleinman ESeif El-Nasr MPfau JKriglstein SWallner GMelhart DYannakakis GZhu JWatson BHarteveld C(2024)Ethics and Transparency in Game DataCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678859(466-470)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678859
Nguyen DMelcer EAdams D(2024)Exploring Gender and Racial/Ethnic Bias Against Video Game Streamers: Comparing Perceived Gameplay Skill and Viewer EngagementProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3650009(1-11)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3650009
Mavromoustakos-Blom PMelhárt DLiapis AYannakakis GBakkes SSpronck P(2023)Multiplayer Tension In the Wild: A Hearthstone CaseProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3582440(1-9)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1145/3582437.3582440
Pinitas KRenaudie DThomsen MBarthet MMakantasis KLiapis AYannakakis G(2023)Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad ActionsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614203(488-497)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614203
Makantasis KLiapis AYannakakis G(2023)The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in GamesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.306087714:1(680-693)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TAFFC.2021.3060877
Yannakakis GMelhart D(2023)Affective Game Computing: A SurveyProceedings of the IEEE10.1109/JPROC.2023.3315689111:10(1423-1444)Online publication date: Oct-2023
https://doi.org/10.1109/JPROC.2023.3315689
Haposan DHidayat MSantosa AIbrahim MRizki Nur Majiid M(2023)Estimated Twitch Streamer Revenue Using Linear Regression Algorithm2023 5th International Conference on Cybernetics and Intelligent System (ICORIS)10.1109/ICORIS60118.2023.10352194(1-5)Online publication date: 6-Oct-2023
https://doi.org/10.1109/ICORIS60118.2023.10352194
Melhart DLiapis AYannakakis G(2022)The Arousal Video Game AnnotatIoN (AGAIN) DatasetIEEE Transactions on Affective Computing10.1109/TAFFC.2022.318885113:4(2171-2184)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TAFFC.2022.3188851
Indulkar Y(2021)PUBG Winner Ranking Prediction using R Interface ‘h2o’ Scalable Machine Learning Platform2021 International Conference on Emerging Smart Computing and Informatics (ESCI)10.1109/ESCI50559.2021.9396823(300-305)Online publication date: 5-Mar-2021
https://doi.org/10.1109/ESCI50559.2021.9396823
Melhart DLiapis AYannakakis G(2021)Towards General Models of Player Experience: A Study Within Genres2021 IEEE Conference on Games (CoG)10.1109/CoG52621.2021.9618902(01-08)Online publication date: 17-Aug-2021
https://dl.acm.org/doi/10.1109/CoG52621.2021.9618902
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents