research-article

Open access

Semi-automated Analysis of Collaborative Interaction: Are We There Yet?

Authors:

Thomas Neumayr,

Mirjam Augstein,

Johannes Schönböck,

Thomas TeichmeisterAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue ISS

Article No.: 571, Pages 354 - 380

https://doi.org/10.1145/3567724

Published: 14 November 2022 Publication History

Abstract

In recent years, research on collaborative interaction has relied on manual coding of rich audio/video recordings. The fine-grained analysis of such material is extremely time-consuming and labor-intensive. This is not only difficult to scale, but, as a result, might also limit the quality and completeness of coding due to fatigue, inherent human biases, (accidental or intentional), and inter-rater inconsistencies. In this paper, we explore how recent advances in machine learning may reduce manual effort and loss of information while retaining the value of human intelligence in the coding process. We present ACACIA (AI Chain for Augmented Collaborative Interaction Analysis), an AI video data analysis application which combines a range of advances in machine perception of video material for the analysis of collaborative interaction. We evaluate ACACIA's abilities, show how far we can already get, and which challenges remain. Our contribution lies in establishing a combined machine and human analysis pipeline that may be generalized to different collaborative settings and guide future research.

Supplementary Material

Auxiliary Archive (iss22main-id9934-p-archive.zip)

Supplemental video and captions file.

Download
49.31 MB

Teaser (iss22main-id9934-p-teaser.mp4)

Supplemental video and captions file.

Download
48.25 MB

References

[1]

2019. act4teams® Handbuch Version 2.3.

[2]

Robert F Bales. 1950. Interaction process analysis; a method for the study of small groups. Addison-Wesley, Boston, MA, USA.

[3]

David Benavides, Alexander Felfernig, José A Galindo, and Florian Reinfrank. 2013. Automated analysis in feature modelling and product configuration. In International Conference on Software Reuse. Springer, Berlin, Heidelberg, Berlin, Germany. 160–175. https://doi.org/10.1007/978-3-642-38977-1_11

[4]

Dan Bohus, Sean Andrist, Ashley Feniello, Nick Saw, Mihai Jalobeanu, Patrick Sweeney, Anne Loomis Thompson, and Eric Horvitz. 2021. Platform for Situated Intelligence. arxiv:2103.15975.

[5]

2018. The Cambridge Handbook of Group Interaction Analysis, Elisabeth Brauner, Margarete Boos, and Michaela Kolbe (Eds.). Cambridge University Press, Cambridge. isbn:978-1-107-11333-6 https://doi.org/10.1017/9781316286302

[6]

Frederik Brudy, Joshua Kevin Budiman, Steven Houben, and Nicolai Marquardt. 2018. Investigating the Role of an Overview Device in Multi-Device Collaboration. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. 300. https://doi.org/10.1145/3173574.3173874

Digital Library

[7]

Frederik Brudy, Suppachai Suwanwatcharachat, Wenyu Zhang, Steven Houben, and Nicolai Marquardt. 2018. EagleView: A Video Analysis Tool for Visualising and Querying Spatial Interactions of People and Devices. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces (ISS ’18). Association for Computing Machinery, New York, NY, USA. 61–72. isbn:978-1-4503-5694-7 https://doi.org/10.1145/3279778.3279795

Digital Library

[8]

Jessica Burggraaff, Jonas Dorn, Marcus D’Souza, Cecily Morrison, Christian P. Kamm, Peter Kontschieder, Prejaas Tewarie, Saskia Steinheimer, Abigail Sellen, Frank Dahlke, Ludwig Kappos, and Bernard Uitdehaag. 2020. Video-Based Pairwise Comparison: Enabling the Development of Automated Rating of Motor Dysfunction in Multiple Sclerosis. Archives of Physical Medicine and Rehabilitation, 101, 2 (2020), 234–241. issn:0003-9993 https://doi.org/10.1016/j.apmr.2019.07.016

[9]

Bill Buxton. 2009. Mediaspace – Meaningspace – Meetingspace. In Media Space 20 + Years of Mediated Life, Steve Harrison (Ed.). Springer, London. 217–231. isbn:978-1-84882-483-6 https://doi.org/10.1007/978-1-84882-483-6_13

[10]

Alessandro Del Sole. 2018. Introducing Microsoft Cognitive Services. Apress, Berkeley, CA. 1–4. isbn:978-1-4842-3342-9 https://doi.org/10.1007/978-1-4842-3342-9_1

[11]

Manuel Gonzalez-Rivero, Oscar Beijbom, Alberto Rodriguez-Ramirez, Dominic EP Bryant, Anjani Ganase, Yeray Gonzalez-Marrero, Ana Herrera-Reveles, Emma V Kennedy, Catherine JS Kim, and Sebastian Lopez-Marcano. 2020. Monitoring of coral reefs using artificial intelligence: A feasible and cost-effective approach. Remote Sensing, 12, 3 (2020), 489. https://doi.org/10.3390/rs12030489

[12]

Germán González and Conor L. Evans. 2019. Biomedical Image Processing with Containers and Deep Learning: An Automated Analysis Pipeline: Data architecture, artificial intelligence, automated processing, containerization, and clusters orchestration ease the transition from data acquisition to insights in medium-to-large datasets. BioEssays, 41, 6 (2019), 1900004. https://doi.org/10.1002/bies.201900004 Publisher: Wiley Online Library

[13]

Jessica Harris, Maryanne Theobald, Susan Danby, Edward Reynolds, Sean Rintel, and Members of the Transcript Analysis Group (Tag). 2012. ‘What’s going on here?’ The pedagogy of a data analysis session. In Reshaping Doctoral Education, Alison Lee and Susan Danby (Eds.). Routledge, London, UK. 109–121.

[14]

Yashar D Hezaveh, Laurence Perreault Levasseur, and Philip J Marshall. 2017. Fast automated analysis of strong gravitational lenses with convolutional neural networks. Nature, 548, 7669 (2017), 555–557. https://doi.org/10.1038/nature23463

[15]

Jesse Hoey, Tobias Schröder, Jonathan Morgan, Kimberly B Rogers, Deepak Rishi, and Meiyappan Nagappan. 2018. Artificial intelligence and social simulation: Studying group dynamics on a massive scale. Small Group Research, 49, 6 (2018), 647–683. https://doi.org/10.1177/1046496418802362

[16]

Petra Isenberg, Danyel Fisher, Sharoda A. Paul, Meredith Ringel Morris, Kori Inkpen, and Mary Czerwinski. 2012. Co-located collaborative visual analytics around a tabletop display. IEEE Transactions on visualization and Computer Graphics, 18, 5 (2012), 689–702. https://doi.org/10.1109/TVCG.2011.287

Digital Library

[17]

Gail Jefferson. 2004. Glossary of transcript symbols with an introduction. Pragmatics and beyond new series, 125 (2004), 13–34.

[18]

Jelliffe Jeganathan, Ziyad Knio, Yannis Amador, Ting Hai, Arash Khamooshian, Robina Matyal, Kamal R Khabbaz, and Feroze Mahmood. 2017. Artificial intelligence in mitral valve analysis. Annals of cardiac anaesthesia, 20, 2 (2017), 129.

[19]

Stephen Jolly. 2000. Understanding body language: Birdwhistell’s theory of kinesics. Corporate Communications: An International Journal, 5 (2000), Sept., 133–139. https://doi.org/10.1108/13563280010377518

[20]

Simone Kauffeld, Nale Lehmann-Willenbrock, and Annika L. Meinecke. 2018. The Advanced Interaction Analysis for Teams (act4teams) Coding Scheme. In The Cambridge Handbook of Group Interaction Analysis, Elisabeth Brauner, Margarete Boos, and Michaela Kolbe (Eds.). Cambridge University Press, Cambridge. 422–431. isbn:978-1-107-11333-6 https://doi.org/10.1017/9781316286302.022

[21]

Simone Kauffeld and Annika L. Meinecke. 2018. History of group interaction research. In The Cambridge handbook of group interaction analysis. Cambridge University Press, New York, NY, US. 20–42. isbn:978-1-107-53387-5 978-1-107-11333-6 https://doi.org/10.1017/9781316286302.003

[22]

Pawel Korzynski, Michael Haenlein, and Mika Rautiainen. 2021. Impression management techniques in crowdfunding: An analysis of Kickstarter videos using artificial intelligence. European Management Journal, 39, 5 (2021), 675–684. issn:0263-2373 https://doi.org/10.1016/j.emj.2021.01.001

[23]

Melvin Kranzberg. 1986. Technology and History: "Kranzberg’s Laws". Technology and Culture, 27, 3 (1986), 544–560. issn:0040165X, 10973729 https://doi.org/10.2307/3105385

[24]

Oren Z Kraus, Ben T Grys, Jimmy Ba, Yolanda Chong, Brendan J Frey, Charles Boone, and Brenda J Andrews. 2017. Automated analysis of high-content microscopy data with deep learning. Molecular systems biology, 13, 4 (2017), 924. https://doi.org/10.15252/msb.20177551

[25]

Kenya Kusunose, Akihiro Haga, Takashi Abe, and Masataka Sata. 2019. Utilization of artificial intelligence in echocardiography. Circulation Journal: Official Journal of the Japanese Circulation Society, 83, 8 (2019), 1623–1629. https://doi.org/10.1253/circj.CJ-19-0420

[26]

Curtis Lebaron, Paula Jarzabkowski, Michael Pratt, and Greg Fetzer. 2017. An Introduction to Video Methods in Organizational Research. Organizational Research Methods, 21 (2017), 10, https://doi.org/10.1177/1094428117745649

[27]

2014. Interact 14 Benutzerhandbuch Version 14.1.4.

[28]

Lorenza Mondada. 2018. Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality. Research on Language and Social Interaction, 51, 1 (2018), 85–106.

[29]

Cecily Morrison, Edward Cutrell, Martin Grayson, Anja Thieme, Alex Taylor, Geert Roumen, Camilla Longden, Sebastian Tschiatschek, Rita Faia Marques, and Abigail Sellen. 2021. Social Sensemaking with AI: Designing an Open-Ended AI Experience with a Blind Child. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA. Article 396, 14 pages. isbn:9781450380966 https://doi.org/10.1145/3411764.3445290

Digital Library

[30]

Cecily Morrison, Kit Huckvale, Bob Corish, Jonas Dorn, Peter Kontschieder, Kenton O’Hara, ASSESS MS Team, Antonio Criminisi, and Abigail Sellen. 2016. Assessing Multiple Sclerosis with Kinect: Designing Computer Vision Systems for Real-World Use. Human-Computer Interaction, 31, 3-4 (2016), January, 191–226. https://doi.org/10.1080/07370024.2015.1093421

Digital Library

[31]

Misgana Negassi, Rodrigo Suarez-Ibarrola, Simon Hein, Arkadiusz Miernik, and Alexander Reiterer. 2020. Application of artificial neural networks for automated analysis of cystoscopic images: a review of the current status and future prospects. World journal of urology, 38 (2020), 2349–2358. https://doi.org/10.1007/s00345-019-03059-0

[32]

Thomas Neumayr, Hans-Christian Jetter, Mirjam Augstein, Judith Friedl, and Thomas Luger. 2018. Domino: A Descriptive Framework for Hybrid Collaboration and Coupling Styles in Partially Distributed Teams. Proceedings of the ACM on Human-Computer Interaction, 2, CSCW (2018), 128. https://doi.org/10.1145/3274397

Digital Library

[33]

Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria-Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, Ioannis Kompatsiaris, Katharina Kinder-Kurlanda, Claudia Wagner, Fariba Karimi, Miriam Fernandez, Harith Alani, Bettina Berendt, Tina Kruegel, Christian Heinze, Klaus Broelemann, Gjergji Kasneci, Thanassis Tiropanis, and Steffen Staab. 2020. Bias in Data-Driven Artificial Intelligence Systems – An Introductory Survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10, 3 (2020), 1–14. https://doi.org/10.1002/widm.1356

[34]

M. B. Parten. 1932. Social participation among pre-school children. The Journal of Abnormal and Social Psychology, 27, 3 (1932), 243–269. issn:0096-851X(Print) https://doi.org/10.1037/h0074524 Place: US Publisher: American Psychological Association

[35]

Rebecca S Portnoff, Sadia Afroz, Greg Durrett, Jonathan K Kummerfeld, Taylor Berg-Kirkpatrick, Damon McCoy, Kirill Levchenko, and Vern Paxson. 2017. Tools for automated analysis of cybercriminal markets. In Proceedings of the 26th International Conference on World Wide Web. ACM, New York, NY, USA. 657–666. https://doi.org/10.1145/3038912.3052600

Digital Library

[36]

Ramachandran Rajalakshmi, Radhakrishnan Subashini, Ranjit Mohan Anjana, and Viswanathan Mohan. 2018. Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye, 32, 6 (2018), 1138–1144. https://doi.org/10.1038/s41433-018-0064-9

[37]

Johnny Saldaña. 2015. The coding manual for qualitative researchers. Sage, London, UK.

[38]

Carsten C Schermuly and Wolfgang Scholl. 2012. The Discussion Coding System (DCS)—A new instrument for analyzing communication processes. Communication Methods and Measures, 6, 1 (2012), 12–40. https://doi.org/10.1080/19312458.2011.651346

[39]

Carsten C Schermuly and Franziska Schölmerich. 2017. Analyse von Gruppen in Organisationen. In Handbuch Empirische Organisationsforschung. Springer Fachmedien Wiesbaden, Wiesbaden, Germany. 491–512. https://doi.org/10.1007/978-3-658-08493-6_18

[40]

Ursula Schmidt-Erfurth, Sebastian M Waldstein, Sophie Klimscha, Amir Sadeghipour, Xiaofeng Hu, Bianca S Gerendas, Aaron Osborne, and Hrvoje Bogunović. 2018. Prediction of individual disease conversion in early AMD using artificial intelligence. Investigative ophthalmology & visual science, 59, 8 (2018), 3199–3208. https://doi.org/10.1167/iovs.18-24106

[41]

Stacey D. Scott, M. Sheelagh T. Carpendale, and Kori M. Inkpen. 2004. Territoriality in collaborative tabletop workspaces. In Proceedings of the 2004 ACM conference on Computer supported cooperative work. ACM, New York, NY, USA. 294–303. https://doi.org/10.1145/1031607.1031655

Digital Library

[42]

Anthony Tang, Melanie Tory, Barry Po, Petra Neumann, and Sheelagh Carpendale. 2006. Collaborative coupling over tabletop displays. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, New York, NY, USA. 1181–1190. https://doi.org/10.1145/1124772.1124950

Digital Library

[43]

John C Tang. 1991. Findings from observational studies of collaborative work. International Journal of Man-machine studies, 34, 2 (1991), 143–160. https://doi.org/10.1016/0020-7373(91)90039-A

Digital Library

[44]

Thomas Daniel Ullmann. 2019. Automated analysis of reflection in writing: Validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29, 2 (2019), 217–257. https://doi.org/10.1007/s40593-019-00174-2

[45]

J. Wayne Wrightstone. 1934. An Instrument for Measuring Group Discussion and Planning. The Journal of Educational Research, 27, 9 (1934), May, 641–650. issn:0022-0671 https://doi.org/10.1080/00220671.1934.10880446 Publisher: Routledge _eprint: https://doi.org/10.1080/00220671.1934.10880446

[46]

Bin Xu, Jason Ellis, and Thomas Erickson. 2017. Attention from Afar: Simulating the Gazes of Remote Participants in Hybrid Meetings. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS ’17). ACM, New York, NY, USA. 101–113. isbn:978-1-4503-4922-2 https://doi.org/10.1145/3064663.3064720 event-place: Edinburgh, United Kingdom

Digital Library

Cited By

Holm-Janas VCaro Miya Marshall OLi ZBruun JSpikol D(2024)Design Framework for Multimodal Learning Analytics Leveraging Human ObservationsTechnology Enhanced Learning for Inclusive and Equitable Quality Education10.1007/978-3-031-72312-4_13(106-112)Online publication date: 13-Sep-2024
https://doi.org/10.1007/978-3-031-72312-4_13

Index Terms

Semi-automated Analysis of Collaborative Interaction: Are We There Yet?
1. Computing methodologies
  1. Artificial intelligence
2. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
  2. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
    2. Interaction devices

Recommendations

Towards an Automated Model to Evaluate Collaboration through Non-Verbal Interaction in Collaborative Virtual Environments

Virtual environments represent a helpful resource for learning and training. In their multiuser modality, Collaborative Virtual Environments CVE support geographical distant people to experience collaborative learning and team training; a context in ...
Analyzing collaborative learning activities in wikis using social network analysis
CHI EA '09: CHI '09 Extended Abstracts on Human Factors in Computing Systems

This paper investigates the potential of existing methods for analyzing collaboration in wiki environments. After a short description of the motivation for this research paper a presentation of analysis methods in CSCL will be given, including a special ...
A model-based framework to automate the analysis of users' activity in collaborative systems

The automated analysis of users' activity in collaborative systems aims at characterizing the collective process established between the participants and contributing to improving the deficiencies that occur during the users' activity and that are ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue ISS

December 2022

746 pages

EISSN:2573-0142

DOI:10.1145/3554337

Editor:
Jeffrey Nichols
Apple, USA

Issue’s Table of Contents

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2022

Published in PACMHCI Volume 6, Issue ISS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Austrian Science Fund FWF

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
463
Total Downloads

Downloads (Last 12 months)173
Downloads (Last 6 weeks)19

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Holm-Janas VCaro Miya Marshall OLi ZBruun JSpikol D(2024)Design Framework for Multimodal Learning Analytics Leveraging Human ObservationsTechnology Enhanced Learning for Inclusive and Equitable Quality Education10.1007/978-3-031-72312-4_13(106-112)Online publication date: 13-Sep-2024
https://doi.org/10.1007/978-3-031-72312-4_13

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents