research-article

Public Access

An Analysis of Speech as a Modality for Activity Recognition during Complex Medical Teamwork

Authors:

Swathi Jagannath,

Aleksandra Sarcevic,

Ivan MarsicAuthors Info & Claims

PervasiveHealth '18: Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare

Pages 88 - 97

https://doi.org/10.1145/3240925.3240941

Published: 21 May 2018 Publication History

Abstract

We analyzed the nature of verbal communication among team members in a dynamic medical setting of trauma resuscitation to inform the design of a speech-based automatic activity recognition system. Using speech transcripts from 20 resuscitations, we identified common keywords and speech patterns for different resuscitation activities. Based on these patterns, we developed narrative schemas (speech "workflow" models) for five most frequently performed activities and applied linguistic models to represent relationships between sentences. We evaluated the narrative schemas with 17 new cases, finding that all five schemas adequately represented speech during activities and could serve as a basis for speech-based activity recognition. We also identified similarities between narrative schemas of different activities. We conclude with design implications and challenges associated with speech-based activity recognition in complex medical processes.

References

[1]

American College of Surgeons, Advanced Trauma Life Support® (ATLS®), 7th Edition, Chicago, IL, 2005.

[2]

Engelbert A.G. Bergs, Frans L.P.A Rutten, Tamer Tadros, Pieta Krijnen, and Inger B. Schipper. 2005. Communication during trauma resuscitation: do we know what is happening? Injury 36, 8 (Aug. 2005), 905--911.

[3]

Elizabeth A. Carter, Lauren J. Waterhouse, Mark L. Kovler, Jennifer Fritzeen, and Randall S. Burd. 2013. Adherence to ATLS primary and secondary surveys during pediatric trauma resuscitation. Resuscitation 84, 1 (Jan. 2013), 66--71.

[4]

Nathanael Chambers and Daniel Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proc. Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 602--610.

Digital Library

[5]

John R. Clarke, Bonnie L. Webber, Abigail Gertner, Jonathan Kaye, and Ron Rymon. 1994. On-line decision support for emergency trauma management. In Proc. Annual Symposium on Computer Application in Medical Care, American Medical Informatics Association, 1028.

[6]

John R. Clarke, Stanley Z. Trooskin, Prashant J. Doshi, Lloyd Greenwald, and Charles J. Mode. 2002. Time to laparotomy for intra-abdominal bleeding from trauma does affect survival for delays up to 90 minutes. J. Trauma and Acute Care Surgery 52, 3 (Mar. 2002), 420--425.

[7]

Demetrios Demetriades, Brian Kimbrell, Ali Salim, George Velmahos, Peter Rhee, Christy Preston, Ginger Gruzinski, and Linda Chan. 2005. Trauma deaths in a mature urban trauma system: is "trimodal" distribution a valid concept? J. Amer College of Surgeons 201, 3 (Sep. 2005), 343--348.

[8]

Mark Fitzgerald, Peter Cameron, Colin Mackenzie, Nathan Farrow, Pamela Scicluna, Robert Gocentas, Adam Bystrzycki, Geraldine Lee, Gerard O'Reilly, Nick Andrianopoulos, Linas Dziukas, Jamie D. Cooper, Andrew Silvers, Alfredo Mori, Angela Murray, Susan Smith, Yan Xiao, Frank T. McDermott, Jeffrey V. Rosenfeld. 2011. Trauma resuscitation errors and computer-assisted decision support. Archives of Surgery 146, 2 (Feb. 2011), 218--225.

[9]

Germain Forestier, Florent Lalys, Laurent Riffaud, Brivael Trelhu, and Pierre Jannin. 2012. Classification of surgical processes using dynamic time warping. Journal of Biomedical Informatics 45, 2 (Apr. 2012), 255--264.

Digital Library

[10]

Lea Frermann, Ivan Titov, and Manfred Pinkal. 2014. A hierarchical Bayesian model for unsupervised induction of script knowledge. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL-14), 49--57.

[11]

Theodoros Giannakopoulos and Georgios Siantikos. 2016. A ROS framework for audio-based activity recognition. In Proc. 9th ACM Int'l Conf. Pervasive Technologies Related to Assistive Environments. ACM Press, New York, 41.

Digital Library

[12]

Russell L. Gruen, Gregory J. Jurkovich, Lisa K. McIntyre, Hugh M. Foy, and Ronald V. Maier. 2006. Patterns of errors contributing to trauma mortality: lessons learned from 2594 deaths. Annals of Surgery 244, 3 (Sep. 2006), 371--380.

[13]

Yue Gu, Xinyu Li, Shuhong Chen, Jianyu Zhang, and Ivan Marsic. 2017. Speech Intention Classification with Multimodal Deep Learning. Mouhoub M., Langlais P. (eds) Advances in Artificial Intelligence. AI 2017. Lecture Notes in Computer Science, Springer, Cham, vol 10233, 260--271.

[14]

Yue Gu, Xinyu Li, Shuhong Chen, Hunagcan Li, Richard A. Farneth, Ivan Marsic, Randall S. Burd. 2017. Language-based process phase detection in trauma resuscitation. In Healthcare Informatics (ICHI), 2017 IEEE International Conference on, IEEE, 239--247.

[15]

Lars Hertel, Huy Phan, and Alfred Mertins. 2015. Comparing time and frequency domain for audio event recognition using deep learning. In Neural Networks (IJCNN), 2016 International Joint Conference on, IEEE, 3407--3411.

[16]

Xinyu Li, Dongyang Yao, Xuechao Pan, Jonathan Johannaman, JaeWon Yang, Rachel Webman, Aleksandra Sarcevic, Ivan Marsic, and Randall S. Burd. 2016. Activity recognition for medical teamwork based on passive RFID. In RFID (RFID), 2016 IEEE International Conference on, IEEE, 1--9.

[17]

Xinyu Li, Yanyi Zhang, Jianyu Zhang, Moliang Zhou, Shuhong Chen, Yue Gu, Yueyang Chen, Ivan Marsic, Richard A. Farneth, and Randall S. Burd. 2017. Progress Estimation and Phase Detection for Sequential Processes. Proc. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (Sep. 2017), 73.

Digital Library

[18]

Ashutosh Modi, Ivan Titov, Vera Demberg, Asad Sayeed, and Manfred Pinkal. 2017. Modeling Semantic Expectation: Using Script Knowledge for Referent Prediction. Retrieved from: arXiv:1702.03121

[19]

Thomas Neumuth, and Christian Meißner. 2012. Online recognition of surgical instruments by information fusion. International journal of computer assisted radiology and surgery 7, 2 (Mar. 2012), 297--304.

[20]

John Walker Orr, Prasad Tadepalli, Janardhan Rao Doppa, Xiaoli Fern, and Thomas G. Dietterich. 2014. Learning Scripts as Hidden Markov Models. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI-14). 1565--1571.

Digital Library

[21]

Karl Pichotta and Raymond J. Mooney. 2016. Learning Statistical Scripts with LSTM Recurrent Neural Networks. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16). 2800--2806.

Digital Library

[22]

Karl Pichotta and Raymond J. Mooney. 2016. Using sentence-level LSTM language models for script inference. In Proc. 54th Annual Meeting of the Assoc. Computational Linguistics (ACL 2016). 279--289.

[23]

Michaela Regneri, Alexander Koller, and Manfred Pinkal. 2010. Learning script knowledge with web experiments. In Proc. 48th Annual Meeting of the Assoc. for Computational Linguistics (ACL-10). 979--988.

Digital Library

[24]

Nicole K. Roberts, Reed G. Williams, Cathy J. Schwind, John A. Sutyak, Christopher McDowell, David Griffen, Jarrod Wall, Hilary Sanfey, Audra Chestnut, Andreas H. Meier, Christopher Wohltmann, Ted R. Clark, Nathan Wetter. 2014. The impact of brief team communication, leadership and team behavior training on ad hoc team performance in trauma care settings. The American Journal of Surgery 207, 2 (Feb. 2014), 170--178.

[25]

Aleksandra Sarcevic, Ivan Marsic, Michael E. Lesk, and Randall S. Burd. 2008. Transactive memory in trauma resuscitation. In Proc. 2008 ACM Conference on Computer Supported Cooperative Work. ACM, New York, NY, 215--224.

Digital Library

[26]

Richard Socher, John Bauer, and Christopher D. Manning. 2013. Parsing with compositional vector grammars. In Proc. 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013). vol. 1, 455- 465.

[27]

Johannes A. Stork, Luciano Spinello, Jens Silva, and Kai O. Arras. 2012. Audio-based human activity recognition using non-Markovian ensemble voting. In RO-MAN 2012 IEEE, IEEE, 509--514.

[28]

Homer C.N. Tien, Vincent Jung, Ruxandra Pinto, Todd Mainprize, Damon C. Scales, and Sandro B. Rizoli. 2011. Reducing time-to-treatment decreases mortality of trauma patients with acute subdural hematoma. Annals of surgery 253, 6 (Jun. 2011), 1178--1183.

[29]

Achyut Mani Tripathi, Diganta Baruah, and Rashmi Dutta Baruah. 2015. Acoustic sensor based activity recognition using ensemble of one-class classifiers. In Evolving and Adaptive Intelligent Systems (EAIS), 2015 IEEE International Conference on, IEEE, 1--7.

[30]

Mithra Vankipuram, Kanav Kahol, Trevor Cohen, and Vimla L. Patel. 2011. Toward automated workflow analysis and visualization in clinical environments. Journal of Biomedical Informatics 44, 3 (Jun. 2011), 432--440.

Digital Library

[31]

Terry Winograd, and Fernando Flores. 1986. Understanding Computers and Cognition. Ablex Publishing Corp, Norwood, NJ.

Digital Library

[32]

Zhan Zhang and Aleksandra Sarcevic. 2015. Constructing awareness through speech, gesture, gaze and movement during a time-critical medical task. In Proceedings of the European Conference on Computer-Supported Cooperative Work (ECSCW '15). Springer, Cham, 163--182.

Cited By

Gao CMarsic ISarcevic AGestrich-Thompson WBurd R(2023)Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in Dynamic Clinical SettingsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35807987:1(1-28)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580798
Piliuk KTomforde S(2023)Artificial intelligence in emergency medicine. A systematic literature reviewInternational Journal of Medical Informatics10.1016/j.ijmedinf.2023.105274180(105274)Online publication date: Dec-2023
https://doi.org/10.1016/j.ijmedinf.2023.105274
Keyvanpour MMehrmolaei SShojaeddini SEsmaeili F(2023)HAR-CO: A comparative analytical review for recognizing conventional human activity in stream data relying on challenges and approachesMultimedia Tools and Applications10.1007/s11042-023-16795-883:14(40811-40856)Online publication date: 10-Oct-2023
https://doi.org/10.1007/s11042-023-16795-8
Show More Cited By

Index Terms

An Analysis of Speech as a Modality for Activity Recognition during Complex Medical Teamwork
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Activity recognition and understanding
2. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. Activity centered design

Recommendations

A Speech-Based Model for Tracking the Progression of Activities in Extreme Action Teamwork
CSCW1

Designing computerized approaches to support complex teamwork requires an understanding of how activity-related information is relayed among team members. In this paper, we focus on verbal communication and describe a speech-based model that we ...
Assessing the Feasibility of Speech-Based Activity Recognition in Dynamic Medical Settings
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

We describe an experiment conducted with three domain experts to understand how well they can recognize types and performance stages of activities using speech data transcribed from verbal communications during dynamic medical teamwork. The insights ...
Formant measurement in children's speech based on spectral filtering

Children's speech presents a challenging problem for formant frequency measurement.A spectral filtering technique is proposed for analysis of children's speech.The method was tested with child-like simulated speech samples.The new approach produced less ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

PervasiveHealth '18: Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare

May 2018

413 pages

ISBN:9781450364508

DOI:10.1145/3240925

General Chairs:
Noga Minsky
Mount Sinai Health System
,
Venet Osmani
Fondazione Bruno Kessler

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

EAI: The European Alliance for Innovation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

PervasiveHealth '18

PervasiveHealth '18: 12th EAI International Conference on Pervasive Computing Technologies for Healthcare

May 21 - 24, 2018

NY, New York, USA

Acceptance Rates

Overall Acceptance Rate 55 of 116 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
238
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)6

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao CMarsic ISarcevic AGestrich-Thompson WBurd R(2023)Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in Dynamic Clinical SettingsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35807987:1(1-28)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580798
Piliuk KTomforde S(2023)Artificial intelligence in emergency medicine. A systematic literature reviewInternational Journal of Medical Informatics10.1016/j.ijmedinf.2023.105274180(105274)Online publication date: Dec-2023
https://doi.org/10.1016/j.ijmedinf.2023.105274
Keyvanpour MMehrmolaei SShojaeddini SEsmaeili F(2023)HAR-CO: A comparative analytical review for recognizing conventional human activity in stream data relying on challenges and approachesMultimedia Tools and Applications10.1007/s11042-023-16795-883:14(40811-40856)Online publication date: 10-Oct-2023
https://doi.org/10.1007/s11042-023-16795-8
Jagannath SKamireddi NZellner KBurd RMarsic ISarcevic A(2022)A Speech-Based Model for Tracking the Progression of Activities in Extreme Action TeamworkProceedings of the ACM on Human-Computer Interaction10.1145/35129206:CSCW1(1-26)Online publication date: 7-Apr-2022
https://dl.acm.org/doi/10.1145/3512920
Ann Zellner KCoates MLee AJagannath SSarcevic AC. Alberto EHarvey AS. Burd RMarsic I(2021)Characterizing Speech in Life Saving Interventions to Inform Computerized Clinical Decision Support for Complex Medical TeamworkCompanion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3462204.3481746(199-202)Online publication date: 23-Oct-2021
https://dl.acm.org/doi/10.1145/3462204.3481746
Chen Y(2020)Learning Health-Care Worker Networks from Electronic Health Record UtilizationTeamwork in Healthcare [Working Title]10.5772/intechopen.93703Online publication date: 25-Sep-2020
https://doi.org/10.5772/intechopen.93703
Jagannath SSarcevic AKamireddi NMarsic IBrewster SFitzpatrick GCox AKostakos V(2019)Assessing the Feasibility of Speech-Based Activity Recognition in Dynamic Medical SettingsExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290607.3312983(1-6)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290607.3312983

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents