Work in Progress

An Interactive Garment for Orchestra Conducting: IoT-enabled Textile & Machine Learning to Direct Musical Performance

Authors:

Pauline Vierne,

Paul Biessmann,

Alexandra Börner,

Kaspar Schleiser,

Emmanuel Baccelli,

Christopher Verworner,

Felix BiessmannAuthors Info & Claims

TEI '21: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

Article No.: 46, Pages 1 - 6

https://doi.org/10.1145/3430524.3442451

Published: 14 February 2021 Publication History

Abstract

We present an overview and initial results from a project bringing together orchestra conducting, e-textile material studies, costume tailoring, low power computing and machine learning (ML). We describe a wearable interactive system comprising of textile sensors embedded into a suit, low-power transmission and gesture recognition using creative computing tools. We introduce first observations made during the semi-participatory approach, which placed the conductor’s movements and personal performative expressiveness at the centre for technical and conceptual development. The project is a two-month collaboration between the Verworner-Krause Kammerorchester (VKKO), technical and design researchers, currently still running. Preliminary analyses of the data recorded while the conductor is wearing the prototype demonstrate that the developed system can be used to robustly decode a large number of conducting and performative movements. In particular the user interface of the ML system is designed such that the training of the algorithms can be intuitively controlled by the conductor, in sync with the MIDI clock.

References

[1]

Emmanuel Baccelli. 2018. RIOT: An open source operating system for low-end embedded devices in the IoT. IEEE Internet of Things Journal 5, 6 (2018), 4428 – 4440.

[2]

Rebecca Fiebrink. 2009. Wekinator.

[3]

Rebecca Fiebrink and Laetitia Sonami. 2020. Reflections on Eight Years of Instrument Creation with Machine Learning. In International Conference on New Interfaces for Musical Expression.

[4]

Adrian Freed. 2017. opensoundcontrol.org : An Enabling Encoding for Media Applications. http://opensoundcontrol.org

[5]

Berit Greinke. 2011. Twiddletone. Technical Report. Queen Mary University of London. http://beritgreinke.net/research/research-placement-culture-lab/

[6]

Oliver Hahm and others. 2015. Operating systems for low-end devices in the Internet of Things: a survey. IEEE Internet of Things Journal 3, 5 (2015), 720 – 734.

[7]

Cedric Honnet, Hannah Perner-Wilson, Marc Teyssier, Bruno Fruchard, Jürgen Steimle, Ana C. Baptista, and Paul Strohmeier. 2020. PolySense: Augmenting Textiles with Electrical Functionality using In-Situ Polymerization. In CHI ’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, 1 – 13. https://doi.org/10.1145/3313831.3376841

Digital Library

[8]

Shelly Knotts, Jack Armitage, and Alex McLean. 2020. Heritage algorithms. https://hybrid-livecode.pubpub.org/workshop2020

[9]

Tellef Kvifte. 2016. Musical Instruments and User Interfaces in Two Centuries. In Material Culture and Electronic Sound. Smithsonian Institution Scholarly Press, Washington, Chapter 8, 203–229.

[10]

Louis McCallum and Mick S Grierson. 2020. Supporting Interactive Machine Learning Approaches to Building Musical Instruments in the Browser. In Proceedings of the International Conference on New Interfaces for Musical Expression. Birmingham City University, Birmingham, UK, 322–324.

[11]

Alex McLean, Giovanni Fanfani, and Ellen Harlizius-Klück. 2018. Cyclic Patterns of Movement across Weaving, Epiplokē and Live Coding. Dancecult: Journal of Electronic Dance Music Culture 10, 1 (2018), 5–30.

[12]

Parag Mital. 2017. Generate your own sounds with NSynth. https://magenta.tensorflow.org/nsynth-fastgen

[13]

Roberto Morabito and Jaime Jiménez. 2020. IETF protocol suite for the Internet of Things: Overview and Recent Advancements.

[14]

Charlotte Nordmoen, Jack Armitage, Fabio Morreale, Rebecca Stewart, and Andrew McPherson. 2019. Making Sense of Sensors : Discovery Through Craft Practice With an Open-Ended Sensor Material. In Proceedings of the 2019 on Designing Interactive Systems Conference. Association for Computing Machinery, New York, 135–146. https://doi.org/10.1145/3322276.3322368

Digital Library

[15]

Hannah Perner-Wilson and Mika Satomi. 2018. Trombone Breathing Vest. https://www.kobakant.at/KOBA/trombone-breathing/

[16]

Afroditi Psarra. 2014. Lilytronica. http://afroditipsarra.com/index.php?/on-going/lily/

[17]

Sophie Skach, Anna Xambó, Luca Turchet, Ariane Stolfi, Rebecca Stewart, and Mathieu Barthet. 2018. Embodied Interactions with E-Textiles and the Internet of Sounds for Performing Arts. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction - TEI ’18. Association for Computing Machinery, New York, 80–87. https://doi.org/10.1145/3173225.3173272

Digital Library

[18]

Paola Torres. 2017. The Augmented Soma Suit : Wearables for Enhanced Somatic Experiences. Technical Report. Sodertorn Hogskola.

[19]

Hannes Tschofenig and Emmanuel Baccelli. 2019. Cyberphysical Security for the Masses: A Survey of the Internet Protocol Suite for Internet of Things Security. IEEE Security and Privacy 17, 5 (9 2019), 47–57. https://doi.org/10.1109/MSEC.2019.2923973

Cited By

Visi FBasso TGreinke BWood EGschwendtner PHope CÖstersjö S(2024)Networking concert halls, musicians, and interactive textiles: Interwoven Sound SpacesDigital Creativity10.1080/14626268.2024.231190635:1(52-73)Online publication date: 20-Feb-2024
https://doi.org/10.1080/14626268.2024.2311906
Wu ZGao ZBraud T(2023)Unlogical instrument: Material-driven gesture-controlled sound installationCompanion Publication of the 2023 ACM Designing Interactive Systems Conference10.1145/3563703.3596643(86-89)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563703.3596643
Inayatulloh Hartono ICahya S P(2023)Conceptual Model of Citizen Science with Machine Learning to Increase The Effectiveness of Land Transportation of Urban Communities2023 International Conference on Informatics, Multimedia, Cyber and Informations System (ICIMCIS)10.1109/ICIMCIS60089.2023.10348969(458-461)Online publication date: 7-Nov-2023
https://doi.org/10.1109/ICIMCIS60089.2023.10348969
Show More Cited By

Index Terms

An Interactive Garment for Orchestra Conducting: IoT-enabled Textile & Machine Learning to Direct Musical Performance
1. Applied computing
  1. Arts and humanities
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Synchronization and fluctuation of rhythm in musical cooperative performance
HCII'11: Proceedings of the 14th international conference on Human-computer interaction: design and development approaches - Volume Part I

A live musical performance gives us better impression than recorded music heard from a portable music player. From player's point of view, live performance also gives better impression than playing music with metronome or recorded music. Thus, the ...
Personal orchestra: conducting audio/video music recordings
WEDELMUSIC'02: Proceedings of the Second international conference on Web delivering of music

Personal Orchestra lets anybody conduct an electronic orchestra at a new level of realism: users interact not with a synthetic, but an original audio and video recording of a real orchestra--the Vienna Philharmonic. Nevertheless, they can interactively ...
PopMash: an automatic musical-mashup system using computation of musical and lyrical agreement for transitions
Abstract
Musical-mashup is a popular form of music re-creation, aiming at combining multiple pieces of music to create new music artworks. Presently, it is also a challenge in the field of music information study. In this work, an effective framework for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

TEI '21: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 2021

908 pages

ISBN:9781450382137

DOI:10.1145/3430524

Editor:
Raphael Wimmer
University of Regensburg, Germany
,
General Chairs:
Martin Kaltenbrunner
University of Art and Design Linz, Austria
,
Martin Murer
University of Salzburg, Austria
,
Program Chairs:
Katrin Wolf
Beuth University of Applied Sciences, Berlin, Germany
,
Ian Oakley
UNIST, Ulsan, Republic of Korea

Copyright © 2021 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 February 2021

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Conference

TEI '21

TEI '21: Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 14 - 17, 2021

Salzburg, Austria

Acceptance Rates

TEI '21 Paper Acceptance Rate 40 of 136 submissions, 29%;

Overall Acceptance Rate 393 of 1,367 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
226
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)5

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Visi FBasso TGreinke BWood EGschwendtner PHope CÖstersjö S(2024)Networking concert halls, musicians, and interactive textiles: Interwoven Sound SpacesDigital Creativity10.1080/14626268.2024.231190635:1(52-73)Online publication date: 20-Feb-2024
https://doi.org/10.1080/14626268.2024.2311906
Wu ZGao ZBraud T(2023)Unlogical instrument: Material-driven gesture-controlled sound installationCompanion Publication of the 2023 ACM Designing Interactive Systems Conference10.1145/3563703.3596643(86-89)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563703.3596643
Inayatulloh Hartono ICahya S P(2023)Conceptual Model of Citizen Science with Machine Learning to Increase The Effectiveness of Land Transportation of Urban Communities2023 International Conference on Informatics, Multimedia, Cyber and Informations System (ICIMCIS)10.1109/ICIMCIS60089.2023.10348969(458-461)Online publication date: 7-Nov-2023
https://doi.org/10.1109/ICIMCIS60089.2023.10348969
Johansen Svan Berkel NFritsch J(2022)Characterising Soundscape Research in Human-Computer InteractionProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533458(1394-1417)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533458
Reed CSkach SStrohmeier PMcPherson A(2022)Singing Knit: Soft Knit Biosensing for Augmenting Vocal PerformancesProceedings of the Augmented Humans International Conference 202210.1145/3519391.3519412(170-183)Online publication date: 13-Mar-2022
https://dl.acm.org/doi/10.1145/3519391.3519412

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents