Abstract
In this paper, we introduce Music in a Universal Sound Environment(MUSE), a system for gesture recognition in the domain of musical conducting. Our system captures conductors’ musical gestures to drive a MIDI-based music generation system allowing a human user to conduct a fully synthetic orchestra. Moreover, our system also aims to further improve a conductor’s technique in a fun and interactive environment. We describe how our system facilitates learning through a intuitive graphical interface, and describe how we utilized techniques from machine learning and Conga, a finite state machine, to process inputs from a low cost Leap Motion sensor in which estimates the beats patterns that a conductor is suggesting through interpreting hand motions. To explore other beat detection algorithms, we also include a machine learning module that utilizes Hidden Markov Models (HMM) in order to detect the beat patterns of a conductor. An additional experiment was also conducted for future expansion of the machine learning module with Recurrent Neural Networks (rnn) and the results prove to be better than a set of HMMs. MUSE allows users to control the tempo of a virtual orchestra through basic conducting patterns used by conductors in real time. Finally, we discuss a number of ways in which our system can be used for educational and professional purposes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Stinson, L. (2014). Conduct a virtual symphony with touchscreens and an interactive baton. [Online]. Available: http://www.wired.com/2014/05/interactive-baton/.
Toh, L.-W., Chao, W., & Chen, Y.-S. (2013). An interactive conducting system using kinect. In 2013 I.E. International Conference on Multimedia and Expo (ICME) (pp. 1–6).
Levin, G., Shakar, G., Gibbons, S., Sohrawardy, Y., Gruber, J., Semlak, E., Schmidl, G., Lehner, J., & Feinberg, J. (2001). Dialtones (a telesymphony). [Online]. Available: http://www.flong.com/projects/telesymphony/.
Api reference. [Online]. Available: https://developer.leapmotion.com/documentation/index.html.
Lee, E., Grüll, I., Kiel, H., & Borchers, J. (2006). Conga: A framework for adaptive conducting gesture analysis. In Proceedings of the 2006 Conference on New interfaces for Musical Expression (pp. 260–265). IRCAM – Centre Pompidou.
Smus, B. (2013). Gestural music direction. [Online]. Available: http://smus.com/gestural-music-direction/.
Schlömer, T., Poppinga, B., Henze, N., & Boll, S. (2008). Gesture recognition with a Wii controller. In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, ser. TEI’08 (pp. 11–14). New York: ACM. [Online]. Available: http://doi.acm.org/10.1145/1347390.1347395.
Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., & Moore, R. (2013). Real-time human pose recognition in parts from single depth images. Communication ACM, 56(1), 116–124. [Online]. Available: http://doi.acm.org/10.1145/2398356.2398381.
Schramm, R., Rosito Jung, C., & Reck Miranda, E. (2015). Dynamic time warping for music conducting gestures evaluation. IEEE Transactions on Multimedia, 17(2), 243–255.
Schramm, R., Jung, C. R., & Miranda, E. R. (2015). Dynamic time warping for music conducting gestures evaluation. IEEE Transactions on Multimedia, 17(2), 243–255.
Rabiner, L. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.
Qt – home. (2016). [Online]. Available: http://www.qt.io/.
Scavone, G. P. Introduction. [Online]. Available: https://www.music.mcgill.ca/~gary/rtmidi/.
Koftinoff, J. jdksmidi. [Online]. Available: https://github.com/jdkoftinoff/jdksmidi.
The industry’s foundation for high performance graphics. [Online]. Available: https://www.opengl.org/.
Roelofs, G., & Adler, M. (2013). zlib home site. [Online]. Available: http://www.zlib.net/.
Schmidt, D. (2007). Acceleration-based gesture recognition for conducting with hidden markov models. Ph.D. dissertation, Ludwig-Maximilians-Universitaet.
Chollet, F. (2016). keras. https://github.com/fchollet/keras.
Acknowledgements
This material is based in part upon work supported by: The National Science Foundation under grant number(s) IIA-1329469. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Carthen, C.D., Kelley, R., Ruggieri, C., Dascalu, S.M., Colby, J., Harris, F.C. (2018). MUSE: A Music Conducting Recognition System. In: Latifi, S. (eds) Information Technology - New Generations. Advances in Intelligent Systems and Computing, vol 558. Springer, Cham. https://doi.org/10.1007/978-3-319-54978-1_49
Download citation
DOI: https://doi.org/10.1007/978-3-319-54978-1_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54977-4
Online ISBN: 978-3-319-54978-1
eBook Packages: EngineeringEngineering (R0)