article

Realtime audiovisual rendering and contemporary audiovisual art

Authors:

Jarmo Hiipakka,

Rami Hänninen,

Tapio TakalaAuthors Info & Claims

Organised Sound, Volume 3, Issue 3

Pages 219 - 233

https://doi.org/10.1017/S1355771898003069

Published: 01 December 1998 Publication History

Abstract

Visual rendering is the process of creating synthetic images of digital models. The modelling of sound synthesis and propagation in a virtual space is called sound rendering. In this article we review different audiovisual rendering techniques suitable for realtime rendering of three-dimensional virtual worlds. Virtual environments are useful in various application areas, for example in architectural visualisation. With audiovisual rendering, lighting and acoustics of a modelled concert hall can be experienced early in the design stage of the building. In this article we demonstrate an interactive audiovisual rendering system where an animated virtual orchestra plays in a modelled concert hall. Virtual musicians are conducted by a real conductor who wears a wired data dress suit and a baton. The conductor and the audience hear the music rendered according to the acoustics of the virtual concert hall, creating a lifelike experience.

References

[1]

Allen, J. B., and Berkley, D. A. 1979. Image method for efficiently simulating small-room acoustics. Journal of the Acoustical Society of America 65(4): 943-50.

[2]

Astheimer, P. 1995. Acoustic simulation for visualization and virtual reality. In R. Veltkamp (ed.) Eurographics State-of-the-Art Reports, pp. 1-23. Maastricht, Netherlands.

[3]

Begault, D. 1994. 3-D Sound for Virtual Reality and Multi-media . Cambridge, MA: Academic Press.

[4]

Borish, J. 1984. Extension of the image model to arbitrary polyhedra. Journal of the Acoustical Society of America 75(6): 1, 827-36.

[5]

Cook, P. R. 1991. Tbone: an interactive waveguide brass instruments synthesis workbench for the NeXT machine. Proc. Int. Computer Music Conf., pp. 297-9. Montreal, Canada.

[6]

Ellis, S. R., Breant, F., Menges, B., Jacoby, R., and Adelstein, B. D. 1997. Factors influencing operator interaction with virtual objects viewed via head-mounted see-through displays: viewing conditions and rendering latency. Proc. Virtual Reality Annual Internation Symp. (VRAIS'97), pp. 138-45.

[7]

Foley, J., van Dam, A., Feiner, S., and Hughes, J. 1990. Computer Graphics, Principles and Practice, 2nd edn. Reading, MA: Addison-Wesley.

Digital Library

[8]

Gardner, W. G. 1997. 3-D Audio Using Loudspeakers, PhD Thesis, MIT Media Lab.

[9]

Gardner, W. G. 1998. Reverberation algorithms. In M. Kahrs and K. Brandenburg (eds.) Applications of Digital Signal Processing to Audio and Acoustics, Chap. 3, pp. 85-131. Boston, MA: Kluwer Academic Publishers.

[10]

Glassner, A. 1995. Principles of Digital Image Synthesis, Vol. 2. San Francisco: Morgan Kaufmann.

[11]

Granier, E., Kleiner, M., Dalenbäck, B.-I., and Svensson, P. 1996. Experimental auralization of car audio installations. Journal of the Audio Engineering Society 44(10): 835-49.

[12]

Hiipakka, J., Hänninen, R., Ilmonen, T., Napari, H., Lokki, T., Savioja, L., Huopaniemi, H., Karjalainen, M., Tolonen, T., Välimäki, V., Välimäki, S., and Takala, T. 1997. Virtual orchestra performance. Visual Proceedings of SIGGRAPH'97, p. 81. ACM SIGGRAPH, Los Angeles.

[13]

Huopaniemi, J., Zacharov, N., and Karjalainen, M. 1999. Objective and subjective evaluation of head-related transfer function filter design. Journal of the Audio Engineering Society 47(4): 218-39.

[14]

Huopaniemi, J., Savioja, L., and Karjalainen, M. 1997. Modeling of reflections and air absorption in acoustical spaces - a digital filter design approach. Proc. of IEEE 1997 Workshop on Applications of Signal Processing to Audio and Acoustics. Mohonk, New Paltz, New York.

[15]

Ilmonen, T. 1999. Tracking Conductor of an Orchestra using Artificial Neural Networks. Master's Thesis, Telecommunications Software and Multimedia Laboratory, Helsinki University of Technology.

[16]

Jaffe, D., and Smith, J. O. 1983. Extensions of the Karplus-Strong plucked string algorithm. Computer Music Journal 7(2): 56-69. Reprinted in C. Roads (ed.) The Music Machine, pp. 481-94. Cambridge, MA: MIT Press, 1989.

[17]

Jaffe, D., and Smith, J. O. 1995. Performance expression in commuted waveguide synthesis of bowed strings. Proc. Int. Computer Music Conf., pp. 343-6.

[18]

Java 3D. 1998. SUN, Inc. JAVA 3D API Specification 1.1. URL: http://www.javasoft.com/products/java-media/ 3D/

[19]

JTC1/SC29/WG11. 1999. ISO/IEC JTC1/SC29/WG11 IS 14496 (MPEG-4).

[20]

JTC/SC24. 1997. ISO/IEC JTC/SC24 IS 14772-1. Information Technology - Computer Graphics and Image Processing - The Virtual Reality Modeling Language (VRML97). April 1997. URL: http://www.vrml.org/ Specifications/VRML97/

[21]

Karjalainen, M., Laine, U. K., Laakso, T. I., and Välimäki, V. 1991. Transmission-line modeling and real-time synthesis of string and wind instruments. Proc. Int. Computer Music Conf., pp. 293-6. Montreal, Quebec, Canada.

[22]

Karjalainen, M., and Välimäki, V. 1993. Model-based analysis/synthesis of the acoustic guitar. Proc. Stockholm Music Acoustics Conf. (SMAC'93), pp. 443-7. Stockholm, Sweden, published in 1994. Sound examples included on the SMAC'93 CD.

[23]

Karjalainen, M., Välimäki, V., and Tolonen, T. 1998. Plucked string models - from Karplus-Strong algorithm to digital waveguides and beyond. Computer Music Journal 22(3): 17-23.

[24]

Karplus, K., and Strong, A. 1983. Digital synthesis of plucked string and drum timbres. Computer Music Journal 7(2): 43-55. Reprinted in C. Roads (ed.) The Music Machine. Cambridge, MA: MIT Press, 1989.

[25]

Kleiner, M., Dalenbäck, B.-I., and Svensson, P. 1993. Auralization - an overview. Journal of the Audio Engineering Society 41(11): 861-75.

[26]

Krokstad, A., Strom, S., and Sorsdal, S. 1968. Calculating the acoustical room response by the use of a ray tracing technique. Journal of Sound and Vibration 8(1): 118-25.

[27]

Kuttruff, H. 1995. Sound field prediction in rooms. Proc. 15th Int. Congr. Acoust., Vol. 2, pp. 545-52. Trondheim, Norway.

[28]

Lytle, W. 1990. More bells and whistles {video}. SIGGRAPH'90 Electronic Theater (1990). Technically described in J. Hahn (ed.) Sound Synchronization and Synthesis for Computer Animation and Virtual Reality, Course Note 12 (SIGGRAPH'94 Conf., Orlando, FL, 1994). Also described in IEEE Computer 24(7): 4 and cover (July 1991).

[29]

Masataka, G., Hidaka, I., Matsumoto, H., Yosuke, K., and Yoichi, M. 1996. A jazz session system for interplay among all players. In Proc. of the Int. Computer Music Conf., pp. 346-9, Hong Kong.

[30]

McElheran, B. 1989. Conducting Technique for Beginners and Professionals. Oxford/New York: Oxford University Press.

[31]

Möller, H. 1992. Fundamentals of binaural technology. Applied Acoustics 36(3-4): 171-218.

[32]

Möller, H., Hammershöi, M. S. D., and Jensen, C. 1995. Head-related transfer functions of human subjects. Journal of the Audio Engineering Society 43(5): 300-21.

[33]

Perlin, K., and Goldberg, A. 1996. IMPROV: a system for scripting interactive actors in virtual worlds. Proc. of SIGGRAPH'96, pp. 205-16. ACM SIGGRAPH, New Orleans, LA.

[34]

Pulkki, V. 1997. Virtual sound source positioning using vector base amplitude panning. Journal of the Audio Engineering Society 45(6): 456-66.

[35]

Rojas, R. 1996. Neural Networks: A Systematic Introduction , p. 502. Berlin: Springer Verlag.

[36]

Savioja, L., Huopaniemi, J., Lokki, T., and Väänänen, R. 1999. Creating interactive virtual acoustic environments. Journal of the Audio Engineering Society 47(9): 675-705.

[37]

Savioja, L., Rinne, T., and Takala, T. 1994. Simulation of room acoustics with a 3-D finite difference mesh. Proc. Int. Computer Music Conf., pp. 463-6. Arhus, Denmark.

[38]

Scavone, G. P. 1996. Modeling and control of performance expression in digital waveguide models of woodwind instruments. Proc. Int. Computer Music Conf., pp. 224- 7. Hong Kong.

[39]

Smith, J. O. 1998. Principles of digital waveguide models of musical instruments. In M. Kahrs and K. Brandenburg (eds.) Applications of Digital Signal Processing to Audio and Acoustics, Chap. 10, pp. 417-66. Boston, MA: Kluwer Academic Publishers.

[40]

Smith, J. O. 1985. A new approach to digital reverberation using closed waveguide networks. Proc. Int. Computer Music Conf., pp. 47-53. Vancouver, BC, Canada.

[41]

Smith, J. O. 1986. Efficient simulation of the reed-bore and bow-string mechanisms. Proc. Int. Computer Music Conf., pp. 275-80. The Hague, The Netherlands.

[42]

Smith, J. O. 1987. Music applications of digital waveguides. Technical Report STAN-M-39. Department of Music, CCRMA, Stanford University, Stanford, CA.

[43]

Smith, J. O. 1993. Efficient synthesis of stringed musical instruments. Proc. Int. Computer Music Conf., pp. 64- 71. Tokyo, Japan.

[44]

Takala, T., and Hahn, J. 1992. Sound rendering. Computer Graphics 26(2): 211-20. Proc. SIGGRAPH'92.

Digital Library

[45]

Takala, T., Hänninen, R., Välimäki, V., Savioja, L., Huopaniemi, J., Huotilainen, T., and Karjalainen, M. 1996. An integrated system for virtual audio reality. 100th Audio Engineering Society (AES) Convention Preprint no. 4229. Copenhagen, Denmark.

[46]

Tolonen, T. 1998. Model-Based Analysis and Resynthesis of Acoustic Guitar Tones. Master's Thesis, Helsinki University of Technology, Laboratory of Acoustics and Audio Signal Processing, Espoo, Finland.

[47]

Väänänen, R., Välimäki, V., and Huopaniemi, J. 1997. Efficient and parametric reverberator for room acoustics modeling. Proc. Int. Computer Music Conf., pp. 200- 3. Thessaloniki, Greece.

[48]

Välimäki, V., Huopaniemi, J., Karjalainen, M., and Janosy, Z. 1996. Physical modeling of plucked string instruments with application to realtime sound synthesis. Journal of Audio Engineering Society 44(5): 331-53.

[49]

Välimäki, V., Karjalainen, M., Janosy, Z., and Laine, U. K. 1992. A real-time DSP implementation of a flute model. Proc. 1992 IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, Vol. 2, pp. 249-52. San Francisco, CA.

[50]

Välimäki, V., Karjalainen, M., and Laakso, T. I. 1993. Modeling of woodwind bores with finger holes. Proc. Int. Computer Music Conf., pp. 32-9. Tokyo, Japan.

[51]

Välimäki, V., and Takala, T. 1996. Virtual musical instruments - natural sound using physical models. Organised Sound 1(2): 75-86.

Digital Library

[52]

Välimäki, V., and Tolonen, T. 1998. Development and calibration of a guitar synthesizer. Journal of the Audio Engineering Society 46(9): 766-78.

Cited By

Kimura KHoshuyama ONarumi TTanikawa THirose MRomão TCorreia NInami MDias EChambel TKato HPrada RTerada T(2011)Sound-power visualization system for real-world interaction based on ultrasonic power transmissionProceedings of the 8th International Conference on Advances in Computer Entertainment Technology10.1145/2071423.2071499(1-8)Online publication date: 8-Nov-2011
https://dl.acm.org/doi/10.1145/2071423.2071499
Manocha DCalamia PLin MManocha DSavioja LTsingos N(2009)Interactive sound renderingACM SIGGRAPH 2009 Courses10.1145/1667239.1667254(1-338)Online publication date: 3-Aug-2009
https://dl.acm.org/doi/10.1145/1667239.1667254
Cádiz R(2006)A Fuzzy-Logic Mapper for Audiovisual MediaComputer Music Journal10.1162/01489260677602126330:1(67-82)Online publication date: 1-Mar-2006
https://dl.acm.org/doi/10.1162/014892606776021263
Show More Cited By

Realtime audiovisual rendering and contemporary audiovisual art
1. Applied computing
  1. Arts and humanities

Recommendations

Audiovisual Tool for Solfège Assessment

Solfège is a general technique used in the music learning process that involves the vocal performance of melodies, regarding the time and duration of musical sounds as specified in the music score, properly associated with the meter-mimicking performed ...
The Audiovisual Object
Audiovisual Playground: A Music Sequencing Tool for 3D Virtual Worlds
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

As electronic music has made its way into the mainstream, the popularity of music production tools for both professional and amateur use has grown. Although the users of music technology are becoming more prevalent and diverse, there have been very few ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Organised Sound

Organised Sound Volume 3, Issue 3

December 1998

54 pages

ISSN:1355-7718

Issue’s Table of Contents

Publisher

Cambridge University Press

United States

Publication History

Published: 01 December 1998

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kimura KHoshuyama ONarumi TTanikawa THirose MRomão TCorreia NInami MDias EChambel TKato HPrada RTerada T(2011)Sound-power visualization system for real-world interaction based on ultrasonic power transmissionProceedings of the 8th International Conference on Advances in Computer Entertainment Technology10.1145/2071423.2071499(1-8)Online publication date: 8-Nov-2011
https://dl.acm.org/doi/10.1145/2071423.2071499
Manocha DCalamia PLin MManocha DSavioja LTsingos N(2009)Interactive sound renderingACM SIGGRAPH 2009 Courses10.1145/1667239.1667254(1-338)Online publication date: 3-Aug-2009
https://dl.acm.org/doi/10.1145/1667239.1667254
Cádiz R(2006)A Fuzzy-Logic Mapper for Audiovisual MediaComputer Music Journal10.1162/01489260677602126330:1(67-82)Online publication date: 1-Mar-2006
https://dl.acm.org/doi/10.1162/014892606776021263
Lokki TSavioja LVäänänen RHuopaniemi JTakala T(2002)Creating Interactive Virtual Auditory EnvironmentsIEEE Computer Graphics and Applications10.1109/MCG.2002.101669822:4(49-57)Online publication date: 1-Jul-2002
https://dl.acm.org/doi/10.1109/MCG.2002.1016698
Kragtwijk MNijholt AZwiers J(2001)Implementation of a 3D virtual drummerProceedings of the Eurographic workshop on Computer animation and simulation10.5555/776350.776352(15-26)Online publication date: 1-Sep-2001
https://dl.acm.org/doi/10.5555/776350.776352

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents