Abstract
Sound design has been a fundamental component of audiovisual storytelling in linear media. However, with recent technological developments and the shift towards non-linear and immersive media, things are rapidly changing. More sensory information is available and, at the same time, the user is gaining agency upon the narrative, being offered the possibility of navigating or making other decisions. These new characteristics of immersive environments bring new challenges to storytelling in interactive narratives and require new strategies and techniques for audiovisual narrative progression. Can technology offer an immersive environment where the user has the sensation of agency, of choice, where her actions are not mediated by evident controls but subliminally induced in a way that it is ensured that a narrative is being followed? Can sound be a subliminal element that induces attentional focus on the most relevant elements for the narrative, inducing storytelling and biasing search in an immersive non-linear audiovisual environment? Herein, we present a literature review that has been guided by this prospect. With these questions in view, we present our exploration process in finding possible answers and potential solution paths. We point out that consistency, in terms of coherency across sensory modalities and emotional matching may be a critical aspect. Finally, we consider that this review may open up new paths for experimental studies that could, in the future, provide new strategies in the practice of sound design in the context of non-linear media.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Adams E (2014) Fundamentals of game design, 3rd edn. Pearson Education, New Riders
Albert G (2012) Immersion as category of audiovisual experience: from long beach to hollywood. Worlds of Audiovision
Alves V, Roque L (2009) A proposal of soundscape design guidelines for user experience enrichment. Audio Mostly 2009:27–32
Andersen RA, Snyder LH, Bradley DC, Xing J (1997) Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annu Rev Neurosci 20(1):303–330
Arsenault D (2005) Dark waters: spotlight on immersion. In: Game-On North America 2005 conference: eurosis. Elsevier, Ghent, pp 50–52
Asutay E, Västfjäll D, Tajadura-Jimenez A, Genell A, Bergman P, Kleiner M (2012) Emoacoustics: a study of the psychoacoustical and psychological dimensions of emotional sound design. J Audio Eng Soc 60(1/2):21–28
Bach DR, Neuhoff JG, Perrig W, Seifritz E (2009) Looming sounds as warning signals: the function of motion cues. Int J Psychophysiol 74(1):28–33
Begault DR (1992) Perceptual effects of synthetic reverberation on three-dimensional audio systems. J Audio Eng Soc 40(11):895–904
Begault DR, Trejo LJ (2000) 3-d sound for virtual reality and multimedia. AP Professional
Blood AJ, Zatorre RJ (2001) Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proc Natl Acad Sci 98(20):11818–11823
Bradley MM (2008) Natural selective attention: orienting and emotion. Psychophysiology 46:1–11
Bregman AS (1990) Auditory scene analysis: the perceptual organization of sound. MIT Press
Brown E, Cairns P (2004) A grounded investigation of game immersion. In: CHI’04 extended abstracts on Human factors in computing systems. ACM, pp 1297–1300
Van der Burg E, Olivers CN, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34(5):1053
Callejas A, Lupiáñez J, Tudela P (2004) The three attentional networks: on their independence and interactions. Brain Cogn 54(3):225–227
Callejas A, Lupianez J, Funes MJ, Tudela P (2005) Modulations among the alerting, orienting and executive control networks. Exp Brain Res 167(1):27–37
Chen L, Vroomen J (2013) Intersensory binding across space and time: a tutorial review. Attention, Perception, & Psychophysics 75(5):790–811
Chion M (1994) Audio-vision: sound on screen. Columbia University Press
Cho J, Yi E, Cho G (2001) Physiological responses evoked by fabric sounds and related mechanical and acoustical properties. Text Res J 71(12):1068–1073
Chueng P, Marsden P (2003) Designing auditory spaces: the role of expectation. In: Proceedings of 10th international conference on human computer interaction. Citeseer, pp 616–620
Chun MM, Golomb JD, Turk-Browne NB (2011) A taxonomy of external and internal attention. Annu Rev Psychol 62:73–101
Colby CL, Goldberg ME (1999) Space and attention in parietal cortex. Annu Rev Neurosci 22(1):319–349
Collins K (2011) Making gamers cry: mirror neurons and embodied interaction with game sound. In: Proceedings of the 6th audio mostly conference: a conference on interaction with sound. ACM, pp 39–46
Compton RJ (2003) The interface between emotion and attention: a review of evidence from psychology and neuroscience. Behav Cogn Neurosci Rev 2(2):115–129
Dakic V (2009) Sound design for film and television
Darken RP, Bernatovich D, Lawson JP, Peterson B (1999) Quantitative measures of presence in virtual environments: the roles of attention and spatial comprehension. CyberPsychol Behav 2(4):337–347
Demarque TC, Lima E (2013) Auditory hallucination: audiological perspective for horror games. SBC–Proceedings of SBGames 2013
Doyle J (2013) Subtlety of sound: a study of foley art. Senior honors projects, University of Rhode Island
Driver J, Spence C (1998) Cross-modal links in spatial attention. Philos Trans R Soc London B: Biol Sci 353(1373):1319–1331
Driver J, Spence C (1998) Crossmodal attention. Curr Opin Neurobiol 8(2):245–253
Dura MT (2006) The phenomenology of the music-listening experience. Arts Educ Polic Rev 107(3):25–32
Ekman I (2008) Psychologically motivated techniques for emotional sound in computer games. In: Proc. audio mostly 2008, 3rd conference on interaction with sound, Piteå, pp 20–26
Ermi L, Mäyrä F (2005) Fundamental components of the gameplay experience: analysing immersion. Worlds in Play: International Perspectives on Digital Games Research 37(2):37–53
Fan J, Gu X, Guise KG, Liu X, Fossella J, Wang H, Posner MI (2009) Testing the behavioral interaction and integration of attentional networks. Brain Cogn 70(2):209–220
Freeman J, Lessiter J (2001) Here, there and everywhere: the effects of multichannel audio on presence. In: Proceedings of the seventh international conference on audio display. GIT, Helsinki, pp 231–234
Garner T, Grimshaw M, Nabi DA (2010) A preliminary experiment to assess the fear value of preselected sound parameters in a survival horror game. In: Proceedings of the 5th audio mostly conference: a conference on interaction with sound. ACM, p 10
Garner TA, Grimshaw M (2013) Psychophysiological assessment of fear experience in response to sound during computer video gameplay. In: IADIS HCCI Conference
Gaver WW (1993) How do we hear in the world? Explorations in ecological acoustics. Ecol Psychol 5(4):285–313
Grani F, Overholt D, Erkut C, Gelineck S, Triantafyllidis G, Nordahl R, Serafin S (2015) Spatial sound and multimodal interaction in immersive environments. In: Proceedings of the audio mostly 2015 on interaction with sound, pp 1–5
Grau O (2003) Virtual art: from illusion to immersion. MIT Press
Grimshaw M (2007) The resonating spaces of first-person shooter games. In: Proceedings of the 5th international conference on game design and technology
Grimshaw M, Charlton J, Jagger R (2011) First-person shooters: immersion and attention. Eludamos J Comput Game Culture 5(1):29–44
Handel S (2006) Perceptual coherence: hearing and seeing. Oxford University Press
Hendrix C, Barfield W (1996) Presence within virtual environments as a function of visual display parameters. Presence: Teleoperators Virt Environ 5(3):274–289
Ihde D (2007) Listening and voice: phenomenologies of sound, 2nd edn. Suny Press
Iordanescu L, Guzman-Martinez E, Grabowecky M, Suzuki S (2008) Characteristic sounds facilitate visual search. Psychon Bull Rev 15(3):548–554
Iuppa N, Borst T (2012) Story and simulations for serious games: tales from the trenches. CRC Press
James W (1890) The principles of psychology, vol 1. Henry Holt, New York
Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, Walton A (2008) Measuring and defining the experience of immersion in games. Int J Human-Comput Stud 66(9):641–661
Jennett CI (2010) Is game immersion just another form of selective attention? an empirical investigation of real world dissociation in computer game immersion. PhD thesis, UCL (University College London)
Kitamura MS, Watanabe K, Kitagawa N (2016) Positive emotion facilitates audiovisual binding. Front Integr Neurosci 9:66
Larsson P, Västfjäll D, Kleiner M (2004) Perception of self-motion and presence in auditory virtual environments. In: Proceedings of seventh annual workshop presence, pp 252– 258
Larsson P, Västfjäll D, Olsson P, Kleiner M (2007) When what you hear is what you see: presence and auditory-visual integration in virtual environments. In: Proceedings of the 10th annual international workshop on presence, pp 11–18
Lipscomb SD (1999) Cross-modal integration: synchronization of auditory and visual components in simple and complex media. J Acoust Soc Am 105(2):1274
McMahan A (2003) Immersion, engagement and presence. Video Game Theory Reader 67:86
Mole C (2008) Attention and consciousness. J Conscious Stud 15(4):86–104
Moncrieff S, Dorai C, Venkatesh S (2001) Affect computing in film through sound energy dynamics. In: Proceedings of the ninth ACM international conference on multimedia. ACM, pp 525– 527
Murray JH (2017) Hamlet on the holodeck: the future of narrative in cyberspace. MIT press
Nilsson NC, Nordahl R, Serafin S (2016) Immersion revisited: a review of existing definitions of immersion and their relation to different theories of presence. Hum Technol, 12
Noesselt T, Bergmann D, Hake M, Heinze HJ, Fendrich R (2008) Sound increases the saliency of visual events. Brain Res 1220:157–163
Ozawa K, Ohtake S, Suzuki Y, Sone T (2003) Effects of visual information on auditory presence. Acoust Sci Technol 24(2):97–99
Parker JR, Heerema J (2008) Audio interaction in computer mediated games. Int J Comput Games Technol 2008:1
Pellegrini RS (2001) Quality assessment of auditory virtual environments. In: Hiipakka J, Zacharov N, Takala T (eds) Proceedings of the 7th international conference on auditory display. Laboratory of acoustics and audio signal processing and the telecommunications software and multimedia laboratory, pp 161–168
Petersen SE, Posner MI (2012) The attention system of the human brain: 20 years after. Ann Rev Neurosci 35:73–89
Posner MI (ed) (2012) Cognitive neuroscience of attention, 2nd edn. Guilford Press, New York
Posner MI, Boies SJ (1971) Components of attention. Psychol Rev 78(5):391–408
Posner MI, Petersen SE (1990) The attention system of the human brain. Annu Rev Neurosci 13(1):25–42
Raymond J (2009) Interactions of attention, emotion and motivation. Prog Brain Res 176:293–308
Raz A, Buhle J (2006) Typologies of attentional networks. Nat Rev Neurosci 7(5):367–379
Rooney B, Benson C, Hennessy E (2012) The apparent reality of movies and emotional arousal: a study using physiological and self-report measures. Poetics 40(5):405–422
Rueda MR, Fan J, McCandliss BD, Halparin JD, Gruber DB, Lercari LP, Posner MI (2004) Development of attentional networks in childhood. Neuropsychologia 42(8):1029–1040
Ryan ML (2003) Narrative as virtual reality: immersion and interactivity in literature and electronic media. Johns Hopkins University Press
Salselas I, Penha R (2019) The role of sound in inducing storytelling in immersive environments. In: Proceedings of the 14th international audio mostly conference: a journey in sound on ZZZ. ACM, pp 191–198
Schutz M, Lipscomb S (2007) Hearing gestures, seeing music: vision influences perceived tone duration. Perception 36(6):888–897
Serafin G, Serafin S (2004) Sound design to enhance presence in photorealistic virtual reality. In: Georgia Institute of Technology
Slater M (2009) Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos Trans R Soc London: Biol Sci 364(1535):3549–3557
Sodnik J, Tomazic S, Grasset R, Duenser A, Billinghurst M (2006) Spatial sound localization in an augmented reality environment. In: Proceedings of the 18th Australia conference on computer-human interaction: design: activities, artefacts and environments, pp 111–118
Sonnenschein D (2001) Sound design: the expressive power of music, voice, and sound effects in cinema. Michael Wiese Productions
Spence C, Driver J (1997) Audiovisual links in exogenous covert spatial orienting. Percept Psychophys 59 (1):1–22
Spence CJD (1997) Engineering psychology and cognitive ergonomics: volume two – job design and product design, chap Audiovisual Links in Attention Implications for Interface Design, Routledge
Tajadura-Jiménez A, Larsson P, Väljamäe A, Västfjäll D, Kleiner M (2010) When room size matters: acoustic influences on emotional responses to sounds. Emotion 10(3):416
Västfjäll D (2003) The subjective sense of presence, emotion recognition, and experienced emotions in auditory virtual environments. CyberPsychol Behav 6(2):181–188
Västfjäll D, Larsson P, Kleiner M (2002) Emotion and auditory virtual environments: affect-based judgments of music reproduced with virtual reverberation times. CyberPsychol Behav 5(1):19–32
Viers R (2011) Sound effects bible. Michael Wiese Productions
Visch VT, Tan ES, Molenaar D (2010) The emotional and cognitive effect of immersion in film viewing. Cogn Emotion 24(8):1439–1445
Watzl S (2017) Structuring mind: the nature of attention and how it shapes consciousness. Oxford University Press
Wenzel E, Wightman F, Kistler D, Foster S (1988) Acoustic origins of individual differences in sound localization behavior. J Acoust Soc Am 84(S1):S79–S79
Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence 7(3):225–240
Wu W (2011) What is conscious attention? Philos Phenomenol Res 82(1):93–120
Yewdall DL (2012) Practical art of motion picture sound. Routledge
Yiend J (2010) The effects of emotion on attention: a review of attentional processing of emotional information. Cogn Emotion 24(1):3–47
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Salselas, I., Penha, R. & Bernardes, G. Sound design inducing attention in the context of audiovisual immersive environments. Pers Ubiquit Comput 25, 737–748 (2021). https://doi.org/10.1007/s00779-020-01386-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00779-020-01386-3