Plucked String Stiffness Affects Loudness Perception

Plucked String Stiffness Affects Loudness Perception Mounia Ziat1, Ilja Frissen2, Gianni Campion3, Vincent Hayward4, and Catherine Guastavino2 1 Department of Psychology, Northern Michigan University, Marquette, MI, USA 2 McGill University & Centre for Interdisciplinary Research on Music, Media and Technology, Montreal, QC, Canada 3 McGill University, Montreal, QC, Canada 4 UPMC Univ Paris 06, Institut des Systèmes Intelligents et de Robotique, Paris, France Abstract. A great variety of interactions between senses, and between motor production and senses, have been reported in previous research. In the present study, we tested whether the mechanics of a plucked string affected how the sound it produced was perceived. To test this hypothesis, we simulated the feel of a plucked string using a high fidelity haptic force-feedback device and simultaneously simulated its acoustic emission. This way, we could independently manipulate the two sensory inputs — how it felt and how it sounded — together with physical correct haptic interaction and with accurate synchronization. This arrangement makes it very plausible that the two sensory inputs came from a common source. We used a two-interval forced-choice discrimination procedure to determine the point of subjective equality of the loudness between a stiff and a soft plucked string. When the stiffness of the string was low, the sound was perceived to be softer. Interestingly, this effect was found only when the first string was less stiff than the second string plucked during a comparison. The results are consistent with the inverse effectiveness principle of multisensory integration. Keywords: loudness perception, haptic stiffness, auditory-tactile integration. 1 Introduction Playing a musical instrument is a sensorimotor task. During this activity, it is expected that numerous interactions take place between the sensory channels, touch, vision, audition, and motor control. Several studies explored the impact of the auditory feedback on haptic perception during musical performance and how performance can be affected by haptic feedback from the instrument [1-3]. Most of these studies focused on timing and motor performance [4-5], rather than on how the produced sound is perceived by the player. Other studies found that touch was affected by audition [6-8], although not necessarily in a musical context. Very few studies investigated the effect of touch on auditory perception, despite the fact that the haptic feedback provided by the instrument is always present when playing an instrument. There is always some physical contact, even if the sensory-motor loop can be very loose as in the case of electronic or computer-based instruments. A well known and I. Oakley and S. Brewster (Eds.): HAID 2013, LNCS 7989, pp. 79–88, 2013. © Springer-Verlag Berlin Heidelberg 2013 80 M. Ziat et al. frequently discussed example is the case of the digital piano. The sound may be accurately synthesized, but the feel of the keyboard remains an issue for many performers. It is permitted to think that the light touch of the electronic keyboard, compared to the complicated dynamics of the escapement mechanism of a traditional piano, may affect how the sound is perceived by the player. Here we are interested in characterizing such as effect in the case of a plucked string, the feel of which is strongly dependent on its tension at rest. Another hint of the importance of the link that exists between the feel given by an instrument and its acoustic qualities can be found in the degree of inharmonicity of the sound of a plucked string. Inharmonicity, i.e., the degree to which overtone frequencies deviate from perfect integral multiples of the fundamental is known to add “warmth” to the sound [9]. The stiffer the strings are, the more inharmonicity they exhibit [10], which, in turn, affects loudness perception. This and other strong correlations of this type between the feel and the sound may be internalized by the auditory systems of performers and listeners alike as perceptual invariants. A possible outcome of our research could be suggestions aimed at improving the playability of digital instruments, which are often commented to provide a impoverished user interface for the player, compared to traditional instruments. Researchers suggested several devices to compensate for the quasi-elimination of haptic feedback in digital instruments, for a review see [11]. Here, we focus on the effects of touch on loudness judgment. Previous studies have found that a more intense haptic interaction with the instrument increased the perceived loudness of the produced sounds. For instance, with pianists, the perceived loudness increased exponentially with the force applied on the piano keys [12]. More recently, Okazaki et al. found that complex, broadband auditory stimuli tend to be perceived louder when the same signal was simultaneously heard and felt rather than when it was heard only [13]. Other results corroborated the effect of vibrotactile inputs on loudness perception, such as whole-body vibrations delivered through the chair while listening to a sound [14], or with irrelevant tactile, stimuli [15]. Schürmann et al., found that when tactile vibrations were delivered simultaneously with a sound, the latter was perceived 12% softer than when the sound was played alone [16], suggesting an increase in loudness perception due to the presence of the tactile stimulus. With the exception of [12], these previous studies typically used musically irrelevant signals delivered in static conditions, that is, eschewing the likely effects of sensori-motor couplings. Besides, participants were not actively creating the vibration that was the source of the sound and of the feel, as in a real instrument. Several recent studies have shown that actively generated movement affect haptic perception [17-18] and in some cases auditory-haptic perception [19]. 2 Pilot Study In the present pilot study, we employed a virtual string paradigm to measure perceived loudness under controlled conditions. A plucked string was simulated with reasonable realism in order to assess participants’ loudness perception when the Plucked String Stiffness Affects Loudness Perception 81 tactile feedback changed as a result of controlled modifications of the string stiffness. We used a range of sound loudness between 35.4 dB and 51.7 dB. In each trail, participants were presented with a pair of sounds, each associated with particular stiffness, and asked to indicate which of two sounded louder. We hypothesized that sounds presented at the same sound level would nevertheless be perceived to be louder when associated with greater stiffness, and vice versa, namely that sounds associated with lower stiffness would be perceived as quieter. 2.1 Participants Ten right-handed participants from McGill University completed the experiment and received $10 for their participation. None of them had any extensive experience with psychophysical procedures. Participants gave their informed consent before participating. Procedures were in accordance with the guidelines set out in the Declaration of Helsinki and approved by the McGill University research ethics board. 2.2 Apparatus and Stimuli The main apparatus was a Pantograph, a high-performance haptic device (Fig. 1) developed for rendering virtual surfaces [20]. The Pantograph can produce forces of up to 2 N in a two-dimensional workspace of 100 × 60 mm and has a flat response from DC to 400 Hz. The torque commands were processed by a low pass reconstruction filter, so that the commands to the motors matched the mechanical bandwidth of the system. Fig. 1. The Pantograph: A planar parallel mechanism with a nonslip plate on which the finger pad rests The auditory stimulus was generated using two delay lines following a variation of the well-known Karplus–Strong string synthesis algorithm [21]. The audio output was the sum of the two samples coming out of the delay lines at each sample of time. These two delay lines were terminated by low-pass Butterworth filters. The first delay line had a second order filter set at 500 Hz cut-off frequency, a depth of 50 samples, and a decay of 0.995. The second delay line had a 10th order filter with a cut-off at 82 M. Ziat et al. 400 Hz, a depth of 70 samples and a decay of 0.995. The digital simulation ran at a sampling frequency of 20 kHz. The input to the algorithm was an impulse triggered at the instant of release of the virtual string tension. The mono audio stimulus was presented to the two ears over sound-isolated headphones (Direct Sound EX-29). The subject was instructed to move the plate of the haptic device from left to right. Two strings were simulated side by side. The direction of the force returned by the device was always oriented to the left. The force intensity, | |, increased from a rest position, x0, up to a maximum value, xmax, at which point it was set to zero discontinuously to simulate the plucking effect. The simulation was repeated for the second string. For a stiffness, K, 0 if if 0 if if 0 otherwise | | where x0 and x1 are the positions of the strings in the workspace and x1 > (x0 + xmax). 2.3 Procedure After reading the instructions and having given their informed consent, the participants placed their right index finger on the pantograph’s plate interface. Throughout the experiment, their arm was comfortably supported by a soft gel pack placed near the right elbow. The participants entered their answers on a standard keyboard using their left hand. The participants engaged in an unspeeded, two-interval force-choice (2-IFC) auditory loudness discrimination task. On each trial the participant plucked the virtual string two times in succession and heard the corresponding auditory stimulus. In one of the intervals a standard (i.e., reference) stimulus was presented and in the other a comparison stimulus, and the task was to judge which one of the two strings sounded louder. We manipulated the loudness of the auditory stimulus as well as the stiffness of the string. Two standards were created by combining a string stiffness of 4 N/cm with one of two acoustic levels, 40.6 or 46.2 dB. For each of these standards six comparison acoustic levels were used, three below and three above the standard and equidistant (i.e., steps of 2.45 dB and 2.73 dB for the 40.6 and 46.2 dB standard, respectively). The comparisons were tested at one of three values of string stiffness, 1, 4, or 7 N/cm. Each combination of 2 (loudness intensity) x 6 (acoustic levels) x 3 (stiffness) was tested 8 times in completely random order. To avoid fatigue the repetitions were divided over two equal sessions. Performance was measured using the method of constant stimuli, from which we could determine the “point of subjective equal loudness” (PSE), which gives the acoustic level of the comparison that is perceived equally loud as the standard. Since we hypothesized that when the string was softer the sound would be perceived as Plucked String Stiffness Affects Loudness Perception 83 quieter, we expected that the level of the comparison would need to be higher than the standard to compensate for the perceptual difference. This would be reflected in a PSE that was larger than the standard loudness. The opposite effect would be observed when the virtual string was stiffer. 2.4 Data Analysis Figure 2 illustrates how the dependent measures were obtained. We first pooled the raw data for the two runs. We calculated, for each comparison and for each string stiffness, the proportion of trials in which the comparison had been perceived as the louder of the two intervals. Individual psychometric functions were obtained by fitting cumulative Gaussians using the software package ‘psignifit’ [22]. From the fits we calculated the PSE. Statistical analyses were conducted using RStudio running R version 2.15.1. For repeated measures ANOVAs we employed the ‘car’ package [23]. We used a significance level of 0.05. Fig. 2. An illustrative example of the analysis of the psychometric function of one participant for the condition with standard 40.6 db and string stiffness 4 N/cm for both standard and comparison stimulus. The figure demonstrates how the PSE are extracted from cumulative Gaussian fits to the response data. On the abscissa are the comparison acoustic levels and on the ordinate, the proportion of times that the auditory stimulus was perceived louder. 2.5 Results Figure 3a shows the PSEs as a function of the stiffness of the comparison string for the S-C order (a) and the C-S order (c), from which a number of observations can be made. First of all, there were clear effects of the ordering of the stimuli on the overall pattern of results. Whereas C-S showed apparent effects of stiffness on PSE this was not the case for S-C. A 2 (Order) x 2 (Acoustic Reference) x 3 (Stiffness) ANOVA with Order as a between subjects factor and Acoustic Reference and Stiffness as within subject factors showed that all main and effects and interactions were significant (all p-values < 0.05). We therefore discuss the results separately for the two order groups. 84 M. Ziat et al. The effects most pertinent to the current question were observed in the C-S group. Changing the stiffness of the string appeared to change its perceived loudness, although this effect was apparently restricted to when the stiffness was lowered to 1 N/cm. When the comparison stiffness was lower than the standard, the comparison sound had to be presented at a higher level in order to be perceived at equal loudness. In other words, consistent with the hypothesis, sounds associated with a softer string were perceived to be quieter. The reverse, however, was not observed. Making the comparison string stiffer did not increase the perceived loudness of the sounds. For the S-C group on the other hand there was no apparent effect of stiffness at all. This appreciation of the results was followed up with a separate 2 (Acoustic Reference) x 3 (Stiffness) repeated measures ANOVA for each order group. Indeed, for the C-S group there was a significant effect of Stiffness (F(2,8) = 21.85, p < 0.01), Acoustic Reference (F(1,4) = 241.6, p < 0.001), and their interaction (F(2,8) = 4.69, p < 0.05). For the S-C group, on the other hand, there was only a significant effect of Acoustic Reference (F(1,4) = 134.1, p < 0.001), but none for Stiffness (F < 1) or the interaction term (p = 0.14). Fig. 3. Results shown separately for the group that had the standard always before the comparison (panels a and b) or the comparison always before the standard (panels c and d). The left column shows the PSEs as functions of the stiffness of the comparison string for the two auditory reference levels (40.6 and 46.2 dB). The right column shows the relative change in perceived loudness as a function of the change in string stiffness (±3 N.cm-1 with respect to the standard stiffness). Negative values indicate that the sound was perceived to be less loud, and vice versa. To further illustrate the effects of changing string stiffness we calculated the difference in PSE between the standard stiffness and the two different levels of stiffness such that a negative value corresponds to a reduction in the perceived loudness and a positive value an increase in perceived loudness. These results are plotted as a function of the change in stiffness between the standard and comparisons in Figure 3b and d. Plucked String Stiffness Affects Loudness Perception 85 Thus for the C-S case, when the stiffness decreased by 3 N/cm, the perceived loudness decreased, whereas it did not change for an increase in stiffness. One-sample ttest showed that the shift in perceived loudness was significantly different from zero for the 40.6 dB stimulus [t(4) = 7.89, p < 0.01], and close to significant for the 46.2 dB stimulus (p = 0.053). No significant differences were observed for any of the other cases (all p-values > 0.07). 3 General Discussion In this pilot experiment, we investigated the effect of the stiffness of a virtual string on the perceived loudness of an actively generated musical tone using standard psychophysical methods. The results showed a significant effect of changes in string stiffness on the perceived loudness of the sound created by plucking the string. They also show that this effect is strongly moderated by whether the string stiffness increases or decreases. Thus, for the stiffness value of 1 N/cm, the participants perceived the comparison sound as being softer than the standard sound where string stiffness was 4 N/cm. In other words, the second sound had to be louder to be perceived as equally loud as the first sound. This effect was larger for the 40.6 dB standard sound than the 46.2 dB. Increasing string stiffness had no perceptual effect whatsoever. Moreover, there was a very strong stimulus order effect. The effect of string stiffness was only observed if the comparison was always presented before the standard. We will address these findings in turn. 3.1 Effect of String Stiffness on Perceived Loudness At this point we will restrict our discussion of the effects of changes in string stiffness on perceived loudness to the results obtained with the group of participants for which the comparison stimulus was always presented before the standard. We defer the discussion of stimulus order effects to a later section. The fact that a softer string leads to it being perceived as less loud is in accordance with our hypothesis. Interestingly, whereas earlier studies, such as [16], found an average increase in perceived loudness due to a tactile stimulus of 12% using arbitrary stimuli, the effect observed here was less than 6% which is close to the typical loudness discrimination threshold of around 5% [25]. Before concluding that this then trivializes the effect, we should keep in mind the procedural context of the experiment. Whereas loudness thresholds are typically determined for entirely passively received acoustic stimuli, the stimuli in our experiment were created through an active involvement on the part of the participant. We already mentioned in the introduction that voluntary active movements can affect haptic perception [17] and auditory-haptic interactions [19], the nature of which are still far from understood [17]. We therefore have to consider the possibility that the previously observed facilitatory, or “boosting”, effect of tactile stimuli on loudness perception can be modulated by the voluntary actions on the part of our participants. 86 M. Ziat et al. The fact that the effect of string stiffness was larger for the softer of the two auditory standards can be understood from the principle of inverse effectiveness which states that multisensory integration is stronger for weaker signals [24]. It can be argued that the responsiveness to softer tones is smaller than for the louder ones and is therefore more susceptible to crossmodal influences, in this case from the tactile sense. If this holds then we would expect larger effects of stiffness on increasingly softer tones. This notion should be tested in future experiments. Inverse effectiveness, however, cannot easily explain why we did not observe an effect of an increase in stiffness since the auditory input per se was physically identical to the one when there was a reduction in stiffness. We can speculate as to the cause of this asymmetry. It can be that a reduction in stiffness is more salient than an increase. In other words, -3 N/cm effectively presents a larger change than +3 N/cm, and therefore can exercise a larger crossmodal effect. Alternatively, Odgaard et al. found that for a visual-auditory interaction there was no interaction between loudness and brightness when the masking stimulus was kept constant in the two intervals [26]. In our experiment, we explicitly instructed the participants to focus on stimulus intensity which might explain the absence of effect of the highest stiffness on loudness. Since the order of standard and comparison was fixed, it is possible that some participants might have been susceptible to this bias for the strongest stiffness. Since very little research has been done on this it remains unclear how the response bias affects loudness perception specifically. 3.2 Effects of Stimulus Presentation Order A remarkable result of our pilot experiment was that we obtained significant effects of changes in string stiffness when the comparison was presented before the standard (C-S) but none when the presentation order was reversed (S-C). We there-fore need to consider the ramifications of stimulus presentation order. Although the order was randomized across participants for each individual participant the order was fixed. This could mean that we did not observe a change in participants’ perception of the auditory stimulus but a modification in their response criterion, which could lead to biased results. The interval-bias hypothesis, for instance, states that the influence of presentation order affects performance accuracy in a discrimination task, a phenomenon known as the time-order error (TOE) and is supposedly an attentional effect [27] in which the first stimulus cues attention to the occurrence of the second stimulus. A consequence is that participants start favoring one interval over another. These kinds of biases are apparently complicated further by other factors such as sensory modality. For instance, Okazaki et al [13] tested a number of standard and comparison conditions. On half the conditions the standard was haptic and in the other the standard was auditory. Participants would always perceive the second stimulus as more intense than the first, irrespective of the actual order of the stimuli. However, this response pattern was not observed when the standard was auditory. Although, in our case both standard and comparisons are always auditory-haptic, a condition not considered in [13], it remains unclear to what extent these kinds of response order effects played a role. We need to keep in mind that although these kinds of extraneous influences can introduce “false results” or cancel out genuine perceptual effects they do so by adding Plucked String Stiffness Affects Loudness Perception 87 constant offsets to the results. Close inspection of the present results, however, shows that they cannot be entirely explained by simple offsets. This becomes most apparent when considering the differences between the two acoustic references. Consider Figure 3a and assume for the sake of argument that participants tend to favor the second interval as they response, which in this case would be the comparison. The influence of such a bias would lead to a systematic underestimation of (i.e., decrease in) the PSE, which is what we observe for the 46.2 dB reference. However, it is immediately clear that this bias cannot explain the tendency for the PSE to be higher than the reference for the 40.6 dB reference. Moreover, a similar appreciation can be made when we consider Figure 3c. In short, no single bias can explain the present order effects. We cannot at this point exclude the influence of a complex of biases. Further studies should consider testing procedures that minimize stimulus order effects, such as completely randomizing the presentation order of the standard and the reference or direct comparison and adjustment procedures such as used by Schürmann et al [16]. 4 Conclusion In conclusion, we have shown that changing the stiffness of a virtual string can change the perceived sound pressure level of sounds associated with that string. Although subject to methodological complications, this finding is not only of theoretical interest in our further understanding of multisensory perceptual processes but also is encouraging for auditory-haptic applications related to music performance. Indeed, the cognitive processes of music are different from listening to a single sound or speech. Playing a musical instrument not only requires the auditory feedback but also requires basic motor functions such as timing, sequencing and spatial organization of the movement [28] as well as possibly strong attentional efforts. References 1. Repp, B.H.: Effects of auditory feedback deprivation on expressive piano performance. Music Perception 16, 409–438 (1999) 2. Pfordresher, P.Q., Palmer, C.: Effects of hearing the past, present, or future during music performance. Perception and Psychophysics 68, 362–376 (2006) 3. Drake, C., Penel, A., Bigand, E.: Tapping in time with mechanically and expressively performed music. Music Perception 18, 1–23 (2000) 4. Goebl, W., Palmer, C.: Tactile feedback and timing accuracy in piano performance. Experimental Brain Research 186, 471–479 (2008) 5. Palmer, C., Koopmans, E., Loehr, J.D., Carter, C.: Movement-related feedback and temporal accuracy in clarinet performance. Music Perception 26, 439–449 (2009) 6. Avanzini, F., Crosato, P.: Haptic-auditory rendering and perception of contact stiffness. In: McGookin, D., Brewster, S. (eds.) HAID 2006. LNCS, vol. 4129, pp. 24–35. Springer, Heidelberg (2006) 7. Soto-Faraco, S., Deco, G.: Multisensory contributions to the perception of vibrotactile events. Behavioural Brain Research 196, 145–154 (2009) 8. Guest, S., Catmur, C., Lloyd, D., Spence, C.: Audiotactile interactions in roughness perception. Experimental Brain Research 146(2), 161–171 (2002) 9. Fletcher, H., Blackham, E.D., Stratton, R.: Quality of piano tones. J. Acoust. Soc. Am. 34(6), 749–761 (1962) 88 M. Ziat et al. 10. Järvelälinen, H., Välimäki, V., Karjalainen, M.: Audibility of inharmonicity in string instrument sounds, and implications to digital sound synthesis. In: Proc. Int. Computer Music Conf., Beijing, China, pp. 359–362 (1999) 11. Marshall, M., Wanderley, M.: Vibrotactile feedback in digital musical instruments, Internat. In: Conference on New Interfaces for Musical Expression (NIME 2006), pp. 226–229 (2006) 12. Kinoshita, H., Furuya, S., Aoki, T., Altenmüller, E.: Loudness control in pianists as exemplified in keystroke force measurements on different touches. Journal of Acoustical Society of America 121, 2959–2969 (2007) 13. Okazaki, R., Kajimoto, H., Hayward, V.: Vibrotactile Stimulation Can Affect Auditory Loudness: A Pilot Study. In: Isokoski, P., Springare, J. (eds.) EuroHaptics 2012, Part II. LNCS, vol. 7283, pp. 103–108. Springer, Heidelberg (2012) 14. Merchel, S., Leppin, A., Altinsoy, E.: Hearing with your body: the influence of wholebody vibrations on loudness perception. In: Proceedings of the Sixteenth International Congress on Sound and Vibration (ICSV16), Kraków, Poland, July 5-9 (2009) 15. Gillmeister, H., Eimer, M.: Tactile enhancement of auditory detection and perceived loudness. Brain Research 1160, 58–68 (2007) 16. Schürmann, M., Caetano, G., Jousmäki, V., Hari, R.: Hands help hearing: facilitatory audiotactile interaction at low sound-intensity levels. J. Acoust. Soc. Am. 115(2), 830–832 (2004) 17. Frissen, I., Ziat, M., Campion, G., Hayward, V., Guastavino, C.: The effects of voluntary movements on auditory-haptic and haptic-haptic temporal order judgments. Acta Psychologica 141, 140–148 (2012) 18. Vitello, M.P., Ernst, M.O., Fritschi, M.: An instance of tactile suppression: Active exploration impairs tactile sensitivity for the direction of lateral movement. In: Proceeding of Eurohaptics, Paris, July 3-6, pp. 351–355 (2006) 19. Kitagawa, N., Kato, M., Kashino, M.: Assessing the effect of voluntary action on sensitivity to temporal asynchrony between auditory and somatosensory events. In: 10th International Multisensory Research Forum (IMRF), New York, June 29 - July 2 (2009) 20. Campion, G., Wang, Q., Hayward, V.: The Pantograph mk-II: A haptic instrument. In: Proceedings of the Internat. Conf. on Intelligent Robots and Systems 2005, pp. 723–728 (2005) 21. Karplus, K., Strong, A.: Digital Synthesis of Plucked String and Drum Timbres. Computer Music Journal 7(2), 43–55 (1983) 22. Wichmann, F.A., Hill, N.J.: The psychometric function: I. Fitting, sampling, and goodness of fit. Perception & Psychophysics 63, 1293–1313 (2001) 23. Fox, J., Weisberg, S., An, R.: An R companion to applied regression, 2nd edn. Sage Publications, Thousand Oaks (2011) 24. Stein, B.E., Meredith, M.A.: The merging of the senses. MIT Press, Cambridge (1993) 25. Coren, S., Ward, L.M., Enns, J.T.: Sensation and perception. Harcourt Brace, Fort Worth (1999) 26. Odgaard, E.C., Arieh, Y., Marks, L.E.: Cross-modal enhancement of perceived brightness: Sensory interaction versus response bias. Perception & Psychophysics 65(1), 123–132 (2003) 27. Hairston, I.S., Nagarajan, S.S.: Neural mechanisms of the time-order error: An MEG study. Journal of Cognitive Neuroscience 19, 1163–1174 (2007) 28. Zatorre, R.J., Chen, J.L., Penhune, V.B.: When the brain plays music: auditory-motor interactions in music perception and production. Nat. Rev. Neuroscience, 547–558 (2007)