Open AccessArticle

Music to My Ears: Developing Kanji Stroke Knowledge through an Educational Music Game^†

Oleksandra G. Keehl

^* and

Edward F. Melcer

Computational Media, Baskin School of Engineering, University of California, Santa Cruz, CA 95064, USA

Author to whom correspondence should be addressed.

^†

This paper is an extended version of our paper published in the Proceedings of the 14th International Conference on the Foundations of Digital Games, New York, NY, USA, 26–30 August 2019.

Multimodal Technol. Interact. 2021, 5(12), 83; https://doi.org/10.3390/mti5120083

Submission received: 17 November 2021 / Revised: 5 December 2021 / Accepted: 9 December 2021 / Published: 17 December 2021

(This article belongs to the Special Issue Innovations in Game-Based Learning)

Download

Browse Figures

Figure 1
A visual mnemonic for the meaning of kanji “body” from “Kanji Pict-o-Graphix,” a popular book for learning kanji with mnemonics [<a href="#B9-mti-05-00083" class="html-bibr">9</a>]. "> Figure 2
Two examples from WaniKani, a popular kanji learning website. On the left is a pronunciation mnemonic for the kanji 二, when it is part of the word 二つ, meaning “two things.” On the right is a good example of WaniKani’s shock-value-driven mnemonic. The humorously named “poop” radical is apt to stick out in a student’s mind. "> Figure 3
The anatomy of kanji. Each kanji consists of one or more radicals. In turn, each radical consists of one or more strokes. "> Figure 4
Melodies for the radicals “tree” (above) and “water” (below). We tried to make the pitch of the melody follow the direction of the strokes. A horizontal stroke is represented by a steady note. A long downward stroke is accompanied by a significant drop in pitch, while a shorter upward stroke is matched by a moderate rise in pitch. "> Figure 5
The list of radicals and kanji used in our first study. From left to right: person, mouth, tree, rest, episode, annoyed. "> Figure 6
Radical Tunes screenshots: (a) meaning pre- and post-test; (b) stroke order pre- and post-test; (c) stroke learning screen; and (d) timed and scored game portion. "> Figure 7
Screenshots of our app. Left: the meaning test screen. Middle: the learning screen. The dotted line indicates where the next stroke should be drawn, with the biggest dot indicating the beginning of the stroke. Right: the writing test screen. The second line of text displays how many strokes the target kanji has and how many the participants already drew. The “listen” button in the treatment condition would play the kanji’s melody. The button is absent in the control version of the app. "> Figure 8
The scores for meaning retention (calculated by subtraction of post-test meaning scores from the longitudinal test scores) plotted against the time that passed between the two tests. The linear fit function shows no negative correlation between the length of time and the changes in scores. "> Figure 9
The 15 characters selected for our study. All of the kanji consist of some combination of the six common radicals (highlighted). "> Figure 10
Two of the post-test images for the kanji “foreign.” For scoring purposes, the color indicates the order in which the strokes were drawn, while the large initial dot indicates the direction of the stroke. The participant on the left drew the strokes in the correct order, but the first stroke of the moon radical (purple) was drawn in the wrong direction. The participant on the right drew all the strokes in the correct direction (with the exception of the missing last stroke), but switched the order of the first two strokes in the “ten” radical. The participant on the left received 9/9 on order and 8/9 on direction. The participant on the right received 6/9 for order and 8/9 for direction. ">

Review Reports Versions Notes

Abstract

Millions of people worldwide are taking up foreign languages with logographic writing systems, such as Japanese or Chinese. Learning thousands of characters necessary for literacy in those languages is a unique challenge to those coming from alphabetic backgrounds, and sustaining motivation in the face of such a momentous task is a struggle for many students. Many games exist for this purpose, but few offer production memory practice such as writing, and the vast majority are thinly veiled flashcards. To address this gap, we created Radical Tunes—a musical kanji-writing game—which combines production practice with musical mnemonic by assigning a melody to each element of a character. We chose to utilize music as it is a powerful tool that can be employed to enhance learning and memory. In this article, we explore whether incorporating melodies into a kanji learning game can positively affect the memorization of the stroke order/direction and overall shape of several Japanese characters, similar to the mnemonic effect of adding music to text. Specifically, we conducted two experimental studies, finding that (1) music improved immersion—an important factor related to learning; and (2) there was a positive correlation between melody presence and character production, particularly for more complex characters.

Keywords:

educational game; music; mnemonic device; design

1. Introduction

People have many reasons to want to learn a foreign language: some want to enjoy foreign media in its original form, some need it for business or travel, some for personal enrichment, etc. Roughly 1/6 of Earth’s population learn languages with logographic writing systems (LWSs) such as Chinese, Japanese and Korean as their native tongues [1]. These people harbor enormous cultural, economic and academic wealth. As a result, these languages are a popular choice for a variety of learners ranging from serious businessmen to K-pop and anime fans. Whatever the motivation, learning additional languages carries a slew of cognitive benefits and should be encouraged [2].

Maintaining the motivation and persistence needed to learn a language is a difficult task, and the problem is amplified for learners with alphabetic roots tackling an LWS language [3]. One needs to memorize over 2000 characters to be considered literate in their target language [4]. This is routinely achieved through rote memorization, and this prospect discourages many would-be learners. When it comes to LWS as a foreign language, there is a divide between conversational fluency and literacy unlike that in any alphabetic language [3]. Our research aims to help bridge this gap through the development and evaluation of LWS learning games.

In recent years and with some success, many aspiring polyglots are turning to mobile apps to help them overcome this hurdle [5,6]. The advantages are clear: digital media allows for instant feedback, gamification, inclusion of engaging graphics, audio and video and even finger-writing practice anytime and anywhere [5]. Many mnemonic devices have been invented to help students remember the meaning, pronunciation and shape of characters (see Figure 1 and Figure 2). However, in addition to the characters’ complex designs, there is also a correct order and direction to each stroke within the character. To our knowledge, beyond several rules of thumb fraught with exceptions, no one has created mnemonic devices for remembering the correct stroke order and direction (a.k.a. stroke knowledge). Research suggests that stroke knowledge is positively correlated with obtaining reading and writing fluency, especially in the early stages of learning logographic scripts [7,8].

This lack of stroke knowledge mnemonics is something we seek to change. Music has been shown to be a powerful mnemonic device for memorizing text sequences [10]. One does not have to look far to find the examples, such as the ABC or the 50 states songs. Could melody be used to memorize a sequence of strokes?

To address these issues and the above question, we developed an educational kanji-writing musical game. We named our game Radical Tunes [11], since its core idea is adding mnemonic melodies, or tunes, to radicals, the building blocks of kanji (see Figure 3). We hope that by adding the musical elements to the game we can turn the monotonous task of writing kanji over and over into an enjoyable and educationally effective activity. To this end, we are counting on the mnemonic properties of music [10] and the popularity of music games—such as Osu, which boasts more than 15 million registered players (https://osu.ppy.sh/home, accessed 9 November 2021). We hypothesized that if each stroke in a character had a specific melodic element assigned to it, the sequence of strokes within a character would give it a unique melodic signature, which might help students with their stroke knowledge acquisition and retention.

Beyond its mnemonic potential and aesthetic appeal, music has also been shown to notably alter immersion [12,13]. Cairns et al. [14] describe immersion as “the degree of involvement that players have with different aspects of the game leading to a move of the attention, awareness and thoughts of the player from the real world around them to the events happening within the game”. This suggests that immersion is an important part of the gaming experience. Other research shows that it can positively influence learning in several ways [15,16], making it important for educational games in general. Therefore, we also hypothesized that incorporating music into the design of Radical Tunes would improve its immersion as well.

In this article, we provide the design rationale for Radical Tunes—which ties mnemonic melodies to radicals within kanji [11]. We describe two experimental studies exploring melody’s effect on the engagement, immersion and learning of LWS. Our studies showed that music in Radical Tunes enhanced player/learner immersion. We also found that there was a positive correlation between melody presence and character production, particularly for more complex characters.

The contributions of this work are as follows:

A novel mnemonic design for acquiring stroke knowledge;
A scalable technique for creating a responsive musical experience that can enhance logographic writing practice;
Implications for design of audio in logographic learning environments to improve their efficacy.

2. Related Work

2.1. Challenges and Methods of Learning Logographic Characters

In our study, we used Japanese characters, however, due to common origin, similarities in structure, and in fact, a significant overlap (approximately 60% of the 2000+ most commonly used Chinese and Japanese characters were the same or similar as of 2012 [17]) between Japanese kanji and Chinese hanzi, it is fairly safe to say that challenges and methods for learning to write one group of these characters apply to the other as well. This is why we drew on the literature on learning to write in both Japanese and Chinese languages when motivating our study. Notably, learning the pronunciation and meaning of the characters is quite different between the languages and is outside of the scope of this paper.

The traditional and most widely used way of learning these characters is rote memorization, requiring students to write the characters over and over [18]. The importance of learning the correct stroke order is debated by some, since some of the important practical application of that knowledge, such as looking up a character in a dictionary or being able to produce characters by hand, have lost much of their value with the advent of romaji/pinyin digital input methods (romanized phonetic scripts for Japanese and Chinese, respectively), which allow users to write the character’s pronunciation and then select the appropriate character from the list of those that match that the pronunciation [19]. However, stroke knowledge has been linked with successful writing skill development, especially at the early stages of mastering an LWS [8]. Incorrect stroke order has also been linked with errors in hanzi production [20]. This highlights the importance of learning the correct stroke order for LWS.

A survey of strategies of successful students of Japanese kanji revealed that they employed a number of techniques, one of the prominent ones being component analysis, that is breaking up the characters into their components (as can be seen in Figure 3), or graphemes, for further analysis and memorization. Dr. Heath Rose [3], a researcher who extensively studied the struggles of mastering the writing of Japanese kanji for students from an alphabetic background, suggests that developing graphemic awareness at the beginning of a student’s kanji learning quest can aid them in developing their writing skills. We suggest that our design of assigning melodic signatures to radicals within kanji can help students both memorize and develop the graphemic awareness Rose talks about.

In recent years, mobile apps for learning kanji and hanzi have gained popularity. Apple’s App Store and Google Play Store list hundreds of apps and games developed for this purpose, some with millions of downloads (estimates obtained by searching for “kanji learning” and “hanzi learning” in the stores’ web interface and counting the rows of results). The vast majority of the apps and games focus on flashcard-style recall tasks, with few offering writing practice and none, to our knowledge, offering mnemonics for remembering the stroke order and direction—or combining writing with music. A study on the use of multimedia (including but not limited to apps) for learning kanji showed that such methods were effective when used strategically (e.g., taking advantage of the spaced repetition included in many flashcard apps), and were also more enjoyable than the pen and paper flashcards and writing practice [18].

2.2. Music’s Effect on Memory and Immersion

Music has been linked to many aspects of human performance. Hallam et al. suggested that calming music improved children’s performance in mathematics and word memorization tasks, and increased altruistic tendencies [21]. Music has also been tied to immersion in video games [12], which, in turn, has been shown to positively affect learning [16,22]. One of the core components of immersion is engagement [14,16]. Many of the papers described in the Second Language Acquisition (SLA) section, cite engagement as a factor leading to improvements in learning outcomes. Consequently, immersion is an important aspect to consider in our study.

Music’s positive effect on the verbatim recall of text are well documented [23] and have been used by foreign language educators for decades to teach grammar, vocabulary and pronunciation, and to promote a positive atmosphere in classrooms [24]. Music and song are also used for memorizing sequences of words in other disciplines [25]. One would be hard-pressed, however, to draw parallels between learning stroke order and learning a block of text. While, to our knowledge, there are not any studies of music, memory and writing or drawing, there have been some on the music and memory of movement. One study showed that music helped persons with Alzheimer’s memorize a sequence of gestures (though no similar effect was observed in healthy aging adults) [26]. Surveys of professional and casual dancers revealed that dancers employ myriad techniques to help them remember sequences of moves in a particular dance [27,28]. A number of them stated that relating movements to the music was instrumental in their memorization of the dance. While these are promising examples of mnemonic effects of music for memorizing nonverbal sequences, to our knowledge, the theory has not been applied to learning logographic writing. Therefore, we decided it warranted further investigation in the context of stroke knowledge acquisition.

2.3. Second-Language Acquisition Games

Numerous studies have explored the use of games in SLA, including kinesthetic learning [29,30], augmented and alternate realities [31,32], collaborative learning [29,31,32,33,34,35], and roleplaying [22] in games. These games targeted a variety of aspects involved in language acquisition, including vocabulary and meaning [22,36,37], pronunciation and tones [34,35,38], grammar [31], writing [29,33] and even culture [22,32].

According to many of these studies, increased engagement was one of the primary benefits of incorporating games into SLA. Multiple findings showed that the majority of players reported feeling challenged and entertained, which could result in spending more time on the learning activity. Other noted benefits were ease of access for mobile device games [34] and decreased stress levels due to not having to perform in front of peers and teachers [31]. However, to our knowledge, there have not been any studies on the effects of music on the memorization of logographic characters.

It is worth noting that the negative effects of games in language learning have also been observed. Dehaan et al. used a commercial musical game for vocabulary learning, Parappa the Rapper 2, to study how the cognitive load of interactive games affects learning [36]. They had some students play the game, while other students observed the gameplay with no interaction. The authors found that the students who played the game retained fewer vocabulary words than the students who merely watched, leading to the conclusion that interactivity in games has the potential to negatively impact learning outcomes. We took this concern into account while designing Radical Tunes. In Parappa the Rapper 2, the gameplay required the player to perform sequences of button presses in time with the music. This activity could be mastered with no attention given to the vocabulary words accompanying the music. In Radical Tunes, the melody unfolds with each stroke the player draws correctly, and thus the game’s interactivity is integral to the learning activity rather than distracting from it.

3. Radical Tunes

Radical Tunes is an educational, musical kanji-writing game. Kanji are the Japanese logographic characters originally derived from the Chinese written script. The building blocks of kanji are called radicals and strokes (Figure 3). Each kanji consists of at least one radical. Each radical carries a meaning, which sometimes, but not always, relates to the meaning of the kanji. These meanings are often used in designing mnemonics for learning characters. For instance, WaniKani uses their own made-up humorous meanings for radicals (https://www.wanikani.com, accessed 9 November 2021). They may not have much to do with the original meaning, but possess a strong mnemonic effect through the sheer shock value (Figure 2, right). Radicals, in turn, are composed of one or more strokes. Individual strokes are purely graphical elements with no assigned meaning.

Numerous mnemonics have been created to assist with the memorization of the shape, meaning and pronunciations of kanji [9]. However, to our knowledge, none of them are intended to help learn the correct writing stroke order—which is crucial for handwriting proficiency [8]. We created Radical Tunes to address this need. Music has been shown to have a positive effect on text memorization, particularly when the melody rhythmically aligned with the material [10]. We wanted to see whether this musical mnemonic effect could be extended to the memorization of the correct stroke order for writing kanji.

In Radical Tunes, we implemented the musical mnemonic mechanism by assigning a unique melody to each radical, thus providing each radical with a melodic signature of sorts. Furthermore, each stroke within a radical is assigned an element of that melodic signature. As a result, when a student writes a character, they will produce a unique melody corresponding to that character, unfolding with each correctly written stroke. We attempted to further amplify the effect by matching the changes in the melody’s pitch to the direction of the strokes (Figure 4). For example, a downward stroke would go down in pitch. This design leverages the concept of conceptual/embodied metaphors [39,40], which have been shown to effectively map movement to music [41,42]. The melody of the “tree” radical 木 consists of the melody for the “ten” radical 十 with the additional two stroke tunes. The version of Radical Tunes used in the first study used pre-recorded melodic elements for each stroke, which allowed us to use different instruments for each radical, to further distinguish them from each other. However, if the player drew the stroke too quickly or too slowly, the sound and the drawing would be out of sync. For the second study, we used procedural sound generation, which made for a more responsive drawing experience, as the sound could last for as long as the player’s finger touched the screen. However, this limited us to only the single type of melodic sound that we could procedurally generate. This is relevant, as an informal post hoc survey after the second study revealed that many of the melodies were perceived as too similar, which may have muddled the mnemonic effect. This is further discussed in the Implications for Design section.

4. Study 1

4.1. Methods

With our pilot study, we set out to investigate the effects of music on learning within Radical Tunes. Considering that immersion in educational games can be a notable factor affecting learning outcomes [15,16] and that music was shown to affect immersion [12], we also wanted to observe whether the musical nature of our game positively affected player immersion.

We created two versions of our game for the first study: (1) the original Radical Tunes with unique tunes for each radical and melodic elements accompanying each stroke; and (2) the control version, where instead of music, the writing activity was accompanied with the sound of chalk on a blackboard.

For the purpose of our initial study, we chose three radicals from the six most commonly occurring in the Jōyō kanji (the 2136 characters required for baseline literacy in Japan). To increase complexity, we also added three kanji that were composed exclusively of those radicals, for a total of six characters for our participants to learn (Figure 5).

4.1.1. Procedure

The participants were informed that the study was about an educational game, and that they would be learning how to write six Japanese characters. We assigned them randomly to either the control or music group and provided them with an Android tablet with the corresponding version of Radical Tunes. To facilitate the study, the meaning and stroke order tests were incorporated into the beginning and end of the app (Figure 6a,b).

For the assessment of meaning knowledge, the participants had to select the correct meaning from six answer options for each of the characters (Figure 6a). Both pre- and post-tests had the same options available. To test the stroke order accuracy, the participants had to trace the outlines of the characters on the screen. The orange dots placed along the lines of the characters would turn green once touched, to help the participants keep track of which parts they already traced (Figure 6b).

We gave the participants no feedback on these evaluations to prevent their learning during the pre-test via the confirmation of their guesses.

After the pre-tests were completed, the participants proceeded to the learning part of Radical Tunes. First, they watched the animation of the proper stroke order with accompanying sounds. They then had to trace the strokes by following the arrows on the screen (Figure 6c). Finally, they had to trace the character without any visual cues. If they made a mistake, a soft ping would sound, and they would have to draw the latest stroke again. We limited each step to two repetitions that could not be skipped to ensure that all participants received an identical amount of instruction.

The subjects then played the Radical Tunes game, where they had to trace the characters appearing on the screen as quickly and accurately as they could (Figure 6d). Each character appeared twice during the game in a randomized order. All participants had the same randomized order.

At the end of the game, the final score and time were displayed. The participants were then asked to take the post-tests for meaning and stroke order. Finally, they were directed to fill out the demographic and experience questionnaires on a laptop.

4.1.2. Participants

A total of 24 subjects participated in the study. We recruited them through classroom announcements and by word of mouth outside of the university. There were 12 male, 11 female and 1 non-binary participant. The ages ranged from 19 to 51 with the average age of 25.6 (SD = 7.8). Despite the wide variability in age among the participants, age did not appear to be a significant predictor for any of our dependent variables (all p’s > 0.09).

All participants were assigned to one of the two conditions on a random basis: music (5 male, 7 female) and control (1 non-binary, 4 female, 7 male). None of the participants reported any prior knowledge of Japanese or Chinese and were not foreign language learners.

4.1.3. Measures

Immersive Experience Questionnaire (IEQ)
To measure the immersion levels experienced by our participants, we used the IEQ [43]. The developers of this questionnaire conducted an in-depth validation study and a subsequent evaluation by Denisova et al. reported the Cronbach’s $α$ for the IEQ at 0.91 [44]. IEQ measures five aspects of immersion: cognitive involvement, emotional involvement, real-world dissociation, control and challenge. The cognitive involvement accounts for how much effort and attention the player devoted to the game. Emotional involvement measures the extent to which the player found the game enjoyable and whether they wanted to see more of it. The real-world dissociation measures how much the player lost track of time and the sense of their surroundings. The control dimension accounts for whether the player perceived the controls of the game as responsive and easy to use. The challenge dimension measures the player’s perception of the game’s difficulty. In the paper introducing the IEQ, Jennett et al. claimed that immersive qualities are a key factor contributing to whether a game is perceived as good. If Radical Tunes is to succeed in its educational goals, it must be capable of holding the players’ attention and giving them a pleasant experience that encourages them to return. We chose IEQ to help us assess how well Radical Tunes performed in that regard.
Radical Tunes scores
To measure the learning outcomes in our experiment, we used the relative improvement between the scores our participants received during the pre- and post-tests on the meaning and stroke order of the six characters (Figure 6a,b).

4.2. Results

We performed Levene’s test for the equality of variances. We detected no unequal variances for any of our outcome variables (all p’s > 0.052). Because of that, we assumed that it was safe to use the parametric tests.

4.2.1. Prior Knowledge and Experience

A number of independent samples t-tests showed that the subjects across the two conditions did not significantly differ in regards to self-reported video game and rhythm game experience; and on the pre-test of the learning outcome measures (Table 1 and Table 3).

Consequently, we can assume that subjects in both conditions had similar prior game experience and knowledge of kanji for the following analyses.

4.2.2. Immersion Experience

We used an independent samples t-test to assess the differences between IEQ scores of the two groups. The descriptive statistics of the IEQ scores, along with the significant differences and effect sizes between conditions can be found in Table 2. The results revealed a significant difference in favor of the musical version of Radical Tunes, with increased overall IEQ scores (p = 0.040, r = 0.41), cognitive involvement (p = 0.026, r = 0.44), and emotional involvement (p = 0.034, r = 0.42). Real-world dissociation, challenge and control did not show any significant differences (all p’s > 0.063).

4.2.3. Learning Outcomes

In order to understand the players’ learning outcomes, we examined the scores of the meaning and stroke order pre- and post-test, as well as the final scores from the game portion of Radical Tunes. Table 3 lists the descriptive statistics of these measures. A series of independent samples t-tests determined that both groups significantly improved in both the meaning and the stroke order scores from pre- to post-test: control (meaning—p < 0.001, r = 0.95; stroke order—p < 0.001, r = 0.96), and music (meaning—p < 0.001, r = 0.92; stroke order—p < 0.001, r = 0.94). This suggests that Radical Tunes succeeded in teaching kanji meanings and writing stroke order to the players, at least in the short term. That said, we did not observe any significant differences in improvement between the two groups across any of the learning measurements (all p’s > 0.11). As suggested by our second study, the reason was likely the small number of kanji included in the experiment, which did not provide sufficient difficulty or time delay for the music mnemonic approach to significantly impact the learning outcomes.

4.2.4. Perception of Usefulness

We asked our subjects to rate the extent to which they would recommend Radical Tunes to someone learning Japanese or Chinese on a 7-point Likert scale (1—not at all to 7—very likely). The music condition had notably higher scores (

μ

= 6.58,

σ

= 0.669) than the control condition (

μ

= 5.42,

σ

= 1.782). An independent samples t-test confirmed a significantly higher likelihood of musical version players recommending the game (p = 0.045, t = 2.124, r = 0.4).

5. Study 2

5.1. Rationale

With our second study, we wanted to address the limitations of the Radical Tunes pilot by expanding the learning content, increasing the pool of participants and adding a longitudinal post-test that the participants were encouraged to take approximately 24 h after the completion of the learning task. Additionally, since music improved immersion in the pilot study, we decided to add music to both versions of our app in this study. We used procedural generation to create ever-changing melodies for our control group and hand authored stroke-specific melodies for our treatment group. This was performed so that both groups could benefit from the music’s immersive properties (or be equally hindered by the increased cognitive load) while only the treatment group was exposed to radical-specific mnemonic melodies.

5.2. Methods

5.2.1. Participants

A total of 69 participants completed the first part of the study. They were recruited via Amazon’s Mechanical Turk and were randomly assigned to either the control or treatment group. The participants were provided a link to an Android version of the app and given instructions to not take any notes and only complete the study once. This was done to ensure that each participant received an equal amount of instruction and review. Of the 69 first-round participants, 47 returned for the follow-up test. Of these, two were eliminated for admitting to prior knowledge of kanji, and one for completing step one of the study twice. Using the inter-quartile range test for outliers, we identified and removed two mild lower outliers on the stroke knowledge score, leaving us with the final count of 22 and 20 people in control and treatment groups, respectively. There were 28 male, 13 female and 1 participant who chose not to disclose their gender. The ages ranged from 18 to 50. There were no statistically significant differences in age and gender breakdown across the two groups, with control group age (

μ

= 33.05,

σ

= 9.810), treatment group age (

μ

= 35.25,

σ

= 8.705); control gender 0.64% male, treatment gender 0.70% male; and p-values at 0.448 and 0.671, respectively. Despite the large variability in age among the participants, age did not appear to be a significant predictor for any of our outcome variables (all p > 0.06).

5.2.2. Procedure

The participants downloaded and installed either a control or treatment version of the app. Then, within the app, they provided basic demographic data, including their age, gender and level of experience with Japanese/Chinese characters. After that, they took a pre-test on their knowledge of the meaning of the 15 kanji selected for the study (Figure 7). The participants could not be expected to draw the characters even approximately before having learned them, so the meaning test was the only pre-test administered.

The learning part of the experiment consisted of three steps for each kanji. First, the participants would watch an animated demo of how the kanji was meant to be drawn. Then, they would practice drawing the character twice by drawing over the dotted lines, one stroke at a time. Finally, they would practice drawing the character three times without dotted guidelines (a hint would appear if the participant attempted to draw the wrong stroke several times in a row). During the demo and each instance of drawing of the kanji, a melody would play, a small part of it corresponding to each stroke. In the treatment group, the melody would always be the same, while for the control group, the melody would change every time.

After the learning part was complete, the participants were instructed to draw the 15 kanji. For this, they were told the meaning of the kanji, and the number of strokes it had (Figure 7, right). Both groups could use the “undo” button to make corrections to their drawing before clicking “submit.” The treatment group had to listen to the kanji’s melody before attempting to write it, and they could listen to the melody as many times as they liked. We made the stroke number explicitly known on the screen to make up for the fact that the melodic reminder in the treatment group could well give a clue as to how many strokes were in a kanji. Then, the participants were given the meaning post-test, identical to the pre-test.

The participants were encouraged to take the follow-up test in the following days. The follow-up test was identical to the post-test, consisting of the kanji writing followed by the meaning test.

One of the limitations of our study was that the Mechanical Turk provided limited means for controlling when the participants would return, and there was significant variance in the interval between the first session and the follow-up test, with the control group taking on average 14 h longer to take the test with p = 0.03. However, this did not have an impact on the study results (see Figure 8).

5.2.3. Kanji

For this study, we selected six common radicals and nine kanji that consist of combinations of those radicals for a total of 15 characters (Figure 9). In the treatment group, each radical was accompanied by a designated melody each time it appeared.

5.2.4. Measures and Scoring

Our main objective was to determine whether music could be used as a mnemonic tool to help students acquire stroke knowledge. Stroke knowledge consists of stroke order and stroke direction. Thus, each character was scored on two aspects: the number of strokes drawn in the correct order and the number of strokes drawn in the correct direction. We also collected pre-, post- and longitudinal test scores for the meaning of characters. Even though we did not expect music to affect meaning retention, we were surprised to see a statistically significant advantage in the treatment group. We discuss this unexpected finding in the Discussion section.

We manually calculated the scores for the stroke knowledge by looking at the images that the participants drew during the post- and longitudinal tests and counting the number of strokes drawn in the correct order and direction (Figure 10).

5.3. Results

We observed a number of statistically significant differences in learning outcomes between the two groups, both for remembering the meaning (Table 4) and stroke knowledge (Table 5) of the characters.

With respect to learning kanji meaning, the treatment group performed better on the meaning post- and longitudinal tests (p’s < 0.05) with a medium effect size (g’s > 0.6). For ease of comprehension, we combined the stroke knowledge scores for characters with 2–6 strokes and 7+ strokes (respectively, 8 and 7 total kanji in our experiment). The full list of individual kanji scores are in the Appendix (Table A1 and Table A2). The differences between the two groups appear correlated with the complexity of the kanji, with the differences becoming more pronounced for characters with a higher stroke count, at least in the short term (see Table 5). In Jōyō kanji, over 84% have 7+ strokes. Thus, while our results are mixed, the positive effect on complex kanji is more important than the lack of a perceptible effect in simpler kanji.

Our results showed that consistent melodies indeed helped the participants remember the characters’ stroke composition better, particularly when it came to stroke direction and characters with many strokes in them. It is possible that the simpler characters with fewer strokes were easy enough to remember even without music, and that is why the difference in scores was not as pronounced there.

6. Discussion

6.1. Music and Immersion

The first study showed that the music condition of Radical Tunes significantly increased the overall IEQ scores, cognitive involvement and emotional involvement, though no significant differences were found for real-world dissociation, challenge or control (see Table 2). Notably, one would expect the challenge and control factors to be similar, as the game controls and tasks were identical across the two conditions. Overall, this suggests that the addition of music to Radical Tunes—and likely SLA games in general—can positively affect player factors (cognitive involvement and emotional involvement) but not the game factors (challenge and control) of immersion [43]. Furthermore, these findings fall in line with prior studies reporting that adding pleasing music can increase immersion [12].

6.2. The Meaning Mystery

The second study was meant to explore whether music could help people remember the stroke order and direction of several Japanese kanji. However, we found that the treatment group performed better on the meaning tests as well. We have a possible explanation. During the early testing of our app, we found that people tended not to look at the meaning of the character while going through the learning section of the app, and would consequently not learn any of the meanings even though they learned to write the characters. To help with this, we added a recorded spoken meaning that played each time after the last stroke of a character was drawn during the learning phase. For the treatment group, this meant that the spoken meaning was always attached to the unique melody assigned to that character, while the control group heard the meaning attached to a new melody every time. A study by Wallace et al. showed that accompanying text with a repeated melody helped participants memorize text better than when a variety of melodies was used [10]. We hypothesized that the treatment group benefited from the same effect when it came to the spoken meaning.

6.3. Scalability

While there are thousands upon thousands of logographic characters, they all consist of a finite number of radicals (approximately 80) and even fewer stroke types. It is therefore feasible for a human musician to compose unique melodic elements either for the individual strokes or radicals. An algorithm could then combine them (with pitch adjustment, where needed) based on the composition of each kanji to generate a unique melody for it. As a result, this work presents the groundwork for the design and creation of a scalable approach towards creating music for an otherwise intractable number of characters.

6.4. Implications for Design

Our study results illustrate that music has the potential to serve as a useful mnemonic tool in stroke knowledge acquisition, with increases in learner immersion and perceptions of usefulness. This provides initial evidence for the value of incorporating consistent melodies into the design of logographic learning software. Our results also showed that the mnemonic effect was more pronounced in characters with a greater number of strokes. This suggests that a musical mnemonic design is particularly useful for learning more complex logographic characters, while other memorization techniques, such as repetition, may serve learning simple characters well enough. This is not to say that a musical mnemonic design approach is not beneficial at an introductory level, however, since even among the characters that are taught early on there are some with 9 strokes and more (e.g., 前 “before,” which is part of the word “name,” has 9 strokes, and 時 “time” has 10). Therefore, the musical mnemonic is not something which is only useful for advanced learners.

7. Limitations and Future Work

Both studies were limited by a relatively low number of subjects (24 and 42, respectively), which could raise the question of how generalizable our results are; however, the Levene’s tests did not expose any significant variances in our output variables, and variability in participants’ age did not appear to be a significant predictor of the outcomes. With this, we drew our conclusions under the assumption that our samples were representative of the larger populations (in the case of the second study, of the larger population of Mechanical Turk users). Nonetheless, a greater confidence in the results could be gained from replicating the studies with larger, more representative samples. The two primary limitations of the second study were with the chosen platform for participant recruitment and the music. Amazon’s Mechanical Turk is a platform where one can hire workers to perform remote tasks for a set reward. The workers are incentivized to complete tasks as quickly as possible to maximize their profit, and that sometimes leads to sloppy performance. The evidence for this is some participants’ unwillingness to use the “undo” button to correct a clearly accidentally misplaced stroke (see the white stroke on the lower right of Figure 10). Because we were not able to observe our participants, we cannot be sure of the diligence with which they performed the task.

As for our second limitation, we used procedurally produced sound in our app to create a responsive musical experience. Without a designated sound generation expert on our team, we were limited in the variety of sounds we could generate. While pre-recorded sound could offer infinite variety, we lacked the expertise to make it play in accordance with the speed and direction of the stroke, the way we could with procedurally produced sound. Additionally, we should have taken more care in designing melodies that were clearly distinct from each other. If, as some informal feedback suggested, many of our melodies sounded too similar, that would certainly dampen the music’s mnemonic effect. Future work could explore different approaches to music creation and the effects they have on the melody’s memorability.

In our study, we made a point of using kanji consisting of a small subset of radicals. Future work could explore whether music is equally effective for memorizing characters with no common elements.

8. Conclusions

In this article, we described Radical Tunes—a musical game for learning to write Japanese characters. The game employs unique melodies to help with memorization of stroke order. We also presented two evaluation studies of Radical Tunes. The first compared the musical version of Radical Tunes with one that replaced melodies with the sound of chalk on a blackboard. The second compared a version of Radical Tunes with melodies consistent with a version with procedurally generated ever-changing melodies.

With the help of the Immersive Experience Questionnaire, we confirmed that the presence of music in Radical Tunes significantly improved player immersion. The fact that subjects from both conditions significantly improved their scores the between pre- and post-tests indicates that, at least in the short-term, Radical Tunes proved effective at teaching kanji to the players. The limited scope of our pilot study—only six characters—did not expose any mnemonic effects the music may have on the players’ ability to retain the material long term. However, our follow-up study showed that exposure to consistent melodies lead to significantly higher post-test results, particularly in kanji with a higher number of strokes. We believe that if designed and implemented well, the ideas described in this paper could make a positive difference for the millions of novice learners of LWS.

Author Contributions

Conceptualization, O.G.K.; methodology, O.G.K. and E.F.M.; software, O.G.K.; validation, O.G.K. and E.F.M.; formal analysis, O.G.K. and E.F.M.; resources, O.G.K.; data curation, O.G.K. and E.F.M.; writing—original draft preparation, O.G.K.; writing—review and editing, E.F.M.; visualization, O.G.K. and E.F.M.; supervision, E.F.M.; project administration, E.F.M.; funding acquisition, O.G.K. and E.F.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The Office of Research Compliance Administration has determined that the human subjects research protocol referenced above meets the criteria for exemption described in 45 CFR 46.104 and/or the UCSC Policy on IRB Regulatory Flexibility under the following category(ies): Exempt 3x.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The anonymized data collected in the course of these studies can be found online at this address (accessed 14 December 2021): https://drive.google.com/drive/u/0/folders/1_HSvm6HStINGbQ8Qu4LNn4w0YKMspwZh.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

LWS	Logographic Writing System
SLA	Second Language Acquisition
IEQ	Immersive Experience Questionnaire

Appendix A

Table A1. Descriptive statistics of the stroke knowledge scores.

Post-Test	Control		Treatment		Significance
Stroke Knowledge Scores	$μ$	$σ$	$μ$	$σ$	p (t)
十 Order	0.640	0.492	0.550	0.510	0.581 (0.557)
十 Direction	0.932	0.234	10.00	0.000	0.186 (−1.368)
口 Order	0.803	0.394	0.933	0.205	0.183 (−1.362)
口 Direction	0.788	0.394	0.933	0.205	0.137 (−1.525)
氵 Order	0.860	0.351	0.900	0.308	0.723 (−0.358)
氵 Direction	0.758	0.392	0.867	0.332	0.332 (−0.983)
日 Order	0.886	0.296	0.825	0.364	0.555 (0.596)
日 Direction	0.886	0.296	0.825	0.364	0.555 (0.596)
月 Order	0.950	0.213	0.950	0.224	0.947 (0.067)
月 Direction	0.921	0.236	0.900	0.235	0.780 (0.281)
木 Order	0.773	0.369	0.925	0.245	0.121 (−1.588)
木 Direction	0.864	0.316	0.963	0.122	0.184 (−1.361)
古 Order	0.709	0.384	0.820	0.330	0.321 (−1.006)
古 Direction	0.818	0.332	0.860	0.325	0.682 (−0.412)
回 Order	0.583	0.445	0.608	0.398	0.849 (−0.192)
回 Direction	0.598	0.444	0.642	0.406	0.744 (−0.329)
困 Order	0.528	0.433	0.557	0.376	0.819 (−0.230)
困 Direction	0.591	0.424	0.621	0.391	0.809 (−0.243)
汨 Order	0.461	0.427	0.700	0.379	0.062 (−1.920)
汨 Direction	0.435	0.408	0.693	0.374	0.039 (−2.136)
沐 Order	0.597	0.349	0.771	0.291	0.086 (−1.761)
沐 Direction	0.558	0.352	0.814	0.301	0.015 (−2.537)
明 Order	0.750	0.366	0.794	0.357	0.697 (−0.392)
明 Direction	0.756	0.315	0.794	0.323	0.701 (−0.387)
沽 Order	0.295	0.383	0.581	0.443	0.032 (−2.227)
沽 Direction	0.386	0.404	0.625	0.441	0.076 (−1.823)
枯 Order	0.591	0.385	0.783	0.313	0.082 (−1.782)
枯 Direction	0.646	0.328	0.822	0.289	0.073 (−1.844)
胡 Order	0.348	0.340	0.636	0.400	0.017 (−2.497)
胡 Direction	0.394	0.358	0.669	0.348	0.016 (−2.524)

Table A2. Descriptive statistics of the stroke knowledge scores.

Longitudinal Test	Control		Treatment		Significance
Stroke Knowledge Scores	$μ$	$σ$	$μ$	$σ$	p (t)
十 Order	0.770	0.429	0.650	0.489	0.395 (0.861)
十 Direction	0.950	0.213	10.00	0.000	0.329 (−1.000)
口 Order	0.864	0.351	0.950	0.163	0.308 (−10.037)
口 Direction	0.848	0.352	0.900	0.219	0.569 (−0.575)
氵 Order	0.820	0.395	0.950	0.224	0.187 (−1.346)
氵 Direction	0.742	0.411	0.950	0.224	0.047 (−2.059)
日 Order	0.807	0.377	0.950	0.224	0.140 (−1.511)
日 Direction	0.796	0.375	0.938	0.228	0.143 (−1.498)
月 Order	0.864	0.316	10.00	0.000	0.056 (−2.027)
月 Direction	0.818	0.329	0.938	0.111	0.121 (−1.604)
木 Order	0.818	00.329	0.950	0.154	0.102 (−1.687)
木 Direction	0.898	0.252	0.988	0.056	0.117 (−1.628)
古 Order	0.745	0.376	0.890	0.200	0.125 (−1.574)
古 Direction	0.791	0.373	0.950	0.157	0.078 (−1.831)
回 Order	0.621	0.386	0.717	0.394	0.433 (−0.792)
回 Direction	0.629	0.363	0.742	0.388	0.338 (−0.970)
困 Order	0.583	0.451	0.585	0.428	0.988 (−00.015)
困 Direction	0.634	0.435	0.692	0.373	0.644 (−0.465)
汨 Order	0.617	0.414	0.721	0.386	0.403 (−0.846)
汨 Direction	0.656	0.365	0.714	0.382	0.616 (−0.505)
沐 Order	0.747	0.344	0.829	0.215	0.357 (−0.932)
沐 Direction	0.747	0.330	0.921	0.182	0.039 (−2.150)
明 Order	0.767	0.367	0.888	0.222	0.202 (−10.301)
明 Direction	0.744	0.357	0.856	0.230	0.231 (−1.219)
沽 Order	0.614	0.393	0.681	0.402	0.586 (−0.550)
沽 Direction	0.635	0.381	0.800	0.343	0.147 (−1.479)
枯 Order	0.626	0.390	0.617	0.370	0.935 (0.082)
枯 Direction	0.757	0.321	0.728	0.354	0.777 (0.285)
胡 Order	0.396	0.399	0.583	0.419	0.147 (−1.478)
胡 Direction	0.416	0.431	0.672	0.370	0.045 (−2.071)

References

Eberhard, D.M.; Simons, G.F.; Fennig, C.D. Ethnologue: Languages of the World, Twenty-Fourth Edition. Dallas, Texas: SIL International. Available online: http://www.ethnologue.com (accessed on 9 November 2021).
Bak, T.H.; Nissan, J.J.; Allerhand, M.M.; Deary, I.J. Does bilingualism influence cognitive aging? Ann. Neurol. 2014, 75, 959–963. [Google Scholar] [CrossRef]
Rose, H. Unique challenges of learning to write in the Japanese writing system. L2 Writ. Engl. 2019, 5, 78–94. [Google Scholar]
List of Jōyō Kanji. Agency of Cultural Affairs, Government of Japan. 2010. Available online: https://www.bunka.go.jp/kokugo_nihongo/sisaku/joho/joho/kijun/naikaku/pdf/joyokanjihyo_20101130.pdf (accessed on 9 November 2021).
Chuang, H.Y. Mobile assisted language learning APPs for the Chinese classroom. JTCLT 2016, 7, 113–119. [Google Scholar]
Mason, A.; Zhang, W. An exploration of the use of mobile applications to support the learning of Chinese characters employed by students of Chinese as a foreign language. In Beyond the Language Classroom: Researching MOOCs and Other Innovations; Research-publishing.net: Voillans, France, 2017; pp. 99–112. [Google Scholar]
Ch’ng, L.C.; Ting, H.L.; Chuah, K.M. Common stroke mistakes in Chinese character writing among Business Administrative students. AJoBSS 2018, 2, 1–10. [Google Scholar]
Lo, L.Y.; Yeung, P.S.; Ho, C.S.; Chan, D.W.; Chung, K. The role of stroke knowledge in reading and spelling in Chinese. J. Res. Read. 2016, 39, 367–388. [Google Scholar] [CrossRef]
Rowley, M. Kanji Pict-o-Graphix: Over 1000 Japanese Kanji and Kana Mnemonics; Stone Bridge Press: Berkeley, CA, USA, 1992. [Google Scholar]
Wallace, W. Memory for music: Effect of melody on recall of text. J. Exp. Psychol. Learn. Mem. Cogn. 1994, 20, 1471–1485. [Google Scholar] [CrossRef]
Keehl, O.K.; Melcer, E.F. Radical tunes: Exploring the impact of music on memorization of stroke order in logographic writing systems. In Proceedings of the 14th International Conference on the Foundations of Digital Games (FDG ’19), New York, NY, USA, 26–30 August 2019; pp. 1–6. [Google Scholar]
Sanders, T.; Cairns, P. Time perception, immersion and music in videogames. In Proceedings of the 24th BCS Interaction Specialist Group Conference, Dundee, UK, 6–10 September 2010; pp. 160–167. [Google Scholar]
Kallinen, K. The effects of background music on using a pocket computer in a cafeteria: Immersion, emotional responses, and social richness of medium. In Extended Abstracts on Human Factors in Computing Systems, Proceedings of CHI; ACM: New York, NY, USA, 2004. [Google Scholar]
Cairns, P.; Cox, A. Immersion in digital games: Review of gaming experience research. In Handbook of Digital Games; Wiley-Blackwell: Hoboken, NJ, USA, 2014. [Google Scholar]
Dede, C. Immersive interfaces for engagement and learning. Science 2009, 323, 66–69. [Google Scholar] [CrossRef] [Green Version]
Cheng, M.T.; She, H.C.; Annetta, L.A. Game immersion experience: Its hierarchical structure and impact on game-based science learning. J. Comput. Assist. Learn. 2015, 31, 232–253. [Google Scholar] [CrossRef]
Li, Z.; Cheng, M.; Maesako, T. A Bibliometrical Analysis of Comparative Studies on Chinese Hanzi and Japanese Kanji Characters. Scott. J. Arts Soc. Sci. Sci. Stud. 2013, 16, 3–11. [Google Scholar]
Librenjak, S.; Vučković, K.; Dovedan, Z. Multimedia assisted learning of Japanese kanji characters. In Proceedings of the 35th International Convention MIPRO, Opatija, Croatia, 21–25 May 2012; pp. 1284–1289. [Google Scholar]
Zhang, H. A review of stroke order in hanzi handwriting. Lang. Learn. High Educ. 2014, 4, 423–440. [Google Scholar] [CrossRef]
Li, X. Investigation of teaching stroke order of Chinese characters in beginner CSL learners. Lang. Teach. Linguist. Stud. 2006, 23–25. [Google Scholar]
Hallam, S.; Price, J.; Katsarou, G. The Effects of Background Music on Primary School Pupils’ Task Performance. Educ. Stud. 2002, 28, 111–122. [Google Scholar] [CrossRef]
Cheng, A.; Yang, L.; Andersen, E. Teaching language and culture with a virtual reality game. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, 6–11 May 2017; pp. 541–549. [Google Scholar]
Ludke, K.; Ferreira, F.; Overy, K. Singing can facilitate foreign language learning. Mem. Cogn. 2013, 7, 41–52. [Google Scholar] [CrossRef] [PubMed]
Werner, R. Music, movement and memory: Pedagogical songs as mnemonic aids. TESOL J. 2018, 9, 1–11. [Google Scholar] [CrossRef]
Ezzeldin, A.M.; Ashraf, A.H. Review of mnemonic devices and their applications in cardiothoracic surgery. J. Egypt. Soc. Cardio-Thorac. Surg. 2017, 25, 79–90. [Google Scholar]
Moussard, A.; Bigand, E.; Belleville, S.; Peretz, I. Music as a Mnemonic to Learn Gesture Sequences in Normal Aging and Alzheimer’s Disease. Front. Hum. Neurosci. 2014, 8, 294. [Google Scholar] [CrossRef] [Green Version]
Hebert, C. Movement Memory: How We Learn, Retain and Remember Dance. The Dance Current, Canada’s Dance Magazine. 2017. Available online: https://www.thedancecurrent.com/feature/movement-memory (accessed on 6 September 2020).
Roberson, M. Dance Memory: Studying how the mind remembers physical movement. Duke Univ. 2010. Available online: https://research.duke.edu/dance-memory (accessed on 6 September 2020).
Yap, K.; Zheng, C.; Tay, A.; Yen, C.; Do, E.Y. Word out: Learning the Alphabet through Full Body Interactions. In Proceedings of the 6th Augmented Human International Conference, Singapore, 9–11 March 2015; ACM: New York, NY, USA, 2015; pp. 101–108. [Google Scholar]
Edge, D.; Cheng, K.; Whitney, M. SpatialEase: Learning Language through Body Motion. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, 27 April–2 May 2013; ACM: New York, NY, USA, 2013. [Google Scholar]
Connolly, T.; Stansfield, M.; Hainey, T. An Alternate Reality Game for Language Learning: ARGuing for Multilingual Motivation. Comput. Educ. 2011, 57, 1389–1415. [Google Scholar] [CrossRef]
Liu, Y.; Holden, D.; Zheng, D. Analyzing students’ Language Learning Experience in an Augmented Reality Mobile Game: An Exploration of an Emergent Learning Environment. Procedia Soc. Behav. Sci. 2016, 228, 369–374. [Google Scholar] [CrossRef] [Green Version]
Syson, M.B.; Estuar, M.R.; See, K.T. Abkd: Multimodal mobile language game for collaborative learning of chinese hanzi and japanese kanji characters. In Proceedings of the 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology, Macau, China, 4–7 December 2012; Volume 3, pp. 311–315. [Google Scholar]
Head, A.; Xu, Y.; Wang, J. Tonewars: Connecting language learners and native speakers through collaborative mobile games. In Proceedings of the International Conference on Intelligent Tutoring Systems, Honolulu, HI, USA, 5–9 June 2014; pp. 368–377. [Google Scholar]
Fan, X.; Luo, W.; Wang, J. Mastery Learning of Second Language through Asynchronous Modeling of Native Speakers in a Collaborative Mobile Game. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), New York, NY, USA, 6–11 May 2017; pp. 4887–4898. [Google Scholar]
deHaan, J.; Reed, W.; Kuwanda, K. The effect of interactivity with a music video game on second language vocabulary recall. Lang. Learn. Technol. 2010, 14, 74–94. [Google Scholar]
Kumar, A.; Reddy, P.; Tewari, A.; Agrawal, R.; Kam, M. Improving literacy in developing countries using speech recognition-supported games on mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12), Austin, TX, USA, 5–10 May 2012; pp. 1149–1158. [Google Scholar]
Nushi, M.; Shafeie, S.; Shafiei, M. Game-based Teaching of Stress Placement on Multi-syllabic English Words. J. Teach. Engl. Lang. Stud. 2017, 1, 46–64. [Google Scholar]
Lakoff, G.; Johnson, M. Metaphors We Live by; University of Chicago Press: Chicago, IL, USA, 2008. [Google Scholar]
Rohrer, T. The body in space: Dimensions of embodiment. Body Lang. Mind 2007, 1, 339–377. [Google Scholar]
Antle, A.N.; Corness, G.; Droumeva, M. What the body knows: Exploring the benefits of embodied metaphors in hybrid physical digital environments. Interact. Comput. 2009, 21, 66–75. [Google Scholar] [CrossRef]
Antle, A.N.; Droumeva, M.; Corness, G. Playing with the sound maker: Do embodied metaphors help children learn? In Proceedings of the 7th International Conference on Interaction Design and Children, Chicago, IL, USA, 11–13 June 2008; pp. 178–185. [Google Scholar]
Jennett, C.; Cox, A.; Cairns, P.; Dhoparee, S.; Epps, A.; Tijs, T.; Walton, A. Measuring and Defining the Experience of Immersion in Games. Int. J. Hum.-Comput. Stud. 2008, 9, 641–661. [Google Scholar] [CrossRef]
Denisova, A.; Nordin, I.; Cairns, P. The Convergence of Player Experience Questionnaires. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY ’16, Austin, TX, USA, 16–19 October 2016; pp. 33–37. [Google Scholar]

Figure 1. A visual mnemonic for the meaning of kanji “body” from “Kanji Pict-o-Graphix,” a popular book for learning kanji with mnemonics [9].

Figure 2. Two examples from WaniKani, a popular kanji learning website. On the left is a pronunciation mnemonic for the kanji 二, when it is part of the word 二つ, meaning “two things.” On the right is a good example of WaniKani’s shock-value-driven mnemonic. The humorously named “poop” radical is apt to stick out in a student’s mind.

Figure 3. The anatomy of kanji. Each kanji consists of one or more radicals. In turn, each radical consists of one or more strokes.

Figure 4. Melodies for the radicals “tree” (above) and “water” (below). We tried to make the pitch of the melody follow the direction of the strokes. A horizontal stroke is represented by a steady note. A long downward stroke is accompanied by a significant drop in pitch, while a shorter upward stroke is matched by a moderate rise in pitch.

Figure 5. The list of radicals and kanji used in our first study. From left to right: person, mouth, tree, rest, episode, annoyed.

Figure 6. Radical Tunes screenshots: (a) meaning pre- and post-test; (b) stroke order pre- and post-test; (c) stroke learning screen; and (d) timed and scored game portion.

Figure 7. Screenshots of our app. Left: the meaning test screen. Middle: the learning screen. The dotted line indicates where the next stroke should be drawn, with the biggest dot indicating the beginning of the stroke. Right: the writing test screen. The second line of text displays how many strokes the target kanji has and how many the participants already drew. The “listen” button in the treatment condition would play the kanji’s melody. The button is absent in the control version of the app.

Figure 8. The scores for meaning retention (calculated by subtraction of post-test meaning scores from the longitudinal test scores) plotted against the time that passed between the two tests. The linear fit function shows no negative correlation between the length of time and the changes in scores.

Figure 9. The 15 characters selected for our study. All of the kanji consist of some combination of the six common radicals (highlighted).

Figure 10. Two of the post-test images for the kanji “foreign.” For scoring purposes, the color indicates the order in which the strokes were drawn, while the large initial dot indicates the direction of the stroke. The participant on the left drew the strokes in the correct order, but the first stroke of the moon radical (purple) was drawn in the wrong direction. The participant on the right drew all the strokes in the correct direction (with the exception of the missing last stroke), but switched the order of the first two strokes in the “ten” radical. The participant on the left received 9/9 on order and 8/9 on direction. The participant on the right received 6/9 for order and 8/9 for direction.

Table 1. Self-reported video game and rhythm game experience levels.

	Control		Music		Significance
Measure	$μ$	$σ$	$μ$	$σ$	p (t)
Video Game Experience	1.33	0.651	1.50	0.905	0.610 (0.235)
Rhythm Game Experience	2.00	0.853	2.08	0.793	0.807 (0.248)

Table 2. IEQ mean scores, standard deviations, significant differences between the two conditions, and the effect size, which is in the medium to large range for significant differences.

	Control		Music		Significance	Effect Size
IEQ Measures	$μ$	$σ$	$μ$	$σ$	p (t)	r
IEQ Overall	100.33	18.042	115.25	15.316	0.040 (2.183)	0.41
Other Cognitive Involvement	30.08	7.012	36.08	5.125	0.026 (2.393)	0.44
Real World Dissociation	25.58	6.487	28.50	5.808	0.258 (1.160)	-
Emotional Involvement	12.33	3.551	15.58	3.476	0.034 (2.266)	0.42
Challenge	16.33	1.497	17.67	1.826	0.063 (1.956)	-
Control	16.00	3.542	17.42	3.147	0.312 (1.036)	-

Table 3. Descriptive statistics of learning outcomes (meaning tests, stroke order tests, and in-game scores).

	Control		Music		Significance
Learning Outcome Measures	$μ$	$σ$	$μ$	$σ$	p (t)
Meaning Pre-Test	1.08	0.900	1.17	0.835	0.816 (0.235)
Meaning Post-Test	5.50	1.000	5.58	0.669	0.813 (0.240)
Meaning Improvement	+4.42	1.730	+4.42	1.165	1.00 (0.000)
Stroke Order Pre-Test	0.33	0.651	0.58	0.515	0.308 (1.043)
Stroke Order Post-Test	5.42	0.900	5.67	1.155	0.560 (0.591)
Stroke Order Improvement	+5.08	0.900	+5.08	1.379	1.00 (0.000)
In-Game Score	102.17	8.932	106.58	2.193	0.121 (1.663)

Table 4. Descriptive statistics of the meaning tests.

	Control		Treatment		Significance	Effect Size
Results for the Meaning Tests	$μ$	$σ$	$μ$	$σ$	p (t)
Pre-Test	2.86	2.232	3.50	2.965	0.441 (−0.780)	-
Post-Test	10.23	4.309	12.60	2.741	0.038 (−2.148)	0.65
Longitudinal Test	10.86	4.223	13.00	2.128	0.044 (−2.098)	0.63

Table 5. Descriptive statistics of the stroke knowledge scores.

	Control		Treatment		Significance	Effect Size
Results for the Stroke Knowledge Tests	$μ$	$σ$	$μ$	$σ$	p (t)
<7 Strokes Kanji Post-Test Order	0.776	0.240	0.814	0.191	0.573 (−0.568)	-
7+ Strokes Kanji Post-Test Order	0.510	0.243	0.689	0.248	0.023 (−2.357)	0.73
<7 Strokes Kanji Post-Test Direction	0.821	0.204	0.874	0.161	0.353 (−0.940)	-
7+ Strokes Kanji Post-Test Direction	0.538	0.237	0.720	0.241	0.018 (-20.459)	0.75
<7 Strokes Kanji Longitudinal Order	0.789	0.242	0.882	0.141	0.132 (−1.543)	-
7+ Strokes Kanji Longitudinal Order	0.621	0.239	0.701	0.215	0.266 (−1.128)	-
<7 Strokes Kanji Longitudinal Direction	0.810	0.206	0.926	0.132	0.035 (−2.191)	0.69
7+ Strokes Kanji Longitudinal Direction	0.656	0.227	0.769	0.191	0.086 (−1.759)	-

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Keehl, O.G.; Melcer, E.F. Music to My Ears: Developing Kanji Stroke Knowledge through an Educational Music Game. Multimodal Technol. Interact. 2021, 5, 83. https://doi.org/10.3390/mti5120083

AMA Style

Keehl OG, Melcer EF. Music to My Ears: Developing Kanji Stroke Knowledge through an Educational Music Game. Multimodal Technologies and Interaction. 2021; 5(12):83. https://doi.org/10.3390/mti5120083

Chicago/Turabian Style

Keehl, Oleksandra G., and Edward F. Melcer. 2021. "Music to My Ears: Developing Kanji Stroke Knowledge through an Educational Music Game" Multimodal Technologies and Interaction 5, no. 12: 83. https://doi.org/10.3390/mti5120083

APA Style

Keehl, O. G., & Melcer, E. F. (2021). Music to My Ears: Developing Kanji Stroke Knowledge through an Educational Music Game. Multimodal Technologies and Interaction, 5(12), 83. https://doi.org/10.3390/mti5120083

Article Menu

Music to My Ears: Developing Kanji Stroke Knowledge through an Educational Music Game †