Player Behavior Influence by Visualizing the Game Sound Landscape

There was a time of human behavioral disruption, particularly during the period of digital convergence, when all types of media migrated from their different physical support to a unique computational platform, digitally based. In this way, mostly visual (space-based), audio (time-based) media; and the audiovisual (space-time based) discontinued to inhabit their traditional physical devices and began to be created or converted to the digital universe. Digital games were also influenced by this change, as they are entirely based on space, time and space-time media, increased by interactivity.

Ruled by their own rules, games create their universe. For Huizinga [6] “any thinking person can see at a glance that play is a thing on its own, even if this language possesses no general concept to express it”. Since the digital systematization of the games, new experiences could be designed, redefining the behavior and the relationship of the player to the game. The experiences created by games can outweigh the player’s interest in the real world. As McGonigal [9] explains, the real world just does not offer up as easily the carefully designed pleasures, the thrilling challenges, and the powerful social bonding afforded by virtual environments.

Technological developments have brought improvements to accessibility issues. People, who were once marginalized, find themselves increasingly immersed in the media that were previously difficult to access. But this evolution is not perfect and is still restrictive in some cases, moving away and segregating diverse groups. Such an event can be seen in games, where universes are created, but people are excluded because some interactions are not accessible to all people. Some problems faced by the deaf are often ignored by the listener. Either by the lack of knowledge of the pains of these people or by not being able to understand their difficulties.

Game designers have explored new possibilities for interaction, creating interactive mechanics that make use of different media. There are cases where visuals and audio impact the formal structure of a game and should be considered part of the rules [14]. However, players with visual or sound limitations often have their experience impaired, because the designer has not taken due care of the accessibility. The visual restrictions, in this case, are more critical, since a video game, as the name shows, is a game used through a video. So, depending on the visual restraint level, it may be impractical for the disabled to play the game. On the other hand, individuals who are hearing impaired, even on severe levels, can play a video game, but their experience is also limited and yet made complicated by the lack of accessibility. It happens because sound art is usually included in the game as background music or sound effects. The audio itself is a relevant part of the game experience and can be a decisive factor in situations an element is heard but is out of view field. Audio is an essential component of aesthetics and sound effects help to tell the story too [15].

The pursuit of entertainment through games by deaf users can be somewhat painful. These people usually need to adapt to an interface, rather than the other way around. Since games hardly offer tools for them to be fully exploited by a deaf person. Taking this into account, there is a latent demand for a feature that solves problems encountered by deaf people while playing. There is an opportunity to create something that softens these issues while bringing visibility to a recurring and often overlooked problem.

In this sense, the objective of this research is to influence the player behavior by enhancing the experience of digital games for the hearing impaired, so they can act in the universe of the video game with similar or same potential of other players. By offering access to the hearing impaired, it is expected that the player will have the ability to own, within the universe of the game, the same behavior as a fully trained player. The improved game experience can extrapolate the sound limits of the interaction, allowing the designer’s dynamics to be fully enjoyed by the player.

This project aims to improve the behavioral experience for deaf gamers. To achieve this objective, the research is based on the visualization of the sound landscape through a radar that allows the audio of the game to be represented visually in real time, allowing the player to perceive what is around him, but out of his field of vision. At first, this project will be focused only on gamers who play on the computer. Such a decision was made, taking into account the difficulty of integrating a program into the consoles since the user needs to be with other software running parallel to some game, to take advantage of the tools.

The overall goal of the project is to alleviate problems that deaf gamers encounter during their game plays. To achieve this, the following specifics objectives will be sought [1]:

• Provide a tool that allows the deaf player to have a visual mapping of the audio emitted by the games, mainly sound effects;

• Create an environment where the deaf gamer feels comfortable in being;

• Encourage discussion of accessibility in games;

• Allow the deaf gamer to know the nuances of a game, taking into account accessibility, before buying it.

The research is on classic and contemporary bibliographical references, as well as documentary analysis. Interviews will be conducted with game players and designers and the hearing impaired to identify critical points in the gaming experience. Based on the information collected will be proposed a series of practices that could be considered by the designer to provide accessibility in games that are still being created. For the existing games will be made a prototype as a proof of concept to support the visualization of the game sound landscape. The results obtained will be made available to the community interested in the accessibility of digital games.

The digital tools shall to have fluid navigation and natural understanding for the user. So, it is possible for him to get the best out of it and have behavioral satisfaction by his interaction. It was necessary to use a method to check the quality of navigation and whether it met the objectives. For this reason, the cognitive path method was chosen.

The cognitive pathway is an analytical method, and its primary objective is to evaluate the learning facility of the system [1]. In this way, the choice for such an approach was focused on the exploration of a system and it’s learning. This method allows the deductions and analysis to come from the researcher himself, as long as he has enough information and content to understand the user and his needs.

To support the cognitive course, this project relied on the best practices found in developing systems through Nielsen heuristics. The Nielsen [10] heuristics evaluation is a method to find usability problems in interfaces and add points to a basic set of heuristics, which are:

• Visibility of system state: the interface should always inform the user about what is happening.

• Correspondence between the system and the real world: the language of the interface must be compatible with that of the user, avoiding technical terms.

• Control and freedom of the user: the user must always be able to undo or go back a step, if there is no possibility, he must be informed of the reason.

• Consistency and standardization: a command, function, text, and icon must have the same purpose during navigation throughout the system.

• Error prevention: the interface should anticipate a possible user error and not allow it to happen.

• Recognize, do not remember: the user should not be forced to remember some interface factor. It should not need to decorate some information from one area, to be used in another.

• Help users diagnose and recover errors: the error message needs to accurately indicate the problem that has occurred and, in a clear way, the ways to correct it.

• Help and documentation: an interface needs to be simple to the point where the user does not need outside help, but it is still necessary to create documentation that can be accessed by the user in case of doubt.

• Flexibility and efficiency of use: the interface must be functional for a beginner, but also provide shortcuts so that the more experienced can accelerate their flow.

• Aesthetic and minimalist design: avoiding irrelevant information is necessary for the user to focus on what is interesting to him.

3.1 Deafness and Entertainment

Deafness is characterized by total or partial loss of listening ability. The causes are diverse, being able to be genetic or by external factors, like by physical traumas. It has several ways of being understood and can be analyzed from a cultural, health and even educational point of view [3].

The history of deaf education has, in general, been perpetuated through discriminatory situations. During Antiquity, more specifically in the Middle Ages, the deaf were considered “stupid” and “untrained” people, even being persecuted by extremist religious groups; those who learned to read and write were regarded as miraculous. It was only in the early sixteenth century that teaching methods began to be adapted for the deaf [8]. According to Lacerda [8], the data are uncertain regarding the pedagogical methods for the education of the deaf, because, at that time teachers worked autonomously, mainly for noble families or with much purchasing power, in this way, the professionals did not they shared methods with each other.

According to Soares [16], deafness in itself does not necessarily imply the inability or difficulty of learning of the individual. These, in turn, are symptoms of society and conventional pedagogical methods, which are not apt to include the deaf in education. According to the author, even if there are laws, they are hardly fulfilled comprehensively. The education of the deaf is often focused on oralization, and the majority of teachers are not bilingual, that is, they do not teach their students in their first language, which makes it difficult for the student to understand since it has variations of spoken.

In this way, deaf students, instead of being included in regular schools to live in a community, were deprived of education and not only that, but their behavior was also deprived because they were in schools mainly listening, their culture reduced and stereotyped, with significant efforts to be a listener. Therefore, it is necessary for the State and society to understand that the deaf subjects have a different identity, language, and culture than those of the listener. Therefore, there is an extremely thin line between inclusion and exclusion, since cultural aspects are rarely mentioned or remembered, which excludes the social identity of the deaf subject. Finally, understanding the history of the education of the deaf and understanding the effects on the development of these people is a crucial point for the evolution of this project.

Authors such as Bisol and Valentine [2] discuss the lack of cultural content and entertainment for deaf people. Although technological developments help to democratize entertainment and include deaf people in places where they were once inaccessible, it is still not fully exploited. It thus generates latent demand. Still, according to the authors, there are significant differences between the listener culture, about the deaf culture. This is mainly due to the difference in sign language for oralized language.

Also, the deaf population often have different references to people, artists, behaviors, writers, fables, plays, musicality, among other elements, all belonging to their own culture. A study by Thoma [17] found that there is a deficiency of representation of the deaf population in the media in general. In movies, for example, there are few films with deaf actors or that approach deafness as a theme. Therefore, it is still complex, but very necessary, to evaluate the conditions of the media that are an available deaf community, considering its totality, as well as its identity.

Regarding television, according to Crepaldi and Mendonça [4], the issue is even more critical, since this is the main means of communication and information of the population currently. In this way, guaranteeing the deaf the right to information, as well as entertainment, is also the duty of the media. There are different types of accessibility tools for need bearers, such as audio description, which is about narration describing programming, usually for blind or blind people and the Closed Captions often used by deaf people. The latter, in turn, is only the programming legend. However, it still has many weaknesses, since there are several text errors and there is the difficulty of understanding since not all deaf people were literate.

Musicality, on the other hand, is much discussed among the deaf community, as it is also a form of entertainment, it is also a way of reproducing and experiencing the art that the deaf often cannot appreciate due to the lack of resources. However, in the contemporary world, some resources have been used to include music in the lives of the deaf, such as the interpretation of music in sign language; as well as multisensory features, such as vibration, colors, and formats. Thus, deaf people can enjoy music, transforming it into a physical-motor experience and not just sound, as listeners are accustomed to [13].

Also, there are theater groups, focused on the deaf population. Theater plays are often used as an educational form. However, there are also theater groups with deaf actors. All this contributes to the experimentation of art and also as a form of entertainment for the deaf population, which respects and keeps alive the culture of this population. According to a study carried out by Barbosa [1], with the improvement of technology and the Internet, it is possible to use several resources for the education of deaf people, including elementary education. One of the alternatives would be online games, for example, since they have colors, shapes, and images that can be used as play tools in the teaching-learning process. Also, they can be used as entertainment. Nonetheless, Barbosa [1] considers that there are weaknesses in this method, since many auditory resources are used in such games.

In this way, it is possible to consider that through technology, communication and entertainment resources would guarantee the inclusion of deaf people in society. However, there is still a lack of investment, both on the part of the Government and on the part of the big media companies, making it difficult for the deaf population to access the various types of entertainment.

3.2 Deaf Gamer

The game industry, which is continually growing, is also of great importance to deaf people, even if significant difficulties are encountered in this environment, especially in online games. To verify the possible sounds present in the games, several games were experienced and compared when played with and without sound aid. This experience was not based on empathy (since, as a listener, there was already a previous memory of the games and would not be a correct method), but to note important points during the game that were demonstrated only through sound, with no visual resources of support.

The game Left 4 Dead 2 [7], developed by Valve Corporation, stood out against other titles used for this research. The survivor horror game (a sub-genre of action games, when it is set in horror scenes and features elements of survival games) is set in a world during a zombie apocalypse. The goal is to go from point A to point B on a map full of enemies. The player has three other survivors for the match; these can be controlled by other people when played online or through BOT’s (robot short). Important details were noted during the playing the game with sounds and were transcribed:

• At first, I hear zombies. Despite being in a quiet place, where you cannot see any, I know there are nearby enemies.

• Next, to the first door seen in the game I start to hear grunts and knocks on the door from the other side.

• There are special zombies that offer more danger to the player, and each one has a distinctive sound that sets them apart from all other enemies.

• When some important element or moment of the game is about to happen, there is a warning sound.

• Apart from enemies, there are other elements that inflict damage to the character, such as fire and gas. If I am suffering damage from something like that, there is a specific sound for each one.

Other important details were noted during the playing the game without sounds and were transcribed:

• The beginning of the game would not be possible to know of the presence of nearby enemies.

• I often ended up meeting special enemies as I had no clue that they were close.

• I opened doors and came across a horde of enemies, even if I thought I was in a quiet area of the game.

• I went to places with fire and only realized after having taken damage because a change occurs in the interface.

• When I’m on fire, even when I leave the danger area, flames appear in the lower corner, showing that the character is losing a life. The only problem is not knowing where the fire was coming from.

There was a sense of frustration every time the character died from some danger that was not being shown at the interface. When researching on a deaf gamers analysis, however, was found compliments to the closed captions feature, which was disabled until then. It needs to be enabled in the options menu, as shown in Fig. 1, and can only function as a caption of the characters’ speeches or as a textual translation of almost all the sound effects that take place in the game.

Audio configuration of Left 4 Dead game [7].

Full size image

With this feature, the experience is entirely different, as it is possible to anticipate enemies and create strategies, as shown in Figs. 2 and 3. It is possible to demonstrate the difference of when the feature is disabled and activated. Each type of sound has a different color, just as it is for each character.

Left 4 Dead [7] gameplay with sound visualization.

Full size image

Left 4 Dead [7] gameplay with sound visualization.

Full size image

In addition to color separations, brackets are used to identify when a sound is not verbal, such as a special enemy approaching or sound of flames near the player.

3.3 Modifications in Games

With the discovery of the possibility of closed captions in a game, a search was made to find out which other games have such functionality. Unfortunately, there are not many titles that have the application. However, during the research, it was possible to find something is very present in the world of games: the modifications. Many games or even consoles have undergone some unofficial modification.

The documentary series Paralelos [11] deals in its last chapter on the historical value of the modifications to the Brazilian gamer community. Modifications to football games were more successful than the original game. The modifier community grew along with the gaming market. When a blockbuster game does not provide Portuguese subtitles, it is possible to find several translation projects, which are freely available to the public.

The project was born facing one problem encountered by deaf gamers: the lack of visual clues that represent the sound. When analyzing the complaints in various forums and groups, it was evident the need for a tool that would improve the behavioral experience of these players. Much of the beginning of the project was useful to know the universe of the deaf. Before we proposed this work, my knowledge of the deaf culture was very limited.

Although the project’s inspiration came from the memory of a deaf childhood friend who then played some online games with me, we did not have any other interactions with the deaf community until this project. With that, in the first few weeks, we tried to understand the interactions between the deaf community and to know a little of its history. Before any direct contact, we had to know who we would be talking to and the pains they often face would miss something in games.

The user will access the radar through the game page or a link sent by someone from the community. The player can save a radar as a favorite, so it stays highlighted. It will be possible to search for radars in the community itself and, if you wish, to customize it to your liking. Most of the sounds created by gamers designers are geared towards some feedback to the player. Many of these sounds are not represented graphically in the game interface. There are several items that build the sound landscape in a game. Huiberts [5] formulates the model IEZA (acronym interface, effect, zone, and affection). His model brings theoretical tools for the development of the sound design.

The diegetic axis at Fig. 4 represents the sounds that occur in the fictional world of the game, the sounds the character could hear. The non-diegetic refers to sounds that are outside the fictitious environment, such as a soundtrack. The other axis represents if the sound that happens through the direct action of the player or caused by the game, but being part of the narrative or if it happens as an effect of ambiance. Therefore:

IEZA Framework [5].

Full size image

• Interface: sounds out of the fiction that communicates some activity of the game;

• Effect: sounds within the fiction that communicate some activity of the game;

• Zone: ambient sound, within the fictional world;

• Affection: Sound that emotionally characterizes an environment, not being within the fictional world.

Knowing these specifications, it was possible to understand the limitations that the radar would have. Although graphically representing the direction of a sound, it could not correctly represent, for example, zone and effect sounds, as shown in Figs. 5, 6, 7 and 8.

2d Sound landscape setup.

Full size image

3d Sound landscape setup.

Full size image

3d Sound landscape setup.

Full size image

3d Sound landscape setup.

Full size image

This could be solved if the sound representation was made specifically for a particular title, where the sound landscape could be studied and graphically represented. The limitation was taken into account, but it was not a deciding factor for a change of plans, as this project seeks to help as many deaf gamers as possible and this would not happen if the focus were on a single title.

This project defined ways to bring accessibility to the deaf gamers behavior, based on the knowledge acquired through research. As a result, an interface has been obtained that allows greater integration of the deaf with games, as well as providing opportunities to soften the limitations that the games bring when they do not offer enough visual clues of the sound landscape that can be used by these people.

The functionalities were built through an evolutionary process. It was possible to cross-check observation, qualitative, interview and find the same problems being reported. Some of these problems are solved by the community itself, which, through unofficial modifications made to the games, improves the experience and usability. The sound landscape can enhance the game experience for deaf players, allowing them to play similar to hearing players.

By exploring applications opportunities, even hearing players could benefit from this research goals. They can be able to have a different behavioral experience by seeing the sound landscape instead of only hearing it.

Considering the evolutionary process of the project, it is also possible to think about future improvements. For example, the creation of a cam chat functionality to simplify entertainment consumption, taking into account the growth of platforms such as Twitch.tv, where people play live games daily while interacting with the community. Another possibility of creation would be to automate the creation of closed captions, through a Machine Learning process, in which the main sounds of a sound landscape could be represented in a more automatic way.

This research work allows knowing a culture that was once far off. This situation led to reflect on how often was negligent a specific group of people during the design of some interface. However, being an antithesis, it also allowed me to know ways in which design can solve impasses created by some medium that does not provide accessible means to be used. It was a process of project evolution, accompanied by personal development.