A Comparative Usability Analysis of Virtual Reality Goggles

Virtual Reality (VR) is a three-dimensional computer-generated environment, which simulates the physical world through interactive devices that send and receive information. [1] At this synthetic environment, the user takes the physical body as a reference with which he/she coordinates his/her actions on the virtual environment, so the user contextually undergo the experience that leads to the sensation of physical involvement [2].

Similar to a book or a picture, which transports the reader or the observer from the physical environment to that of the story or painting, VR transports the person from the physical world to an environment in which he/she is not physically present, although he/she feels like he/she was [3].

Even if it appeals to fiction, the simulation in this environment presents situations with a degree of realism that allows the user to make decisions and solve problems in the physical world.

Considering that the user presents him/herself as information, metaphorized in the virtual environment, on VR environments the biological and virtual bodies coexist and act in a coordinated way, so the user can present him/herself, interact and modify the virtual world [1].

In other words, VR employs multimedia, computer graphics, image processing and other resources to create synthetic environments so the body acts both as a support for cultural prosthetics (user dressed with VR devices) and as a sign (user immersed, metaphorized, on the virtual environment) [1].

In this context, concepts such as immersion, presence, interaction and involvement, fundamental to the study of VR, are indispensable to the understanding of the users’ physical and psychological experience in these systems [3, 4].

Taking these assumptions as a starting point, this study focuses on the immersion, which for Witmer and Singer [3] is related to the psychological state characterized by being involved in, included in and interacting with an environment that provides a continuous flow of stimuli.

Thus, this study aimed to evaluate the usability of two types of VR goggles for smartphone users, since the less the user can perceive the physical world (see, touch or hear), the greater the VR immersion is [5].

These technologies integrate the physical world to a simulated, virtual world, creating in the user a sense of immersion.

The high degree of sophistication provides realistic, immersive and perceptual experiences through multiple technological components (interactional and perceptual devices, software and applications).

Generally, products developed for commercial use are focused on immersive visual, auditory and tactile solutions.

2.1 VR Goggles

VR goggles, many with an integrated audio system, stimulate visual and auditory perception to create realistic perceptual sensations for VR users.

Aware of these issues, the gaming industry and social media, especially Facebook^{Footnote 1}, realized the great potential of these technologies and bet on the development of new devices that contributes to the users’ interaction with other people and characters in simulated virtual environments^{Footnote 2}.

In this context, the development of VR goggles with low prices and greater technological simplicity, contributed to the dissemination of VR among smartphone users.

This was possible due to the emergence of intelligent technology devices and sensors that use the force of gravity to locate a position of an object in space. Thus, VR use the smartphone device as a display, making VR goggles cheaper and not requiring specific configurations such as we can see in more sophisticated VR goggles.

One of the best known versions of this type of goggles is the VR Gear (Fig. 1), from Samsung, which enables the use of applications and videos in virtual reality. However, these goggles are only compatible with some specific devices of the brand itself.

(Source: https://cdn0.vox-cdn.com/thumbor/LJsvpfKqmt3u0CFhAaKRUvuhe_A=/0x0:1100x619/1600x900/cdn0.vox-cdn.com/uploads/chorus_image/image/47262136/gear_vr.0.0.jpg)

Gear VR

Full size image

However, other devices on sale are compatible with a wider range of devices and are cheaper. The Google Cardboard (Fig. 2), one of the cheapest for sale, is designed to support most smartphones. It costs about $15 and is made out of cardboard^{Footnote 3}.

(Source: http://cdn2.hubspot.net/hubfs/307727/Stock_Photos/Cardboard.jpg)

Google cardboard

Full size image

There are other devices that follow the same functional principle of the Google company goggles, being compatible with most appliances and at low cost, although made in plastic. Looking like the VR Gear, the VR Box 2.0 (Fig. 3) and VR Box 360 (Fig. 4) goggles are an example.

(Source: http://www.buyvrguide.com/vr-headsets/vr-box-2-0/)

VR BOX 2.0

Full size image

(Source: https://vrbox360.com/products/vr-box-360?variant=14848729543)

VR BOX 360

Full size image

The VR Box 2.0 goggles are made of ABS plastic; Adjustable side and top handles (with velcro), made of elastic material; They have padded for comfort of the facial region (foam of PU and synthetic leather); Adjustment knobs for the lenses (eye distance and focus); Holes for sound output on the front; Removable (drawer-type) cover (Fig. 5) for attaching 3,5 to 6 inch smartphones with adjustable spring-loaded locking device; Side holes for earphone wiring, and front sliding cover for use by the rear camera of the smartphone in Augmented Reality (AR) applications.

(Source: http://www.shoptime.com.br/produto/16470981/oculos-realidade-virtual-3d-com-controle-vr-box-2.0?condition=NEW&cor=BRANCO&tamanho=UNICO)

Removable drawer-type cover - VR BOX 2.0 goggles

Full size image

The VR Box 360 goggles have practically the same characteristics in regard to the material (ABS plastic); Lens adjustment knobs (however, with different texture); Side holes for wired headphones and padding for facial comfort (PU foam and elastic synthetic fabric). However, the coupling of the smartphones is different, with articulated axle system for the opening of the cover (Fig. 6), through lower hinge, and an attachment of the device by an adhesive contained inside (compatible with smartphones from 4 to 6 inches); The adjustment of the lateral and superior handles is made through a plastic buckle extension system; There is a magnet slider button on the side of the device, responsible for selecting menu options in some VR mobile applications but it does not have access to the back camera of the smartphone in applications.

(Source: https://vrbox360.com/products/vr-box-360?variant=14848729543)

Cover with articulated axle system - VR BOX 360 goggles

Full size image

Despite the similarity of the products in many aspects, the changes in some characteristics like: opening and fastening system for the coupling of the smartphone; adjustment system of the head loops; absence of sliding magnet button on one of the devices; and the difference of material and texture in some parts of the products, among other factors, may cause different results in the usability of the product.

In this study, we evaluated the usability of these devices, through a comparative analysis, performed from the test with users, in order to identify usability problems and propose recommendations for their solution.

There are several definitions for usability. According to Tullis and Albert [6] they are all related to the involvement of a user developing an activity and using an interface.

For Krug [7], usability is the guarantee that something works correctly regardless of the user’s degree of knowledge. Jordan [8] follows the same line of reasoning, stating that the term usability is directly related to the friendly use of products. He underscores the importance of user satisfaction when using the product to achieve his/her purposes that is related to the degree of comfort provided.

ISO 9241-11 [9] defines usability as “the extent to which a product can be used by certain users to achieve specific goals with effectiveness, efficiency, and satisfaction in a certain context of use.” Leventhal and Barnes [10] consider the definition of ISO insufficient to evaluate whether a product is usable or not, although they point out what a product should present to be satisfactory in this matter.

Some authors have defined usability models which are composed by features of an usable interface, how they fit together, what they mean and their contribution to usability from common points, although these models remain unique. In this sense, Leventhal and Barnes [10] rely on three different usability models - Shackel, Nielsen and Eason - to construct their own, bringing together the most relevant points of each.

Shackel [11] defines usability as the capability of a product being effectively and easily used by a wide number of users, to acquire the knowledge and conditions necessary for an exercise of an activity at a specified time and environment.

For Nielsen [12], the goal of usability is to construct transparent interfaces that would be capable to provide ease of use, effectiveness and efficiency in user experience, allowing the activity to be controlled by the user in a freer interaction.

Eason [13] considers usability resulting from the relationship between the user interface and a set of situational variables: characteristics of the task, user, system and user reactions.

In these terms, the Leventhal and Barnes [10] usability model is more complete because it assumes that several variables considered together define whether an interface is usable or not, since the different variables levels may generate different demands on interface.

This study is the result of a usability evaluation of two VR goggles, held in a room of the Design Department at Federal University of Pernambuco, Brazil.

It was adopted the ‘Think Aloud Protocol’, in which the user him/herself narrates about the actions taken, the decisions made, as well as speaks about his/her own opinions and feelings about the evaluated product.

This method had two phases: (1) systematic users’ statements collecting; and (2) statements analysis, in order to obtain a model of the user’s cognitive processes when faced with a problem.

The pilot study was carried out with 02 users to verify the procedures and make adjustments in the experiment.

The convenience sample [6] consisted of 10 smartphone users, of both genders and aged between 18 and 29 years (due to the higher acceptance of VR among young people). Participants were recruited at UFPE Design Department through classroom disclosure, spontaneously attending the study.

During the usability test, the products were randomly delivered to users who had 10 min to perform the task. The devices evaluated were two VR goggles for smartphone users: (1) VR BOX 2.0; and (2) VR BOX 360.

Considering the crescent production of VR goggles over the years and the constant search to novelty by smartphone users, the criteria of choice of the evaluated devices was the compatibility with several smartphone models and the consequent use by a large number of users.

The experiment was performed in an air-conditioned room where the user positioned him/herself in front of a table, where were two VR goggles, its respective guides and an iPhone 5.

Before the usability test has started the user was informed about the test, about the task being performed and asked to speak loudly about what he/she was thinking while performing the task.

After receiving the smartphone with the app ‘Sites in VR’ with the main screen opened, the participants were asked to: (1) insert the smartphone in the VR goggles; (2) set the straps to fit the user head diameter; (3) adjust the lenses; (4) choose the ‘Tower’ option through eyetracking; (5) choose the ‘Eiffel Tower’ option; (6) choose the ‘Garden I’ immersion scenario; (7) explore the interface; (8) remove smartphone from VR goggles; and (9) remove the goggles.

As the participant had already used the smartphone app in the ‘Garden I’ immersion scenario, the following steps were requested: (1) to insert the smartphone in the VR goggles; (2) set the straps to fit the diameter of the head; (3) adjust the lenses; (4) explore the interface; (5) remove smartphone from VR goggles; and (6) remove the goggles.

Users were asked to speak loudly whatever came to their mind during the use of the product. The records were filmed for later qualitative and quantitative data analysis.

After the test with the product, the participants were asked to respond to a post-test questionnaire from which users’ opinions were obtained.

The frequency and the situational constraints of the task as well as the users’ level of expertise were evaluated.

The problems were identified and categorized according to the variables: ease of learning, ease of use, ease of relearning, flexibility, satisfaction and task match.

By the problems identification were discussed the main aspects of the product and were suggested improvement recommendations for the evaluated devices.

Considered by Nielsen [12, 14] the most valuable usability test, the ‘Thinking Aloud Protocol’ corresponds to the users’ narrative about the actions performed, the decisions made, their opinions and their feelings while interacting with the product (or prototype) under evaluation. According to Jasper [14, 15], this method consists of two phases: (1) systematic users’ statements collecting; and (2) statements analysis, in order to obtain a model of the user’s cognitive processes when faced with a problem.

According to Nielsen [12, 14], the method can be performed both individually and in groups, with two variations: the critical response and the periodic report. In the critical response, the user reports their actions during its execution.

The usability test and the users’ statements were filmed to data analysis. The problems identified were categorized from the variables: ease of learning, ease of use, ease of relearning, flexibility and task match according to Leventhal and Barnes’ usability model [10].

This model was chosen because VR goggles alone does not guarantee users’ immersion, since the goggles integrates a human-computer interface that only allows the user experience if the user connects the device to a smartphone and an app.

Regarding the results obtained in the usability test, 80% of the participants (n = 8) had never used any evaluated goggles. About the first variable, ease of learning, when questioned, the participants reported difficulty in handling the devices (60%, n = 6), assessing the level of difficulty for beginners as very high. At the time, the participants classified themselves as non-dominant in the use of VR goggles (60%, n = 6), although they recognized the importance of this type of device (100%, n = 10).

On the second variable, ease of use, 50% of the participants (n = 5) experienced difficulty handling VR BOX 2.0 and suggested changes such as: better fit for the smartphone; leave buttons more visible facilitating handling and instinct; improve the cushioning and drawer where smartphone is located. Regarding VR BOX 360, 70% of the participants (n = 7) reported feeling easier to perform the task, because they found the goggles more practical, comfortable, with better smartphone positioning and better image visualization. Regarding the cover opening the VR goggles, the best evaluated was the hinged axle system, through the lower hinge of the VR BOX 360. However, the participants warned about the possibility of wear of the fixing sticker of the smartphone, which would make it impossible to use the product.

When asked about the third variable, easy to relearning, 60% of the participants (n = 6) reported to be very difficult undo an action in VR BOX 2.0. During filming, on several occasions the participants had to remove their goggles, resort to the manual or even to the experimenter to try to complete the task.

Regarding the item flexibility and task match, 50% (n = 5) expressed and reported discomfort during the task with both goggles. In VR BOX 2.0 the participants reported: nuisance at the nose support, difficulty adjusting the lenses, dizziness and overlapping images. To VR BOX 360 goggles, the complaints were: dizziness, visualization of goggles edges and overlapping images.

During the test it was also possible to obtain informations about the user satisfaction. Regarding this item, 50% of the participants (n = 5) said they were satisfied and 10% (n = 1) was very satisfied with the VR BOX 360. When questioned about which goggles they would choose to buy, 80% (n = 8) choosed the VR BOX 360 goggles, although they find their looks less appealing. The users’ criteria were: the ease of use, the best image visualization and the fact that it was more practical to insert the smartphone.

In conclusion, the post-test questionnaire ratifies the results found in the usability test and allows an even more detailed analysis about the users interactions with the product.

The interaction with VR goggles for smartphones users presents certain peculiarities that distinguish this type of device from the VR goggles to conventional computers.

In this sense, the main aspects to be considered in the design and redesign of this type of device were listed in order to provide the users a better product usability and, consequently, better quality of immersion.

1. 1.

   The first aspect identified was the need to insert a smartphone into the VR goggles. This seemingly simple procedure needs to be so intuitive that the user could easily identify where the smartphone should be stored.

2. 2.

   It is necessary that the handling to open the smartphone compartment be adapted to the size of users hands and to different handles, especially to those who have some type of motor disability.

3. 3.

   The smartphone attachment to the storage compartment should not offer resistance for the user, facilitating the device insertion and removal.

4. 4.

   In addition to intuitiveness, the location for adjusting the lenses should be well signaled so that even a person with little schooling or vision problems can use the device.

5. 5.

   It is important that the focus of the image be adjusted based on the physical characteristics of the user. The device could make use of swingable spherical knobs for this purpose.

6. 6.

   It is essential that VR goggles, with an inserted smartphone, present a maximum weight compatible with the body proportions (consider anthropometric metrics) of the users, avoiding to generate muscle, cervical and postural discomforts.

Regarding the fact that VR goggles depend on the smartphone’s settings to provide an immersion experience, it is recommended that the user, after adjusting the device to his/her own head, can configure the gyroscope and accelerometer only with head movements, avoiding interferences in the experience of immersion and the smartphone removal.

To integrate the goggles to a smartphone, we suggest that when the user purchases the product, it comes with an app. Once installed, the app identifies the user’s smartphone version and configures the needs of VR (like a “Virtual Reality mode”). In this way, the phone remains connected to the internet, but does not ring, vibrate or receive calls during the immersion.

Another recommendation for a better interaction with the interface would be to redesign the goggles by integrating the eye tracking feature. With eye tracking the user could select the command and configuration options on the smartphone screen. This feature would also allow the user to interact with the interface from any VR app, without the need to acquire a bluetooth remote control or remove the goggles.

Finally, it is recommended that the graphic information which indicates the functions (e.g. lens adjustment, cover opening and holes for the passage of the headphone plugs) should be printed in high relief and in contrasting colors with the surface of the product.

Despite the widespread acceptance of VR goggles for smartphone users, much still needs to be improved in order to ensure a better immersion experience.

In this sense, eye tracking feature contributes for greater user autonomy while interacting with the product and it is a small investment compared with its benefits.

We also recommend attention to informational and handling aspects of the product, as improper use can cause serious users discomfort.

In a few words, we can conclude that ergonomics brings important contributions to the design, analysis and development of VR products, in order to provide a better adaptation of the product to the user.

This adaptation, in addition to providing a better performance for the users, also contributes to funnier, safer, and more challenging experiences.