research-article

Usability evaluation of a virtual reality smartphone app for a living museum

Authors:

Mohd Kamal Othman,

Altynai Nogoibaeva,

Mohamad Hardyman BarawiAuthors Info & Claims

Universal Access in the Information Society, Volume 21, Issue 4

Pages 995 - 1012

https://doi.org/10.1007/s10209-021-00820-4

Published: 01 November 2022 Publication History

Abstract

This paper elaborates the empirical evidence of a usability evaluation of a VR and non-VR virtual tour application for a living museum. The System Usability Scale (SUS) was used in between participants experiments (Group 1: non-VR version and Group 2: VR version) with 40 participants. The results show that the mean scores of all components for the VR version are higher compared to the non-VR version, overall SUS score (72.10 vs 68.10), usability score (75.50 vs 71.70), and learnability (58.40 vs 57.00). Further analysis using a two-tailed independent t test showed no difference between the non-VR and VR versions. Additionally, no significant difference was observed between the groups in the context of gender, nationality, and prior experience (other VR tour applications) for overall SUS score, usability score, and learnability score. Α two-tailed independent t test indicated no significant difference in the usability score between participants with VR experience and no VR experience. However, a significant difference was found between participants with VR experience and no VR experience for both SUS score (t(38) = 2.17, p = 0.037) and learnability score (t(38) = 2.40, p = 0.021). The independent t test results indicated a significant difference between participant with and without previous visits to SCV for the usability score (t(38) = −2.31, p = 0.027), while there was no significant differences observed in other components. It can be concluded that both versions passed based on the SUS score. However, the sub-scale usability and learnability scores indicated some usability issue.

references

[1]

Zara, J:. Virtual reality and cultural heritage on the web. In Proceedings of the 7th International Conference on Computer Graphics and Artificial Intelligence (3IA 2004), Limoges, France, pp. 101–112. (2004)

[2]

Yang J, Liu T, Jiang B, Song H, and Lu W 3D panoramic virtual reality video quality assessment based on 3D convolutional neural networks IEEE Access 2018 6 38669-38682

[3]

Tussyadiah IP, Wang D, Jung TH, and tom Dieck, M. C. Virtual reality, presence, and attitude change: empirical evidence from tourism Tour. Manage. 2018 66 140-154

[4]

Huang YC, Backman SJ, and Moore D Exploring user acceptance of 3D virtual worlds in travel and tourism marketing Tour. Manage. 2013 36 490-501

[5]

Pantano E and Corvello V Tourists’ acceptance of advanced technology-based innovations for promoting arts and culture Int. J. Technol. Manage. 2014 64 1 3-16

[6]

Kim MJ and Hall CM A hedonic motivation model in virtual reality tourism: comparing visitors and non-visitors Int. J. Inf. Manage. 2019 46 236-249

[7]

Lombard M and Snyder-Duch J Interactive advertising and presence: a framework J. Interact. Advert. 2013 1 2 56-65

[8]

American Association of Museums. 2011 mobile technology survey. http://www.aam-us.org/docs/center-for-the-future-of-museums/aam_mobile_technology_survey.pdf?sfvrsn=0. (2011)

[9]

Tussyadiah IP, Jung TH, and tom Dieck MC Embodiment of wearable augmented reality technology in tourism experiences J Travel Res. 2018 57 5 597-611

[10]

Huang YC, Backman SJ, and Backman KF Exploring the impacts of involvement and flow experiences in second life on people’s travel intentions J. Hosp. Tour. Technol. 2012 3 1 4-23

[11]

Lee H, Jung TH, tom Dieck MC, and Chung N Experiencing immersive virtual reality in museums Inf. Manag. 2020 57 5 103229

[12]

Wei W, Qi R, and Zhang L Effects of virtual reality on theme park visitors’ experience and behaviors: a presence perspective Tour. Manage. 2019 71 282-293

[13]

Han, D.I.D., Weber, J., Bastiaansen, M., Mitas, O., Lub, X.: Virtual and augmented reality technologies to enhance the visitor experience in cultural tourism. In: Augmented Reality and Virtual Reality, pp. 113–128. Springer, Cham (2019)

[14]

Garau C and Ilardi E The “non-places” meet the “places:” Virtual tours on smartphones for the enhancement of cultural heritage J. Urban Technol. 2014 21 1 79-91

[15]

Powerhouse Museum: Exhibition overview: http://www.powerhousemuseum.com/lovelace/index.php/exhibition-overview/ (2011)

[16]

evoline TR: Audio guide Istanbul. https://play.google.com/store/apps/details?id=com.evolinetr.audioguide (2013)

[17]

Chen W Historical Oslo on a handheld device – A mobile augmented reality application Proced. Comput. Sci. 2014 35 1877 979-985

[18]

Smithsonian National Museum of Natural History: A hall through new eyes: Skin & bones. http://naturalhistory.si.edu/exhibits/bone-hall/ (2015)

[19]

Musée des Beaux-Arts de Lyon: Our mobile application: http://www.mba-lyon.fr/mba/sections/fr/activite-culturelle/actus/appli-mobile (2015)

[20]

Schwarz, A.W., Turchinsky, M., Abbey, K.: Murder goes mobile at the Met! http://www.metmuseum.org/about-the-museum/now-at-the-met/features/2012/murder-goes-mobile (2012)

[21]

Sedano CI, Sutinen E, Vinni M, and Laine TH Designing hyper contextualized games: a case study with LieksaMyst Educ. Technol. Soc. 2012 15 2 257-270

[22]

McCord Museum: The Victorian period. http://www.mccord-museum.qc.ca/en/keys/games/17 (2015)

[23]

Niantic Inc: Ingress. https://play.google.com/store/apps/details?id=com.nianticproject.ingress&referrer=utm_source%3Dingress%26utm_medium%3Dwebsite%26utm_campaign%3Ddownload (2015)

[24]

City of Monterey: Explore historic Monterey. http://www.monterey.org/museums (2015)

[25]

National Park Service: Harry S Truman National Historic Site mobile/cell phone tours. http://www.nps.gov/hstr/learn/photosmultimedia/cell-phone-tour.htm (2015)

[26]

Dueholm J and Smed KM Heritage authenticities – a case study of authenticity perceptions at a Danish heritage site J. Herit. Tour. 2014 9 4 285-298

[27]

Lalicic L and Weismayer C The passionate use of mobiles phones among tourists Information Technology & Tourism. 2015 16 2 153-173

[28]

Mesároš, P., et al.: Use of Augmented Reality and gamification techniques in tourism. e-Rev. Tourism Res. 13(1/2), 366–381 (2016)

[29]

tom Dieck MC, Jung T, and Han D-I Mapping requirements for the wearable smart glasses augmented reality museum application J Hosp. Tour. Tech. 2016 7 3 230-253

[30]

tom Dieck, M.C., Jung, T., Michopoulou, E.: Experiencing virtual reality in heritage attractions: Perceptions of elderly users. In: Augmented reality and virtual reality, pp. 89–98. Springer, Cham (2019)

[31]

Hawkey, R.: Learning with digital technologies in museums, science centres and galleries. NESTA Futurelab (National Endowment for Science Technology and the Arts), UK. http://telearn.archives-ouvertes.fr/hal-00190496 (2004)

[32]

Baltsavias, E., Gruen, A., Van Gool, L., Pateraki, M.: Recording, modelling and visualisation of cultural heritage. Taylor & Francis Group, London, United Kingdom (2006)

[33]

Choromański K, Łobodecki J, Puchała K, and Ostrowski W Development of Virtual Reality application for Cultural Heritage visualisation from multi-source 3D data Int. Arch. Photogr. Remote Sens Sp. Inf. Sci. 2019 2 9 261-267

[34]

Chung N, Lee H, Kim JY, and Koo C The role of augmented reality for experience-influenced environments: the case of cultural heritage tourism in Korea J. Travel Res. 2018 57 5 627-643

[35]

Southall, H., Marmion, M., Davies, A.: Adapting Jake Knapp’s design sprint approach for AR/VR applications in digital heritage. In Augmented reality and virtual reality, pp. 59–70. Springer, Cham (2019)

[36]

Cameron F and Kenderdine S Theorising digital cultural heritage: A critical discourse 2007 Cambridge, MA The MIT Press

[37]

Jones, G., Christal, M.: The future of virtual museums.: Online, immersive, 3D environments. Created Realities Group, Austin, TX (2002)

[38]

Smithsonian National Museum of Natural History: Panoramic virtual tour. http://www.mnh.si.edu/panoramas/ (2015)

[39]

Tukiainen M Online virtual tour by panoramic photography (master’s thesis) 2013 Espoo, Finland Helsinki Metropolia University of Applied Sciences

[40]

Xu, W., Kruminaite, M., Onrust, B., Liu, H., Xiong, Q., Zlatanova, S., Sensing, R.: A 3D model based indoor navigation system for Hubei Provincial Museum. In: Zlatanova, S., Sithole, G. (eds.) ISPRS Acquisition and Modelling Indoor and Enclosed Environments 2013, vol. XL–4/W4, pp. 11–13. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Cape Town, South Africa (2013)

[41]

Purnama IKE, Zaini A, and Sumpeno S Implementation of panorama 360 for virtual touring at Tugu Pahlawan Museum Surabaya J. Theor. Appl. Inf. Technol. 2014 62 3 800-804

[42]

Kumar, A., Vithani, T.: A comprehensive mobile application development and testing lifecycle. In: IT Professional Conference (IT Pro), vol. 2014, pp. 1–27. IEEE (2014)

[43]

Brooke, J.: SUS - A quick and dirty usability scale. http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html (1986)

[44]

Sauro, J.: Measuring usability with the System Usability Scale (SUS). http://www.measuringu.com/sus.php (2011)

[45]

Bangor A, Kortum P, and Miller J Determining what individual SUS Scores mean: adding an adjective rating scale J. Usability Stud. 2009 4 3 114-123

[46]

Nielsen J Usability engineering 1993 Cambridge, MA Academic Press

[47]

Wang CH, Tsai NH, Lu JM, and Wang MJJ Usability evaluation of an instructional application based on Google Glass for mobile phone disassembly tasks Appl. Ergon. 2019 77 58-69

[48]

Habler, F., Peisker, M., Henze, N.: Differences between smart speakers and graphical user interfaces for music search considering gender effects. In: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia, pp. 1–7. (2019)

[49]

Orfanou K, Tselios N, and Katsanos C Perceived usability evaluation of learning management systems: empirical evaluation of the System Usability Scale Int. Rev. Res. Open Distrib. Learn. 2015 16 2 227-246

[50]

Othman, M.K.: Measuring visitors’ experiences with mobile guide technology in cultural spaces (Doctoral dissertation, University of York). (2012)

[51]

Sauro, J.: Does prior experience affect perceptions of usability? http://www.measuringu.com/blog/prior-exposure.php (2011)

[52]

Myers, C.M., Furqan, A., Zhu, J.: The impact of user characteristics and preferences on performance with an unfamiliar voice user interface. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–9. (2019)

[53]

Sauro, J., Lewis, J.R.: Correlations among prototypical usability metrics: evidence for the construct of usability. In: The Proceedings of CHI 2009. (2009)

[54]

Bangor A, Kortum PT, and Miller JT An empirical evaluation of the system usability scale Intl. J Human-Comput. Interact. 2008 24 6 574-594

[55]

Häkkilä, J., Hannula, P., Luiro, E., Launne, E., Mustonen, S., Westerlund, T., Colley, A: Visiting a virtual graveyard: designing virtual reality cultural heritage experiences. In: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia, pp. 1–4. (2019)

[56]

Debailleux, L., Hismans, G., Duroisin, N.: Exploring cultural heritage using virtual reality. In: Digital cultural heritage, pp. 289–303. Springer, Cham (2018)

[57]

Lewis JJR and Sauro J Revisiting the factor structure of the system usability scale J Usability Stud. 2017 12 4 94-103

[58]

Borsci S, Federici S, Bacci S, Gnaldi M, and Bartolucci F Assessing user satisfaction in the era of user experience: Comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience Int. J Human-Comput. Interact. 2015 31 8 484-495

[59]

Lewis, J.J.R., Sauro, J.: Revisiting the factor structure of the system usability scale. J. Usability Stud. 12(4), (2017)

[60]

Mol M, van Schaik A, Dozeman E, et al. Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis BMC Psychiatry 2020 20 218

[61]

Tullis, T.S., Stetson, J.N.: A comparison of questionnaires for assessing website usability. In: Usability professional association conference, vol. 1. (2004)

[62]

Casu, A., Spano, L.D., Sorrentino, F., Scateni, R.: RiftArt: bringing masterpieces in the classroom through immersive virtual reality. In: STAG, pp. 77–84. (2015)

[63]

Ariffin, A.H., Talib, A.Z.: User acceptance of panoramic views as a technique for virtual tour in an educational environment. In: International Conference on Informatics Engineering and Information Science, pp. 117–127. Springer, Berlin, Heidelberg (2011)

[64]

Johnson, C.: On the problems of validating DesktopVR. In: People and Computers XIII, pp. 327–338. Springer, London (1998)

[65]

Rautenbach V, Cöltekin A, and Coetzee S Exploring the impact of visual complexity levels in 3D city models on the accuracy of individuals’ orientation and cognitive maps ISPRS Ann. Photogr. Rem Sens. Sp. Inf. Sci. 2015 2 499-506

[66]

Bano, M., Zowghi, D.: User involvement in software development and system success: a systematic literature review. In: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, pp. 125–130. (2013)

[67]

Kujala S User involvement: a review of the benefits and challenges Behav. Inf. Tech. 2003 22 1 1-16

[68]

Bruno F and Muzzupappa M Product interface design: A participatory approach based on virtual reality Int. J. Hum Comput Stud. 2010 68 5 254-269

[69]

Kim S A cross-cultural study of on-site film-tourism experiences among Chinese, Japanese, Taiwanese and Thai visitors to the Daejanggeum Theme Park South Korea. Curr. Issues in Tour. 2012 15 8 759-776

[70]

Bevan N and Macleod M Usability measurement in context Behav. Inf. Tech. 1994 13 1–2 132-145

[71]

Smith A and Yetim F Global human-computer systems: cultural determinants of usability Interact. Comput. 2004 16 1 1-5

[72]

Etzold, J., Englert, M., Grimm, P., Jung, Y., Klomann, M.: MIPos: towards mobile image positioning in mixed reality web applications based on mobile sensors. In: Proceedings of the 19th International ACM Conference on 3D Web Technologies, pp. 17–25. (2014)

[73]

Guttentag DA Virtual reality: applications and implications for tourism Tour. Manage. 2010 31 5 637-651

[74]

Wan CS, Tsaur SH, Chiu YL, and Chiou WB Is the advertising effect of virtual experience always better or contingent on different travel destinations? Inf. Tech. Tour. 2007 9 1 45-54

[75]

Jung TH, Lee H, Chung N, and Tomdieck MC Cross-cultural differences in adopting mobile augmented reality at cultural heritage tourism sites Int. J Contemp. Hosp. Manag. 2018 30 3 1621-1645

[76]

Jung T, Chung N, and Leue MC The determinants of recommendations to use augmented reality technologies: the case of a Korean theme park Tour. Manage. 2015 49 75-86

[77]

Moorhouse, N., Tom Dieck, M.C. Jung, T.: Augmented reality to enhance the learning experience in cultural heritage tourism: an experiential learning cycle perspective. e-Rev. Tourism Res. 8: 1–5 (2017)

[78]

Buhalis D and Law R Progress in information technology and tourism management: 20 years on and 10 years after the Internet—The state of eTourism research Tour. Manage. 2008 29 4 609-623

[79]

Refsland ST, Ojika T, Addison AC, and Stone R Virtual heritage: breathing new life into our ancient past IEEE Multimed. 2000 7 2 20-21

[80]

Styliani S, Fotis L, Kostas K, and Petros P Virtual museums, a survey and some issues for consideration J. Cult. Herit. 2009 10 4 520-528

[81]

Jackson, R., Bazley, M., Patten, D., King, M.: Using the web to change the relation between a museum and its users. In: Museums and the Web, vol. 98. (1998)

[82]

Cheong R The virtual threat to travel and tourism Tour. Manage. 1995 16 6 417-422

[83]

Stepaniuk, K., Bałakier, U., Januszewska, A.: Virtual tours in the opinion of the users of social networking sites in Poland and Belarus. Ekonomia i Zarządzanie, 6. (2014)

[84]

Letellier, R.O.B.I.N.: Virtual reality: A new tool for sustainable tourism and cultural heritage sites management. In: Proceedings CIPA Symposium. (1999)

[85]

Baker, E., Bakar, J.A., Zulkifli, A.: A conceptual model of mobile augmented reality for hearing impaired museum visitors’ engagement. (2020)

[86]

Sheehy, K., Garcia Carrizosa, H., Rix, J., Seale, J., Hayhoe, S.: Inclusive museums and augmented reality. Affordances, participation, ethics and fun. Int. J. Inclusive Mus. In-Press (2019)

Cited By

Ye DCho DLiu FXu YJia ZChen J(2024)Investigating the impact of virtual tourism on travel intention during the post-COVID-19 era: evidence from ChinaUniversal Access in the Information Society10.1007/s10209-022-00952-123:4(1507-1523)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10209-022-00952-1
Zielke MZakhidov DDeFries EShringarpure MAvila RKeul JCarpenterTurner DHargrove SBateman SYoung C(2024)Using in situ research-based design to explore learning module effectiveness and usability in a virtual reality system for workforce trainingVirtual Reality10.1007/s10055-024-00995-028:4Online publication date: 6-Sep-2024
https://dl.acm.org/doi/10.1007/s10055-024-00995-0
Lam MMuhammad Nizam SArshad HSuwadi N(2023)Tangible interaction technique with authoring capability for kitchen designMultimedia Tools and Applications10.1007/s11042-023-14376-382:19(30125-30150)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1007/s11042-023-14376-3
Show More Cited By

Index Terms

Usability evaluation of a virtual reality smartphone app for a living museum
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interaction paradigms
      1. Virtual reality

Index terms have been assigned to the content through auto-classification.

Recommendations

Evaluation of virtual reality systems for usability
CHI EA '99: CHI '99 Extended Abstracts on Human Factors in Computing Systems

The adoption of 3D virtual interfaces is becoming more widespread. Although design guidelines and usability evaluation methods exist for the conventional 2 dimensional Graphical User Interface (GUI), they may or may not however, be appropriate in the ...
Immersive virtual reality with multimodal interaction and streaming technology
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

In this demo, we present an immersive virtual reality (VR) system which integrates multimodal interaction sensors (i.e., smartphone, Kinect v2, and Myo armband) and streaming technology to improve the VR experience. The integrated system solves the ...
Was I There?: Impact of Platform and Headphones on 360 Video Immersion
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Virtual reality industries claim that 360 videos are a powerful tool for creating empathy because they are an immersive medium, and wearing headphones is encouraged for the full immersive experience. To investigate these claims, we carried out a 2x2 ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Universal Access in the Information Society

Universal Access in the Information Society Volume 21, Issue 4

Nov 2022

229 pages

ISSN:1615-5289

EISSN:1615-5297

Issue’s Table of Contents

© The Author(s) 2021.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 November 2022

Accepted: 06 May 2021

Author Tags

Qualifiers

Research-article

Funding Sources

Universiti Malaysia Sarawak

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ye DCho DLiu FXu YJia ZChen J(2024)Investigating the impact of virtual tourism on travel intention during the post-COVID-19 era: evidence from ChinaUniversal Access in the Information Society10.1007/s10209-022-00952-123:4(1507-1523)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10209-022-00952-1
Zielke MZakhidov DDeFries EShringarpure MAvila RKeul JCarpenterTurner DHargrove SBateman SYoung C(2024)Using in situ research-based design to explore learning module effectiveness and usability in a virtual reality system for workforce trainingVirtual Reality10.1007/s10055-024-00995-028:4Online publication date: 6-Sep-2024
https://dl.acm.org/doi/10.1007/s10055-024-00995-0
Lam MMuhammad Nizam SArshad HSuwadi N(2023)Tangible interaction technique with authoring capability for kitchen designMultimedia Tools and Applications10.1007/s11042-023-14376-382:19(30125-30150)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1007/s11042-023-14376-3
Ricci MEvangelista ADi Roma AFiorentino M(2023)Immersive and desktop virtual reality in virtual fashion stores: a comparison between shopping experiencesVirtual Reality10.1007/s10055-023-00806-y27:3(2281-2296)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1007/s10055-023-00806-y
Menezes PPereira EFerreira BAmorim A(2022)Immersive Technologies for Vestibular RehabilitationInternational Journal of Creative Interfaces and Computer Graphics10.4018/IJCICG.31183513:1(1-20)Online publication date: 21-Oct-2022
https://dl.acm.org/doi/10.4018/IJCICG.311835

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents