Evaluating Learning Outcomes of Virtual Reality Applications in Education: A Proposal for Digital Cultural Heritage

The main contributions of this paper are summarised as follows:

The reminder of the paper is organised as follows. Over the huge amount of recent work in the field of education, the works more related to our experiment have been selected and analysed in Section 2. Afterwards, the methodology used to conduct the tests have been described in Section 3; the experimental setup and the consequent protocol used for gathering the learning outcomes by the students are detailed in Section 4. Results of the experiments, together with the statistical validation of the proposed approach can be found in the dedicated results in Section 5. Discussions, benefits, pros and cons of our methodology, and findings are described in Section 6. Section 7 is dedicated to the concluding remarks and future research directions.

In this section, some VR applications in the DCH domain are described. Moreover, the work of [45] has been examined, since it provides an overview of the state of the art of serious game in the DCH domain, emphasising the educational ambition of games.

Through the Nearpod educational platform,¹ primary classes at the Unified School of San Francisco and a public school complex of Polk County in Florida allow children to take virtual tours of Easter Island, ancient Egypt, a coral reef, and Mars. The Marin School of the Arts in Novato, California, has a wall covered with ultra-flat monitors that are used by many classes of children to create and manipulate \(360^\circ\) scenes while on the opposite side, a protected area has been created for students to use an optical device: the HTC Vive headsets.

In 2014 Mendel High School in Opava, Czech Republic, was the first European high school to create courses integrated with the latest generation VR technology. Using optical devices, Oculus Rift headsets, and a Leap Motion controller, they offer educational units in science and history.²

The Google Expeditions application³ has been designed for classwork, enabling virtual visits to the most attractive places in the world using just a mobile phone and a Google Cardboard headset.

Related to DCH environment are the \(360^\circ\) views of the Vatican app,⁴ which provides tours and information.

Cave Automatic Virtual Environments (CAVE) [46] is a tool providing a fully-immersive VR experience in a room where the walls and floor are projection screens. Users wearing 3D glasses can move freely in that projected world that, while holding interesting educational and teaching potential, is seldom used due to cost and dedicated space requirements. Despite this, CAVE technology has been adopted rarely in the educational field for DCH.

In the context of underwater exploration of DCH, interesting is the work of [47], which proposes a VR application to overcome limitations of the underwater archaeological site. The experimental phase demonstrates that the system can provide a learning experience with a high emotional impact, both for younger and inexperienced users.

In literature, there are different works that study the Bloom’s Taxonomy (BT) for similar purposes. In the paper presented by [48], the authors have compared three different learning experiences using the HTC VIVE VR headsets: interactive 3D model, role play scene, and 3-D creation of space. Considering BT, they have grouped the levels of knowledge and have established the learning experience that can be associated to a group.

Instead, recently, in [49], the authors evaluated the learning outcomes with a VR application in the educational field, considering three levels of the BT (applying, analysing, and evaluating). This choice was raised on the fact that VR has the potential to influence higher levels of BT.

Another recent paper has integrated a VR application for teaching systems of linear equations. Conversely, to evaluate the effectiveness of the learning, they have adopted only the last four levels of knowledge (creating, evaluating, analysing, and applying) of BT [50].

The BT is also considered in [51] to evaluate the levels of knowledge reached using VR applications in education. Starting from the six levels of BT, the authors have developed 18 new indicators to establish the levels of knowledge. Furthermore, in [52], the authors propose a virtual heritage learning game based on an ancient Egyptian temple by using BT.

However, to the best of our knowledge, the current literature has no quantitative evaluations that use KPIs to determine education validity of VR applications for the DCH domain. These indicators are measurable values important for teachers since demonstrate the effectiveness regarding student learning processes and therefore the teaching method [53].

Research presented by [54] proposed a technological pedagogical content knowledge framework as an analysis tool for describing student competence with a constructive map. Teachers used these maps to better understand student competence levels, establishing objectives and following learning procedures. The recent work of [55] studied a technology acceptance model based on structural equation modeling [56] to evaluate student acceptance of tablet use as a technological tool in mathematics classes in a Middle Eastern University. The results highlighted user satisfaction and perceived usefulness.

Our research questions are as follows:

To evaluate how innovative educational paths improve learning, the test was carried out within the SmartMarca project, a platform specifically created to manage AR and VR contents for DCH [57]. The project has several beneficial and advantageous objectives for cultural tourism coming from digitisation, web publication, and utilisation.

This VR application allows an immersive visit to the Roman Theatre of Falerone, providing a contextual and interactive reading of historical and architectural information about the 3D model of the theatre. Elements and details are highlighted while the user moves within the virtual reconstruction. This application, besides providing a 360 \(^{\circ }\) view of a 3D model, exploits immersive visualisation through a stereoscopic mode, placing users completely inside the model (Figure 2).

While the app was initially designed for tourism purposes, its VR permits an in-depth analysis worthy of didactic investigation. The teaching experience undertaken through the SmartMarca⁵ application has provided interesting results regarding the potential of digital tools introduced in daily school practice. This VR application of the Roman Theatre of Falerone is a valid navigation tool within an archaeological site, rebuilding salient architectural parts for the discovery and recognition of their nomenclature and function. The text boxes help learners to move around in a virtual archaeological site and facilitate the understanding of functions of the various parts that would otherwise not be understandable, being often destroyed and missing, as in most archaeological sites.

The main characters of the methodological analysis are 37 students, 14 years old: 18 (16 male and 2 female) of the first year of the C.A.T. Costruzioni Ambiente Territorio (1ACAT), and 19 (14 male and 5 female) of the first year of the Graphics Course (1AGR). However, our analysis shows that gender has no significant effects on the learning and the use of VR. Moreover, it is worth mentioning the students’ inclination towards the use of VR. Indeed, from our previous study, the students were asked to express their attitude toward technology, demonstrating their willingness and preparedness. Interested readers could refer to [58] for details.

After following a common learning path developed through theoretical lessons and a guided tour of the archaeological remains of a Roman theatre, students are invited to carry out an online test. Both groups studied topics related to the test at home, but with a difference: the first group (1ACAT) could familiarise themselves with the app at home deepening their subject studies, and represents the “VR group”. While the second group (1AGR) consulted and studied using the app before completing the test, and represents the “control group”. Then, the “VR group” has the advantage of the time available to study the content using the app, but the “control group” can consult the app only before the test.

The test was carried out using Socrative,⁶ an online application that allows tests to be carried out while collecting results, data, and statistics related to student learning. All the tests have been done at school during the annual course by the same teacher who holds the course. The Table A.1 in the appendix reports the multiple-choice questionnaire administered to the students, where the first three questions represents the pre-test. This pre-test is the prerequisite and the base of knowledge for both groups.

The evaluation process during the teaching practices must not be understood only as a simple listing of percentages or values obtained from the correct answers at the end of a test. The evaluation is obtained through a path that starts from the presentation of the disciplinary contents, to the choice of the learning methodology classroom lesson, participatory lesson, problem solving, case study, up to the identification of the key elements, of a didactic unit, to be transmitted also through the use of a specific language. The first phase of the formative exam follows with queries, summary diagrams and short summaries of the main concepts to allow the students to learn the new explored ideas and make the topic their own. The final exam (which can be oral, written, graphic) contains various evaluation elements of the learning process. The types of test have different parameters of information that make it possible to express an opinion on the level and degree of preparation achieved by the student as a whole.

To assess the levels reached by the students, questions were asked referring to BT levels [35]. These were investigated and classified according to six levels of cognitive process competence: remember, understand, apply, analyse, evaluate and create (Figure 3).

To simplify information collection, the levels were grouped into three descriptors that were used to estimate questionnaire answers, as shown in Figure 4.

In Table 1, it is shown how the levels of knowledge have been grouped and the meaning of each of Bloom’s descriptors and our descriptors to better explain our choice. The choice of adopting BT depended on making the introduction of evaluation metrics for VR consistent; specifically, this taxonomy is widely adopted in the Italian higher schools, and it is the one used by the teachers to evaluate the results of an examination. Thus, it has been transferred into descriptors for the evaluation of learning with the VR application.

At this point, we needed to codify formulas for reading results obtained from the learning tests administered at the end of the app-using process. The level of investigation was expanded through a comparative reading of individual questions (Table 1) by assigning indicators and descriptors for understanding the understanding of students’ information acquisition and students’ ability to transform that information into transversal, interdisciplinary, and metacognitive skills. Merely collecting the correct answer results from the two class-sample groups and a general comparison was not sufficient; instead, it was necessary to analyse question types. The content and information provided by the app on the Falerone Theatre were developed by the teacher and included in a curricular educational path. In this way, students acquired a specific technical language necessary to understand the more complex and articulated content of the discipline, preparing for the continuation of their training activities.

The questions proposed for the learned content test were elaborated on by the teacher, articulated using different indicators within the requests that can be summarised as follows:

The scale of values given to each indicator ranged from 0 to 3, based on the weight the indicator assumed within each question. The scale goes from 0 to 3 because the minimum value 0 corresponds to the question that does not contain elements of the indicator, while the maximum value 3 corresponds to the question that contains all the elements of the indicator, as shown in Table 2.

The questions were structured in order to be classified, according to their complexity (greater or lesser) and type of request, within the research descriptors. Such descriptors were defined as follows: questions requiring a simple mnemonic application of requested content were marked with M (M descriptor), those requiring content reprocessing via expressed request interpretation were marked with T (T descriptor) and those requiring a careful content analysis for a broader contextualization were marked with D (D descriptor). The research proposed by [60, 61, 62] validated the internal questionnaire consistency, Cronbach’s reliability test [63] was carried out using its alpha coefficient to determine interesting result reflections obtained from the two student classes. The latter expressed a measure of the relative weight of variability associated with the items regarding the variability associated with their sum, as in (1). The minimum recommended value for alpha was between 0.60 and 0.70 to ensure sufficient internal consistency of the investigation tool. Values between 0.70 and 0.80 showed fair internal consistency.k = number of item;

\(\sigma _i^2\) = variance of each item;

\(\sigma _x^2\) = total variance of the test;

The values of reliability associated to \(\alpha\) are the following:

The initial analysis, shown in Table 3, indicated the average results achieved by the 1AGR class (68.5%) were better than those achieved by the 1ACAT class (66.4%).

Table 4 represents the results of the statistical analysis obtained considering data in Table 3. Mean, variance, standard deviation (StdDev), and standard error of the mean (StdError) are determined from the percentage of each right question.

Table 5 represents the statistical results after averaging the statistical indicators on the two groups (1ACAT and 1AGR).

Moreover, to compare the rate of correct answers, we have proposed both Figure 5(a) which represents a comparison between the correct answers given by the two groups and Figure 5(b) that graphically reports the percentage of correct answers for each question.

The weight of each indicator assigned to each question by the teacher is shown in Table 6.

The test was carried out after allowing 1AGR (“control group”) to study app contents shortly before the test, while 1ACAT (“VR group”) could study the app contents at home (i.e., traditionally) with longer times and procedures. A more detailed analysis of the exact answers collected was performed. Questions were associated with the three previously defined indicators by simplifying RBT. The percentage of correct answers was then multiplied with the weight of each indicator and subsequently divided with the type of indicator, to obtain a comparative analysis of the individual values. The pairwise comparisons for the pre-test type revealed no statistical significance, indicating that the starting level in the two groups was equal.

Table 7 is obtained by combining the values of Table 3 and Table 6. The values of Table 3 have been updated multiplying them with the weight assigned to each indicator and determined according to Table 2. Table 8 reports the correct answer percentages for each question multiplied by a weight of 3 for each indicator.

For the M (Mnemonic) indicator, the “control group” obtained a better rating than the “VR group” with a difference of about 13%. Being able to study app content just before the test favoured the “control group”, enabling easy storage of information containing numeric data, historical character names and simple definitions.

For the T (Transversal) indicator, the results were 3% better for the “VR group”, indicating a different response path being taken. A general count of the exact answers obtained did not permit detailed highlighting of educational values for individual questions administered, as stated previously. The indicator detected profound acquired knowledge reprocessing ability that allowed the readjustment of its founding cores to other contexts, including transversal ones. Each question required reflection on content already provided to students during previous lessons and different teaching units but was still preparatory to the activity in question. The 1ACAT class, processing app information, had time to assimilate the content and establish necessary cognitive connections with what had already been learned in previous lessons. Study time was the variable that favoured the best results for the 1ACAT class in this indicator.

For the D (Disciplinary) indicator, the “VR group” obtained 4% better results. Proposed questions contained topics covered widely in class and were deepened with images and examples. As such, both groups had the opportunity to learn essential content, connections, and didactic meaning. By detailing the weight within the administered questionnaire, a better learning result was obtained by the “VR group”, who studied at home and were able to reconnect the app content with the lesson content. Therefore, the app enabled deepening of acquired knowledge and content validation by crossing referencing information learned in the classroom with that contained in the app.

To provide additional reflective elements, additional analysis of the data collected through the indicators was carried out. The collected results were added by multiplying the percentage of exact answers associated with the maximum value of each indicator (i.e., the weight of 3, as shown in Table 3).

The weights of the answers provided by 1AGR students are clearly higher for the mnemonic indicator (53.1%) while the other indicators show 1ACAT students surpassing their colleagues: 52.9% and 52.5% for transversal and disciplinary, respectively. The weighted indicator category totals allow further validation of the differences between the results achieved. In Figure 6, a comparison between totals is obtained by adding the results of the exact answers multiplied by a weight of 3 for each indicator. In Figure 7, these values have been translated into a percentage. The mnemonic indicator values show the 1AGR class obtaining the best results by experiencing the app shortly before completing the questionnaire. For the transversal and disciplinary indicators, the 1ACAT class earned the best results by using the app to study at home, extending learning times, and enabling app content comparison with class content.

Returning to the levels indicated in RBT, the mnemonic indicator refers to the taxonomy’s first level (remembering) where the cognitive dimension consists of the basic thought skills of recognition and memorisation. This ability does not require students to employ analytical skills, such as understanding and elaboration, that are prerequisites for achieving more elaborate and consistent levels of knowledge, both in content and over time. These attitudes and processes were detected in higher skill levels (such as understanding, analysis, and evaluation), which our indicators call transversal and disciplinary. Achieving these levels indexed the ability to interpret acquired data through the learning process, implementing one’s own set of skills with personal and creative reprocessing to achieve a more elaborate and complete level of knowledge that can be spent in a transversal and original way.

Our research showed that several factors contribute to achieving knowledge levels in the learning process. Our study’s use of VR and knowledge about constituent parts of a Roman Theatre encouraged greater student involvement. The didactic unit, which develops specific historical and architectural knowledge content of an artefact, received more precise and productive reading involvement app usage, proving that the study of architectural history (as any other artistic expression) is favoured by a combination of image and technology use enabling immersive reading. This positive result is demonstrated by the average percentage of exact answers obtained (Table 3 and Figure 5(a)). Values were between 66% and 68%, confirming that the topic was sufficiently acquired by all students. The content learning path was therefore facilitated with VR app usage, allowing all students to reach satisfactory levels of topic knowledge [64, 65].

The model created for KPI definition was applied to individual questions administered in the questionnaire by associating each of them with a weight (from 0 to 3), classifying the link with each described individual level. This enabled the collection of more detailed information about the real value of the correct answers, permitting the deduction of the actual knowledge levels achieved. It is not exhaustive to collect correct results, but it is fundamental to analyse each answer’s value within RBT, which allows a more precise declination of learning levels. The use of the app with different timing allowed the definition of different levels of learning content (Table 7 and Table 8). The added value, constituted by the introduction of VR technology in reading on the architectural artefact, favoured a study approach differentiation whose fundamental variable was defined by the study time factor.

The questionnaire administered to the students at the end of the teaching activities was structured to include questions requiring different acquired skills. Some only needed mnemonic skill application, facilitated by the use of images present in the questionnaire that stored content. Other questions required learners to use synthesis or content processing skills. An initial analysis revealed the best results being obtained by the class using the SmartMarca application before the assessment. By contrast, home study by both groups was supported by the use of VR technology, allowing the distinguishing of different levels and types of content acquisition. Application use at home enabled focusing on information and its most complete reworking, thanks to more relaxed study times and a comparative mode contrasting book and app content. The combined use of traditional study and VR application within an adequate study time leads to better knowledge and skill results. The class group that was able to review the app just before the test indicated a greater ability to support purely mnemonic topics, such as dates and simple definitions. The skills reached by the students who studied at home with the app, referring to RBT, attest to both conceptual and metacognitive knowledge (Figures 6 and 7). The students were able to transfer what they learned to other situations and forms, showing a deeper and more structured learning profile. Thus, the results validated the use of digital technology in teaching, demonstrating experiential value and student involvement [23]). Considering the small GAP in size of using VR-based learning, our findings are in line with Wu et al. [66]. However, in both cases the percentage of correct answers is higher than 60%, which is over the mean with a traditional lesson. This value has been reported by the experience of the teacher that compared the results of the previous years, when the test over the same argument was done without the application.

Our research, which involves the definition and application of performance indicators within a questionnaire, has shown that the topic, which study is distributed over a long time and supported by VR application use, deepens and strengthens student motivation while providing more solid and aware knowledge levels. Though the levels reached, upon initial analysis, appeared inferior, they were proven to be more performing and complete. Correct answers were more evenly distributed and all students had an average homogeneous preparation validated by Cronbach’s alpha formula. The teacher is understood to provide an irreplaceable function of the primary actor of knowledge transmission as has also been investigated in a recent review [67]. In-depth activities carried out with VR favoured focusing on founding topics and conceptual re-elaboration of the topics dealt with. It was fundamental for students to manage information with the required learning times and possess the metacognitive process that led to positive questionnaire results [16]. The evidence of our study corroborates Spector’ [68] findings, which affirm that advances in educational technology do not guarantee improved learning, and ‘the focus should be on learning rather than on technology’. Our research was conducted following this philosophy, and we answered this question by making an analysis of the results obtained, considering the responses of the two groups, VR confirmed its capability in engaging students, evidenced by their inclination to the research experience; however, the teacher still plays a key role. All the tests conducted within the related teaching experiences have always confirmed that the intervention of the teacher, assisted by the technological tool, has allowed the achievement of better results, in terms of learning. Only within a relationship between teacher and student, in a mechanism of mutual exchange, the learning experience can take place. Moreover, the role of the teacher is important not only to improve the basic and preliminary skills, but even in the creation of the contents, which can be easily modified and adapted to the topic thanks to the digital tool [58].

Our research has a number of limitations that nevertheless constitute the backbone for further developments and investigations. Each evaluation process in the didactic field requires a preliminary definition of disciplinary didactic objectives, which must be pursued throughout the whole teaching-learning process. For this reason, it was necessary to establish topic characteristics for evaluation. RBT levels should be structured within any questions given to students, which means each question should contain information to establish the knowledge level achieved by a student beyond the simple repetition of information received. Each question must then be assigned a separate weighting according to indicators established by the teacher. This process should be repeated each time one undertakes such data collection; while time consuming, it can prevent significant exploitation of the proposed methodology. Moreover, there are different levels and possible didactic methodologies for verifying learning and expressing subject assessment. Generally, all this is expressed with a judgement that refers to a docimological scale, but every teacher knows that within such simple numbers there are several elements of judgement and synthesis related to the path a student has taken to reach their level of knowledge. Each class group also constitutes different and difficult to classify working contexts. Therefore, teaching models are not always fully repeatable. The most receptive class will not need detailed goal setting, but will more easily reach articulated and deep levels of knowledge. Otherwise, in scenarios with classes possessing learning difficulties, the work will be slower and reach more basic levels of knowledge.

The added value of using VR will facilitate knowledge transmission, but it will be articulated according to different contexts. Diversifying the didactic is more necessary in light of the growing presence of students with dyslexia, specific learning disorders (e.g., D.S.A.) or special educational needs (e.g., B.E.S.), as also shown in a work similar to ours, but in a musical context [69]. Each teacher, regardless of discipline, thus requires defined minimum learning objectives and course planning to facilitate teaching all students. Consequently, disciplinary objectives and relative learning levels require remodelling according to individual situations, making non-uniform evaluations necessary and further limiting the value of KPIs defined in our research.

In light of the above-mentioned consideration, we can affirm that the definition of a well-established baseline (e.g., the RBT) constitutes only a starting point for question formulation. It must be followed by weighted attribution identified in relation to defined indicators. The aim of this research, thus, is to provide a possible method that can be rearranged to other learning taxonomies and, obviously, should be moulded according to the topic. The learning process in the DCH domain is favoured by the use of visual contents that constitute a privileged vehicle for information transmission. VR has been a strong aid for the discernment capacity (namely transversal and disciplinary indicators) facilitating their systematisation in the process of re-elaboration carried out by the students. Nonetheless, further studies are needed to confirm these results even in other domains. The need for a transition from a subject-specific context to a discipline-specific context is highlighted by several studies [70]. This work moves in this direction, by proposing a method for the DCH, which will hopefully produce collective evidence on VR benefits. However, further testing is required to assess the validity of the proposed KPIs. Finally, as demonstrated by the recent literature [67], data collection methods have been based on questionnaires, while difference inferential statistics has been the primary analysis method. Thus, the proposed method makes a step beyond the current state of the art, by providing a method that, despite exploiting the existing baseline of evaluation in high school, makes it for the first time comparable with a VR-based learning approach. Digital tools are nowadays ready to gather huge amounts of data, and in the future we are confident that our platform will collect a larger dataset, which will make our findings more reliable.