Open City Museum: Unveiling the Cultural Heritage of Athens Through an -Augmented Reality Based- Time Leap

Cultural heritage, which is described as the values, knowledge and customs passed from one generation to the next one, is of paramount importance as it shapes national identity [1]. However, as humanity proceeds in a globalized culture with common principles, that heritage turns out to be crucial, contributing to the new common value system that is currently shaping [2]. Monuments are the main focus for both tourists and local people, as a way to learn more about their place and its history. Having reached a great technological point and surpassing the era of personal Digital Assistants (PDAs) the question seems to be how we could provide people with a better medium to get informed and experience the history of a monument.

One of the most promising technologies, thought to prevail and change the way that people interact with others as well as with their environment is Augmented Reality (AR) [3], especially considering the fact that smartphones have a growing processing power to facilitate that technology. It is known that has a great impact on people’s daily lives like retail [4], education [5, 6] and tourism [7]. As a result, a great challenge for this technology is to redefine the experience of visiting a monument or cultural heritage in general [8], providing more information, entertainment and in essence, a better understanding of the monument. In this context, Open City Museum (O.C.M.) is designed and developed through an holistic approach to achieve these goals.

O.C.M. is an AR application trying to feature Athens’ immovable cultural heritage through a virtual time leap. This project provides a more interactive way to explore and enjoy the city’s array of cultural heritage monuments through the overlaid information that AR provides. It includes predefined routes based on criteria like the historical era, the distance or even user’s available time. In those, the included monuments are not the most famous ones like Parthenon or Erechtheion; instead they are those “hidden” in neighborhoods of Athens that people may walk by them without paying much attention. The intention is to shed light on their history and redefine their importance for the locals and in parallel, make visitors enrich their experience with the cultural “weight” of these sites, as they wander around the city. In this way, not only do they discover these antiquities while making a virtual time leap but they also explore the city and its overlapping layers of history; its streets, its people and its vibes. A shared belief is that monuments’ significance can only be experienced in the context of the culture and the people surrounding them, a preconception that we try to utilize to create a unique sensation through our app.

Focusing more on time leap, which distinguishes this app from any other, it utilizes an abstract representation of how monuments looked like instead to the usual efforts of strict accurate representation, by processing layers of visual information in a different way. The advantages of this lie on the fact that more techniques in imaging can be used as well as it can utilize AR better and thus, have a more powerful result. Instead of focusing on excessive details that also require more resources to run properly, it focuses on expressionistic tools that aim to address also the emotional part of the brain. In this way, the user not only gets informed but also has an emotional arousal, an emotional connection with it creating a unique experience that makes him understand the area’s Genius-loci [9].

Apart from that, the basic principles of UI/UX were deployed while designing the app, intending to maximize user experience in the application. Discovering visitors’ needs and wills and how they can be satisfied was crucial leading to having interviews with people that could be O.C.M.’s users according to their personas. Implementation of the app was of paramount importance given the fact that markerless object recognition was utilized and what is more, in outdoor space, approaches which are still very challenging in this field. Moreover, the suggested utilities from UI/UX had to be implemented, too, such as navigating users to the monument.

As it becomes evident, our clear goals regarding the representation of the monuments and the gain of the user, urged us to do cross-disciplinary research, trying to achieve the best possible human computer interaction. In the following pages, methodology and workflow are analyzed; achieving the virtual time leap, designing a user friendly AR application focusing on cultural heritage and its implementation with state of the art game engine and AR libraries.

2.1 Visual Layering Theory

The approach presented for the creation of AR apps is based on the way the human brain receives, perceives and memorizes environmental information. Since AR applications superimpose information on “top” of reality, the created mixed visual experience could lead to the assigning of meanings and connotations, enriching the ability of images to become memories. An individual memory is comprised by a collection of data that combines explicit and implicit information. A memory carries information that mixes logic and emotion creating a bit of a place, that holds a very specific value for an individual, and can be the initiation of an immersive experience, meaning that it can start the process of “daydreaming”.

In this perspective, the main target of the applied methodology is to augment the ability to retrieve personal memories through visual “triggers” and facilitating the initiation of small immersive personal narratives to each viewer. The main tool to achieve that is to enhance the perceptible visual information by decomposing the info-rich structure of an image and reconstructing it in six discrete visual layers. Kaplan & Kaplan in their attention and restoration theory (ART) mention the limited capacity of a person’s voluntary attention [10]. Voluntary attention focuses on the important parts of a composition and rules out the irrelevant background information. This visual layering process, that was developed at the Transformable Intelligent Environments Lab of T.U.C., is based on the limited capacity of the human brain to process visual information and provides an abstraction tool with the ability to optimize the communication between the context and the user (Fig. 1).

Visual layering diagram

Full size image

More specifically, image layers are characterized as active or passive based on the characteristics of the contained information, as well as their semantic value. Images are composed of at least three active and three passive layers. The importance is determined by the ability of the human brain to recognize dominant figures, moving objects and foreground action as the more dominant elements of a scene and the surrounding information as a background [11].

Animated objects are considered as the protagonistic elements of a visual scene and belong in the first and in some instances second layer of an image. Fundamental structural elements supporting the main action (e.g. cars, objects, etc.) belong to the third active layer. The active layers construct the main action and are easily perceived by the human brain.

The passive layers provide points of reference that address the right frontal lobe of the human brain and create the general atmosphere of the theme. The overall atmosphere of a place is a synthesis of multiple factors, often involving more than the five Aristotelian senses [12]. Surrounding elements, such as colors and textures, are also classified as ambient background information and provide the “sense of place”.

The visual layering methodology aims to enhance and alter the semantics of each layer and immerse the user via an educational visual experience. In order to make each visual layer more prominent, a blur effect is used on the surrounding elements [13]. The immersion is enhanced by introducing different visual styles according to the era each photo refers to. These stylistic approaches are selected according to the color, texture, and style that represent each historic period.

2.2 Strategic Approach

According to Table 1, research in historical archives, literature review, and interviews with locals take place before the first app development stage. During the third stage, the content has already reached the required level of analysis and can be tested inside the application. The application is being developed since stage 2, since further effort is required to integrate the graphical content with the app.

2.3 Pathfinding

Based on the path of a common touristic route in Athens, close to the more prominent monuments (i.e. The Acropolis), the least prevalent and often overlooked - but still important-monuments were chosen for this analysis. Accessibility (walking distance, cycle, bus, and adjacent public transport hubs) is a major classification factor. The impact radius of each monument, which refers to the influence it has on the surrounding areas, is also holding an important role in the selection process (Fig. 2).

Pathfinding diagram.

Full size image

2.4 Topological Analysis (Fig. 3)

After the determination of the proposed routes, each monument is classified according to these factors:

Topological analysis.

Full size image

1. 1.

   Location

2. 2.

   Relation to other monuments

3. 3.

   Accessibility

4. 4.

   Surrounding environment

5. 5.

   Viewpoints

6. 6.

   Current Decay and Restoration Status

2.5 Processing of Historical Data

Historical archives and archeological services offer the most accurate representations regarding the evolution of each monument in time. However, O.C.M. recognizes the numerous challenges in accurately portraying a significant piece of architecture in different chronological periods. The materiality, the surroundings, and the sound represent some of the most significant parts of the environmental composition. These elements along with colors, textures and artistic styles, are able to achieve the anticipated holistic experience which will immerse the user into the scene (Fig. 4).

Historical data diagram.

Full size image

2.6 Visual Representation Timeline

The sequence to present the visual transition to separate historic periods is presented below:

1. 1.

   the monument is isolated from the surrounding environment

2. 2.

   the background is blurred directing the attention to the central theme

3. 3.

   a sepia effect is applied to the image

4. 4.

   the characteristics of the first era are emphasized by outlining the main differences

5. 5.

   an abstract collage composed of old pictures overlaying the present situation is introduced

   1st time leap

6. 6.

   the blur and sepia effects are reduced

7. 7.

   an abstract collage composed of old pictures overlaying the present situation is introduced the characteristics of the second era are emphasized by outlining the main differences (this step refers to the fourth step)

   2nd time leap

8. 8.

   the blur and sepia effects are further reduced-color saturation is closer to the original image

9. 9.

   a collage composed of old pictures overlaying the present situation is introduced elements from the specific era are added into the scene (e.g. people and vehicles from that era)

This process briefly describes the timeline of events during the AR representation and not the post-processing steps of the visual layering methodology (Fig. 5).

Visual representation sequence example. The Choragic Monument of Lysicrates, Plaka, Athens

Full size image

Nowadays, it is essential to deploy a UX/UI plan to make applications as friendly as possible for the user. User experience (UX) assists people to achieve their goals and take value from the product based on their needs. To make this happen, design thinking is utilized. In the first phase which is called “Empathize” or “Strategy”, a plan is made in order to discover users’ needs and set business goals. The second is called “Define” or “Scope” in which the findings from the previous step are collected and analyzed, the problem statement is defined alongside with the content and the requirements of the application. Then, we move to “Ideate” where brainstorming takes place. The solutions from the previous steps lead to prototypes which are given to users for feedback. On the other hand, user interface (UI) is not only about how the app looks like or the colors but it handles the interaction between user and the application; is the design of interfaces of machines in order to accomplish an easy and efficient experience for the user in terms of accomplishing their goals.

O.C.M. with the above process achieves a more interactive, friendly user experience with easy interactions for all different types of users (Fig. 6).

UI/UX flow diagram.

Full size image

3.1 Empathize

When tourists and travellers search for a new attraction, they are trying to learn the history of a place or to get some intriguing facts. Searching on the web about famous attractions in the city proves to be a hard and time consuming process. Furthermore, the amount and the content of information is not sufficient as it is obsolete or shortly described so, people can not feel the vibe of the ancient period. The lack of interaction in that is evident and should be upgraded. In addition, navigating in the city and getting informed demands more than one application which is certainly not practical. All these state some of the problems that a tourist has to face when visiting a monument.

At this point, it is meaningful to proceed with the Competitive analysis. The last few years there is a growing number of applications using AR in cultural heritage. There are two kinds of applications: the first one is used indoors, in places like museums and the second one outdoors. Being interested in the second category, one of the most noticeable projects is ARCHEOGUIDE which creates routes and reconstructs the ruins of ancient monuments [14]. In addition, more similar applications have been proposed like Chang’s et al. who proposed a geo context based embedding visiting system pointing out that augmented reality had a major impact on the experience [15]. Another interesting approach was project AR-CIMUVE Augmented Reality for the Walled Cities of the Veneto which had educational purposes for primary and middle school students which among others incorporated 3D models [16]. In their application, Pacheco’s et al. [17] used orientation sensors and GPS to track the user’s position and also navigated the user in order to point out the differences in an old german concentration camp. A project that included multiple types of interaction that were used in order to create a story-telling by visiting multiple locations was VisAge [18] offering a quite different experience to the user by having its own content. Last but not least, recently, Panou designed and implemented an AR application that could be used as tourist guide for the city of Chania, superimposing 3D models of monuments in the real ones [19].

Having carried out the research about similar applications, the next step was to have stakeholder interviews. The aim of that was to make clear of the business goals and also, have the same perception about the application: its objectives and the way to achieve them. As a result, some initial goals and functions were set, an outline of the technologies to use and what is more, the type of the customers: tourists and citizens of Athens who want to discover the ancient places of the city. To be our app effective, an important step was to learn from them: what they desire and most importantly, what they need.

3.2 Define

Hence, it was necessary to delve deeper into the user’s behavior. More specifically, to find out pain points, motivations, how they define an experience as good, which applications are mainly used and how the user solves the emerged problems before and during visiting. The interviews included 7 people with experience of traveling and they were conducted through telephone. Most of the people search for the most famous places and they group them together making a “day trip” list so they have a plan for each day. They usually pin famous places on Google Maps, which they also use to navigate. Almost everyone uses Trip Advisor to find museums, attractions and places according to their interests around the city. When they were asked about problems on the procedure or pain points, they answered that they need more filters about the exact pricing, the opening hours they can visit and the public transportation. As it seems, choosing a monument to visit is a complex decision based on a lot of information and apart from this, an important factor is the monument’s proximity to their accommodation. Regarding their experience during the visit some of them mentioned that they get only the basic information from the guide or from a travelling site. As a result, they do not engage, even though they are trying to imagine how it was in the past. Interestingly, one of the participants suggested that a podcast would be a good idea (resembles the PDAs) while another participant said that he had used AR based on QR codes on the exhibits of a museum finding it a nice experience. From that it becomes apparent that users need a more vivid interaction, especially if it includes multimedia like audio and visual representations. Based on their answers and on the previous phases, two personas were created, one of which is described in Fig. 7.

Persona example

Full size image

Before moving to UI testing and to UI implementation, it is important to grasp all the possible constraints that can affect how the O.C.M. is structured and implemented. The first restriction was the size of the time leap feature. The size could not be very large as it would raise an issue regarding storage and battery from the increased energy consumption. So, the decision was made to have a sequence of pictures, as described before. Another significant restriction could be the constant changing conditions in outdoor spaces where differences in ambient lighting, whether conditions and people passing by could affect the AR recognition of the monument. The application will need to inform users about potential environmental restrictions.

3.3 Ideate

The above process of personas and the problem statement offered a good insight about the users’ needs and selecting the business goals for the project. One of the most important requirements was that the application should spark the users interest instead of just providing historical information. This can be achieved with the use of AR and by creating the emotion of time leap. Another requirement was to add a navigation feature in the application. This way the users do not need another application for navigating to the monument, like Google maps or similar applications. When setting the business goals, a new requirement emerged where users should have the ability to review and upload photos of the monument in order to have more information to decide if it is worth visiting or not. It was crucial to prioritize the needs and emphasize on those features that would characterize O.C.M. and make it distinguish from the competitors. Consequently, in this first version of O.C.M. the first two requirements were implemented whereas the last one, was chosen for a later version, as extra features that will provide added value. Having the review of the first release, would help us assess their importance and also, fix any existing bugs.

3.4 Prototypes

At this point, the examination of the architecture of the application is the next step. For that reason were generated prototypes of the application based on the previous phases that include users needs and requirements.

3.5 Testing

In the last phase of design, there are Hi-Fidelity prototypes which should be tested. Through this process the usability of the app is checked and the ability of the user to complete the main scenarios. Hence, user scenarios were written and the user was trying to accomplish them. If one of them takes a lot of time for the user to find out or searches for it without success, then this is considered as failure and must be redesigned. Some of the tasks users were about to implement were to visit a famous monument and get directions, search for a monument that doesn't remember exactly the name, find some nearby monuments and to find some routes. Users generally achieved them easily. The scenario that made users more sceptical was one of the main stakeholder goals, the predefined routes. One reason is that users are not used to having predefined monuments or famous places included, thus, this had to change. Overall, in this way, users pointed us out how this app can be functional and cover their needs, which made implementation more clear.

In order to select a technology for the application implementation a variety of factors were considered. One of the main considerations was the ease of integration with AR technologies that the application was going to use, since AR is a core feature of the application. Another factor to consider was the extensibility and cross platform publishing of the developed application. Even though the first version has been tested on Android only, in the future it will be made available for other platforms. This application tries to enrich the environment of the monument with visual effects that are inspired by different historical periods and by employing the visual layering technique that is described above.

With these factors in mind the decision was made to make use of Unity game engine. Many AR frameworks have integrations with Unity, so the choice of Unity enabled the application to have flexibility in the selection of the AR framework. Unity is cross platform and allows publishing an application in many platforms. So, combining this with the ease of use and the community size of Unity made this a solid choice to develop the application.

Another very important feature of the application was the map integration. The first tests for integrating maps used google maps API. During the first stages of development it was discovered that the Google Maps integration with Unity is not targeted for small (indie) developers and it is not free. Also from information on public forums [20] it seems that there is some middleware code that needs to be developed to make the google maps easy to use it Unity. Another final drawback is that even though Unity supports a variety of deployment platforms the use of google maps API supports only Android and IOS [21], which restricts the future expansions of the application. This led the development team to research alternative map apis to use for the application. The alternative chosen was Mapbox which uses OpenStreetMap as source data for the Mapbox Streets [22]. The integration of Mapbox in the application code was very simple and the getting started information that was provided was very accurate. This made the decision to switch the original maps framework easy to implement and test. Finally the last major component of the application that needs to be analyzed is the database that will contain all the information about the users and the monuments. The framework that was chosen for the database is Firebase realtime database [23]. The greatest benefit of this framework is that it does not require any backend server setup. It is a no-SQL database that works with JSON (Javascript Object Notation) objects that requires minimal setup to get started and it has support for Unity. So this made it very easy to integrate in the application code.

The AR framework that was selected was Wikitude SDK, due to the cross platform support and Scene Recognition ability. It enables the tracking of large objects that are viewed from a variety of angles. The generation of the Tracking Object is made in Wikitude Studio, an online application of Wikitude. The generation follows the principles of photogrammetry. The final object is a 3d representation of the model, that can be tracked form a variety of angles and distances, even in a noisy background. The Object Tracker of the monument was generated by a set of photographs. Each one of the photographs was from a different angle or distance. The first step was to remove the background from the photographs, a process that was made in Photoshop. Before the generation of the Object Tracker in Wikitude Studio (Fig. 8), the set of the photographs were tested in 3df Zephyr (photogrammetry software), in order to evaluate the extracted model and supplement with extra photographs if needed. The final step was to generate the Object Tracker in Wikitude Studio and import the exported model (.wtc) into Unity.

The AR framework that was initially selected was Vuforia AR SDK. During testing the first versions of the application with this framework it was discovered that it is not working efficiently in outdoor spaces where the target has changing weather and lighting conditions. So the decision was made to switch the AR framework to Wikitude AR SDK.

The following activity diagram (Fig. 9) analyzes the high level activities that a user can perform in the O.C.M. application. The first screen of the application is a welcome screen that prompts the user to login or register (Fig. 10a). In case of a new registration the application supports three ways to register a new user. The available registration options are: direct registration, register with a Google account and register with Facebook. Once a user is registered and logged in they have access to the monument list. From this screen a user can see the map of the monuments that are nearby (Fig. 10c) and is able to select the monument that will be loaded to start the time leap sequence.

Point cloud of object target of monument, in Wikitude Studio.

Full size image

The software minimum requirements for the O.C.M. application on android is Android 7.0 (API level 24). This is enforced by the external APIs that the application depends on. Mapbox has this minimum android requirement. Where Wikitude sets the minimum version to Android 5.0 (API Level 21). For hardware minimum requirements a mobile phone needs to have a camera, a gps module and a gyroscope, support for OpenGL 2.0 (or newer) and a CPU a capable armv7a with NEON support (or armv8a).

High level activity diagram of O.C.M. application.

Full size image

O.C.M. application screenshots.

Full size image

This paper describes the design and development of an AR application that focuses on unveiling the cultural heritage of Athens using a virtual time leap. The proposed application suggests an innovative way to suggest which monuments a user should visit, navigate in the city and represent monuments’ history providing users with a unique interaction.

The visual representation of the monument originates from TIE Lab’s Visual Layering Theory which takes advantage of passive and active image layers in order to assign different meanings and semantics to each one of them. Those play a key role in the creation of visual scenes as they project the desired atmosphere for the virtual time leap. Of course, the choice of the monument is important as it relies on historical data as well as on the social context of a time period and its surrounding environment. These affect the representation and are taken into account to the creation of small emergent narratives.

In parallel, O.C.M. does not focus only on the provided experience by AR but also utilizes UI/UX principles in order to discover and fulfil user’s needs. Performing interviews with potential users contributed to draw useful conclusions regarding what they need before and during their visits in the monument leading to develop a more complete application concerning the features it provides. In addition, it was easier to implement it as the goals, the plan and demands had become very clear thanks to this process.

The progress that has been made so far in the field of AR assists a lot to create innovative applications. Open source libraries and game engines can be really crucial for the development enabling to track monuments and also superimpose the desired information. Wikitude seems trustworthy for that task and this is the reason that more and more developers use it [19]. The popularity of iOs and Android increases our outreach to more users and Unity makes that very easy to develop for both operating systems. Mapbox seems also a good alternative for Google Maps.

Currently, the first version of Open City Museum is developed containing key parts that shape its character like routes and time leap. From the interviews and our study based on UX, it is known that it is necessary to expand to social media and create a community. Our future work will be based on that: make people interact with each other and also encourage them to share their own personal view of the monument. Allowing users to generate their own content could be a crucial step to increase engagement and interaction with it. This shall be the next crucial update in a later version alongside with suggested improvements that will emerge later by using the application. Still, the proposed application is a characteristic example of human computer interaction enabling people to enjoy the long history of Athens’ monuments and engage them with a more interactive and emotional way offering a unique experience.