Seminar Report 2020 PDF
Seminar Report 2020 PDF
Seminar Report 2020 PDF
On
B. Tech.
in
Environment
Physical environments
Virtual environment
Continuous discovery
Environmental limitations
Experience size
Responsive Playspace
Types of movement
Accessibility
Modeling
UX
Initialization
Audio exploration
Multiplayer Experience
Interface
Errors
Advantages of AR technology
Disadvantages of AR technology
Applications
1. AUGMENT
2. Sun-Seeker
3. ARGON4
4. AR Browser SDK
5. Pokémon Go:
6. REAL STRIKE
7. AR GPS Drive/Walk Navigation
8. AR GPS Compass Map 3D
CHAPTER Ⅲ: CONCLUSION
SPECIAL SECTION ON HUMAN-CENTERED SMART SYSTEMS AND TECHNOLOGIES
Received December 28, 2017, accepted February 11, 2018, date of publication February 21, 2018, date of current version March 28, 2018.
Digital Object Identifier 10.1109/ACCESS.2018.2808326
ABSTRACT Shipbuilding companies are upgrading their inner workings in order to create Shipyards 4.0,
where the principles of Industry 4.0 are paving the way to further digitalized and optimized processes in an
integrated network. Among the different Industry 4.0 technologies, this paper focuses on augmented reality,
whose application in the industrial field has led to the concept of industrial augmented reality (IAR). This
paper first describes the basics of IAR and then carries out a thorough analysis of the latest IAR systems
for industrial and shipbuilding applications. Then, in order to build a practical IAR system for shipyard
workers, the main hardware and software solutions are compared. Finally, as a conclusion after reviewing
all the aspects related to IAR for shipbuilding, it proposed an IAR system architecture that combines cloudlets
and fog computing, which reduce latency response and accelerate rendering tasks while offloading compute
intensive tasks from the cloud.
INDEX TERMS Industry 4.0, augmented reality, industrial augmented reality, Internet of Things,
cyber-physical systems, industrial operator support, smart factory, task execution, cloudlet, edge computing.
I. INTRODUCTION that provide powerful tools that support the operators that
The shipbuilding industry is aimed at providing integral undertake tasks, helping them in assembly tasks, context-
solutions for delivering fully operational vessels together aware assistance, data visualization and interaction (acting
with their life-cycle maintenance. Shipbuilding main working as a Human-Machine Interface (HMI)), indoor localization,
areas include, among others, the design and construction of maintenance applications, quality control or material man-
hi-tech vessels, the development of naval control and com- agement. Specifically, AR technology is expected to grow
bat systems, overhauls of military and civil vessels, ships significantly in the next years together with Virtual Reality
repairs, and diesel engine and turbine manufacturing. In all (VR), creating a market of US $80 bn in 2025 [1].
those areas, most companies are transferring the Industry IAR is one of the technologies analyzed by Navantia, one
4.0 principles to their shipyards in order to build Ship- of world’s largest shipbuilders. At the end of 2015 Navantia
yards 4.0, seeking to apply the newest technologies related created together with the University of A Coruña (UDC)
to ubiquitous sensing, Internet of Things (IoT), robotics, the Joint Research Unit Navantia-UDC, which studies the
Cyber-Physical Systems (CPS), 3D printing or Big Data to application of different Industry 4.0 technologies to ship-
improve the efficiency of the many processes that occur in a yards through various research lines. One of such lines is
shipyard. called ‘‘Plant Information and Augmented Reality’’ and has
This paradigm shift will change the way that operators among its objectives to evaluate the feasibility of using
perform their daily tasks. Therefore, they must be equipped IAR to provide information to operators about shipyard
with devices that act as an interface for human-machine processes.
communication and collaboration, and even as a Decision The present paper is aimed at analyzing the latest research
Support System (DSS) that would help to optimize their and the best technologies to build an IAR system for a ship-
actions. Augmented Reality (AR), and specifically Indus- yard. Specifically, its main contributions are the following,
trial Augmented Reality (IAR), is one of the technologies which, as of writing, have not been found together in the
2169-3536
2018 IEEE. Translations and content mining are permitted for academic research only.
13358 Personal use is also permitted, but republication/redistribution requires IEEE permission. VOLUME 6, 2018
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
P. Fraga-Lamas et al.: Review on IAR Systems for the Industry 4.0 Shipyard
literature:
• The article reviews the main characteristics of the most
recent IAR systems for industrial and shipbuilding appli-
cations.
• It provides thorough comparisons on the latest hardware
and software technologies for creating IAR solutions.
• This article also identifies and discusses different use
cases where IAR can be useful for shipbuilding.
• It proposes a novel IAR architecture that makes use of
Edge Computing to reduce latency response and accel-
erate rendering tasks.
The remainder of this paper is organized as follows.
Section II reviews the main IAR concepts to provide a
basic understanding of the underlying technology. Section III
summarizes the main industrial applications of IAR, while FIGURE 1. Example of video-mixed display.
Section IV analyzes the main academic and commercial
IAR developments for the shipbuilding industry. Section V technological giants have been actively developing AR pro-
describes the most relevant use cases of IAR for a ship- totypes and products [12]. Moreover, ABI Research [13]
yard. Section VI compares the main IAR hardware and soft- anticipates a fast-growing market with shipments forecast to
ware technologies for developing shipbuilding applications. expand from just over one million units in 2016 to more
Section VII analyzes the traditional IAR architecture and then than 460 million by 2021 [14]. In addition, according to
proposes the ideal architecture for a Shipyard 4.0. Finally, IDC [15], worldwide spending on AR and VR is forecast to
Section VIII is devoted to the conclusions. reach US$ 17.8 billion in 2018 [16]. Furthermore, products
and services will continue to grow at a similar rate through-
II. IAR BASICS out the remainder of the 2017-2021 forecast period, achiev-
A. ORIGINS, CURRENT TRENDS AND PROSPECTS OF IAR ing a five-year Compound Annual Growth Rate (CAGR)
The term IAR can be defined as the application of AR in of 98.8%. Nevertheless, according to numerous sources
order to support an industrial process [2]. This definition is [17]–[19], it will take a few years until IAR reaches the
broader than the one given by other researchers [3], who maturity level to be fully deployed in industrial applications.
disregarded photo-based augmentations, which have proved
to be effective in industrial environments. B. ESSENTIAL PARTS OF AN AR SYSTEM
Although the origins of AR can be traced back to the AR involves a set of technologies that make use of an elec-
1960s, when Ivan Sutherland performed his visionary exper- tronic device to view, directly or indirectly, a real-world
iments [4], [5], the concept of IAR actually emerged in the physical environment that is combined with virtual elements.
early 1990s, when Caudell and Mizell [6], from Boeing, The elements that make up an AR system are:
presented a head-up see-through display used to augment • An element to capture images (i.e., a Charge-Coupled
the visual field of an operator with information related to Device (CCD), stereo, or depth-sensing camera).
the task she/he was carrying out. It was not until the end of • A display to project the virtual information on the
the 1990s when IAR started to gain some traction, mainly images acquired by the capture element. Display tech-
thanks to the German government, who funded the ARVIKA nologies are divided basically into two types: video-
project [7], [8]. In such a project companies like Airbus, mixed (example in Figure 1) and optical see-through
EADS, BMW or Ford participated in the development of displays [20] (e.g., projection-based systems). There is
mobile IAR applications [9]. Such a project represented an also a growing interest in retinal projection, but its use
important milestone and promoted diverse IAR initiatives. is currently very rare in industrial environments. In a
The research efforts of ARVIKA derived in 2006 in the video-mixed display, the virtual and real information
ARTESAS (Advanced Augmented Reality Technologies for previously acquired with a camera are digitally merged
Industrial Service Applications) project [10], which focused and represented on a display. This technology presents,
on the development of IAR technologies for automotive and among others, the disadvantages of providing limited
aerospace maintenance. fields of vision and reduced resolutions. In contrast,
The first IAR systems were mostly experimental, but in the in an optical see-through display, virtual information is
last years they started to move away from research and sev- superimposed on the user’s field of view by means of an
eral relevant commercial initiatives have been launched. For optical projection system. The hardware used by these
instance, in 2013 Google Glasses [11] caught the attention of two display technologies can be classified as:
a broad audience that was not familiarized with AR or IAR – Hand-Held Displays (HHD). They embed a screen
and gained the enthusiasm of certain industries. Since then, that fits in a user’s hand (e.g., tablets, smartphones).
– Spatial Displays. They use digital projectors to • Techniques based on sensors (e.g., Inertial Measuring
show graphic information about physical objects. Unit (IMU), GPS, infrared, ultrasounds).
These displays ease collaborative tasks among • Image-based techniques that make use of markers,
users, since they are not associated with a single which can be synthetic or natural.
user. • Sensor and image fusion techniques.
– Head-Mounted Displays (HMD). They are the dis- In addition, there are different techniques for image recog-
plays included in devices like smart glasses and nition. Some are based in traditional AR markers, which
smart helmets, which allow users to see the entire usually are square or rectangular markers that contain a high-
environment that surrounds them. contrast figure similar to a QR code. These type of mark-
• A processing unit that outputs the virtual information to ers are usually framework-dependent and the software is,
be projected. in most cases, optimized to recognize them by means of spe-
• Activating elements (e.g., images, GPS positions, QR cific algorithms. On the contrary, Natural Feature Tracking
markers, or sensor values from accelerometers, gyro- (NFT) techniques [22], [23] extract characteristic points from
scopes, compasses, altimeters or thermal sensors) that images. Such points are then used to train the AR system to
trigger the display of virtual information. detect them in real time. NFT techniques have the disadvan-
Regarding the internal logic of an AR system, it is essen- tage of being more computationally intensive and slower than
tially composed by the functional modules shown in Figure 2. other alternatives, and less effective at long distances, but they
In such a Figure it is represented a pipeline where, first, provide a seamless integration of the augmented information
the device camera captures a frame, which is then processed with the real world, since there is no need for placing an
by the AR software to estimate the camera position regarding artificial marker visible in the scene [24]. In order to mitigate
to a reference object (e.g., an AR marker). Such an estimation the disadvantages of NFT, small artificial markers called
can also make use of internal sensors, which also help to track fiducial markers are often added to the scene to accelerate the
the reference object. Accurate camera positioning is essential initial recognition, improving the performance of the system
while displaying AR content, since it has to be rotated and and reducing the algorithm computational requirements.
scaled according to the scenario. Usually, the image is ren-
dered for the appropriate perspective and it is presented to III. IAR FOR INDUSTRIAL APPLICATIONS
the user on the display of a device. Note also that interaction The combination of the latest advances in electronics, sen-
with the image is possible through the Interaction Handling sors, networking, and robotics, together with paradigms like
module and that, when certain local or remote information IoT, enable the development of advanced applications for
is required, the Information Management module is the one industrial systems [25], energy efficiency [26], [27], home
responsible for obtaining it. automation [28], precision agriculture [29], high-security IoT
In this pipeline, the major technological challenges are the applications [30]–[34], transportation [35] or for defense and
internal registration of the objects displayed by the system, public safety [36], [37].
the tracking of visual elements and features, and the devel- Among the enabling technologies, IAR has proven to be a
opment of the information display system [21]. In addition, suitable tool for the manufacturing strategies proposed by dif-
rendering (i.e., photometric registration, comprehensive visu- ferent countries (e.g., Industry 4.0 [38], Industrial Internet of
alization techniques, or view-management techniques) and Things [39], Made in China 2025 [40] or smart factories [41]),
real-time data processing are also challenging, since their allowing workers to collaborate among them, to interact with
performance is key when superimposing graphic elements in real-time information and to monitor and control systems.
the environment in a rapid, consistent and realistic way. Thus, Figure 3 summarizes the most relevant industrial tasks and
accurate, fast and robust registration and tracking are required sectors where IAR can bring value.
to develop AR techniques that allow for determining the One of the IAR most common applications is the assis-
position and orientation of the observer and the real/virtual tance to workers in maintenance/repair/control tasks through
objects. Such techniques can be classified into: instructions with textual, visual, or auditory information [42].
Such an information is rendered ubiquitously, so that the Another area where IAR can help is marketing and sales,
worker perceives the instructions with less effort, thus avoid- since AR demos are usually really attractive when show-
ing the change from a real context to a virtual one where the ing other people a certain scenario or the capabilities of a
relevant data is accessed. product. Therefore, AR has the ability to transform the cus-
Remote assistance is also key when companies have tomer experience, enabling the inclusion of different parame-
machines installed in remote locations. Such machines need ters, options, settings or configurations [54]–[57]. Moreover,
to be monitored, operated and repaired with the minimum an improved customer experience is able to reduce the levels
amount of people on-site. IAR can help by easing remote col- of uncertainty about their choices shortening the sales cycle.
laboration between workers [43], [44]. Augmented commu- Furthermore, it can also be used to collect data about product
nications can also be used for collaborative visualization in preferences. A good marketing and sales example was created
engineering processes during stages related to design or man- by BMW [58] for a campaign for selling its latest electric
ufacturing [45]. vehicles.
Likewise, IAR is helpful for assisting workers in the deci- After-sales service to the customers can be also enhanced
sion making in real scenarios, combining the physical expe- through IAR, since it can guide them through repairs and
rience together with the display of information extracted in can connect them with remote experts [59], [60]. Feedback
real time from databases [46]. Furthermore, IAR can provide can be also obtained by using an IAR interface, either by
quick access to documentation like manuals, drawings or 3D receiving it directly from the customers, or by analyzing how
models [47]–[49]. such consumers have used the product [61], [62].
Well-trained operators are also essential for productive The 3D models provided by IAR are a useful tool for
factories. IAR can help during the training process by giving engineers while creating and evaluating designs and products
step-by-step instructions to develop specific tasks. This is [19], [63]. IAR makes it possible to place a virtual object
especially useful when training workers to operate machinery anywhere and observe at full scale whether it fits or not in
like the one used for assembly in sequence, what reduces the a specific scenario. Moreover, IAR enables providing on-site
time and effort dedicated to check manuals [50]. Thus, IAR CAD model corrections, thus improving accuracy, alignment
can reduce the training time for new employees and lower and other details of the model [64], [65]. In addition, during
the skill requirements for new hires. In addition, it is possible product the different product manufacturing stages, IAR can
to adjust the instructions to the experience of the worker, help during the quality assurance controls and show perfor-
what accelerates the learning process by focusing more on mance dashboards [66].
acquiring skills. Manufacturing can also be benefited from IAR, which IAR
IAR training systems are also useful for preserving certain can deliver the right information at the right time to avoid mis-
practical knowledge acquired by the most skilled (and usually takes and increase productivity [67]. This is especially criti-
aged) workers. What happens in different industries is that cal in dangerous manufacturing tasks, where a mistake may
during the 1960s and 1970s there were massive hirings of mean that a worker gets injured or that a costly equipment
workers that are currently retiring [51]. Such workers take is damaged. In addition, in such situations an IAR solution
with them a lot of experience, knowledge and skills that are could be used as a monitoring and diagnosis tool capturing the
difficult to reproduce in traditional ways. Therefore, IAR information provided by control and management systems,
solutions can include such a knowledge in applications to and sensors [68]. For example, Frigo et al. [69] show some
train new hirings [52], [53]. use cases in aerospace manufacturing processes.
Assembly is one of the processes whose performance can Another relevant welding system is presented in [78],
be improved dramatically by IAR [70]. For example, some which is aimed at training welders. Such a system consists
researchers have studied different methodologies for develop- in a torch, a pair of AR glasses, a motion tracking system and
ing CAD model-based systems for assembly simulation [71]. external speakers. Welding is simulated in real time and uses
In addition, it has already been evaluated the performance a neural network to determine the quality and shape of the
of Microsoft HoloLens for assembly processes [72]. Fur- weld based on the speed and orientation of the torch.
thermore, a comprehensive review of AR-based assembly Another activity that takes place in a shipyard is spray
processes can be found in [73]. painting. To simulate the working environment, in [79] it is
Regarding logistics, IAR can enhance the efficiency of the proposed a system that uses a paint gun as user interface.
picking process in warehouse by providing an indoor guid- Such a gun features force feedback and emits painting sound.
ance system [74]. Note that picking items represents from During training, the student uses this gun to paint on virtual
55 % to 65 % of the total cost of warehousing operations [75], models of steel structures that are shown on the screen of the
which are still mostly carried out through a pick-by-paper AR glasses, showing the results immediately, just upon the
approach. Thus, IAR can give instructions to the workers and completion of the exercise.
guide them using the best route. A system potentially applicable to the shipyard is the one
Finally, after reviewing the state of the art, the following proposed in [80]. It provides AR functionalities on a tablet,
aspects can be pointed out as essential in order to develop a allowing the operator to place virtual geo-referenced notes in
successful IAR application: production modules as a method of communication between
• The use cases and applications selected must provide workers. The central element of the system adds and pro-
added-value services. cesses data from different sources, while the applications run-
• Functional discontinuities or gaps in the operating ning on the tablets retrieve and display such an information
modes that can affect the functionality should be for maintenance and for helping in plant processes.
avoided. Regarding maintenance in industrial processes of the ship-
• Reduce cognitive discontinuities or differences between yard, many of them are described in paper or electronic
old and new work practices. Learning new procedures documents that the operator must read and memorize to apply
may hinder the adoption of the technology. on the ongoing maintenance operations. This task is usually
• Reduce physical side-effects caused by the devices on prone to human errors. To avoid them, the system described
users in the short and long term (e.g., headaches, nau- in [81] presents an IAR-based assistant that makes uses of
sea, or loss of visual acuity). a tablet to indicate step-by-step instructions that include the
• Avoid unpredicted effects of the devices on users necessary information for carrying out an operation. Simi-
unfamiliar with the technology, like distractions, sur- larly, in [48] it is detailed an IAR application that provides
prises or shocks. assistance to military mechanics when performing repair and
• Take into consideration the user perception regarding maintenance tasks on the field, inside an armored vehicle. The
ergonomic and aesthetic issues. prototype proposed uses a wrist control panel and an HMD
• Make user interaction as natural and user-friendly as to augment the mechanic’s natural vision with animated text,
possible, avoiding lapses or inconsistencies. labels, arrows and sequences to facilitate the understanding,
localization and execution of tasks, which include installing
IV. IAR FOR SHIPBUILDING and removing locks and warning lights, or connecting wires.
In the last years, several IAR solutions have been presented Note that, since all those tasks have to be performed in the
to help in the accomplishment of daily tasks in a shipyard and reduced interior space of an armored vehicle, the IAR pro-
in the shipbuilding industry. Some examples are the systems totype makes it easier for the mechanics to locate elements,
presented in [76]–[78], which facilitate welding processes. which is also carried out faster than when making use of
In particular, in [76] it is detailed a system that replaces the traditional documentation systems.
traditional welder’s screen with a helmet that incorporates a Specific issues arise when building ships for the first time.
display where useful information is projected. Through the A common situation is that construction and production pro-
screen, a virtual assistant actively suggests corrections during cesses often overlap over time. When discrepancies between
the welding process and indicates possible errors. the construction data and the actual ship happen, it is nec-
Similarly, in [77] it is described a system to control a essary to modify the CAD models. To solve this problem,
welding robot inside a ship. The proposed system uses an the AR system detailed in [82] allows the user to visual-
interface that allows operators to interact with the workspace: ize the pipe construction data and modify them in the case
the operator receives visual information from a projec- of misalignment or collision. The modified pipe geometry
tion system mounted on the robot and controls the robot can be saved and used as an input for bending machines.
using a wireless controller. Thus, the system is designed To ensure the fitting of a pipe, the system integrates an optical
to increase the operator concentration on the actual task, measurement tool into the alignment process. All these tasks
avoiding the distractions related to the use of a traditional are performed by using an IAR application in a tablet with
screen. a built-in camera that, before the actual installation, allows
TABLE 1. Characteristics of the most relevant HMD devices for IAR (first part).
TABLE 1. (Continued.) Characteristics of the most relevant HMD devices for IAR (second part).
• Due to the limited memory of current IAR hardware Gesture recognition is also useful when ambient noise
devices, their use tends to be limited to reduced areas, prevents voice recognition.
which have to be mapped (usually on-the-fly) by using • Reconstruction of 3D environments to get an under-
different sensors. It is possible to load data dynamically, standing of the surroundings.
as the operator enters a new production area, but that • Overlapping 3D virtual elements (with or without AR
complicates the development and requires fast storage markers).
and processing hardware.
All these features should be addressed by the different
• Since IAR algorithms make use of computer vision
pieces of an IAR Software Development Kit (SDK), which
techniques based in thresholding and color measurement
is responsible for interacting with the hardware of the device
metrics, the lighting characteristics (light of the environ-
to obtain data from the environment in order to show con-
ment, type of light, light temperature) can impact the
textual information to the user depending on the surrounding
system performance significantly.
scenario.
• Most IAR systems are based on detecting physical char-
The most relevant SDKs for developing IAR applications
acteristics of an object or a place. Such a detection is
are shown in Table 2. They are compared according to sev-
complex and compute intensive and, therefore, powerful
eral requirements. First, their license type: open-source (i.e.,
hardware is required. Moreover, the detection of physi-
ARToolkit, ArUco, BazAr, UART, OpenSpace3D), free (i.e.,
cal patterns is also influenced by ambient light, being
ALVAR, Armedia, Vuforia, Wikitude), or commercial ver-
often even more sensitive to changes in the environment
sions (i.e., Armedia, Vuforia, Wikitude, ZappCode Creator).
than other marker-based detection techniques.
Other criteria are the platforms supported (i.e., Android, iOS,
• Electrical interference produced by the industrial
Linux or Windows) and requirements about the marker gen-
machinery inside the shipyard may affect sensor read-
eration, tracking or the overlaying capabilities.
ings and their accuracy. There are IAR systems that
Beyond the general requirements explained, the SDK
complement visual pattern recognition with embedded
should be compatible with the chosen HMD device (i.e.,
sensor measurements. A problem may occur in a ship-
Vuforia is compatible with Epson Moverio, ODG R-7 and
yard when a system relies on a GPS receiver or in Wi-Fi
HoloLens).
signals to locate a user or a room, since in certain envi-
Moreover, the compatibility with Unity should be con-
ronments (for instance, indoors or inside a ship under
sidered. Unity is currently one of the most advanced game
construction) such positioning techniques may not work
engines in the market and it is possible to use it for developing
properly.
and deploying IAR applications. Examples of SDKs with this
• Some IAR systems rely on incremental tracking, com-
feature are UART, Vuforia, ARKit or ARCore. Additionally,
bining information collected from different sensors and
the SDK chosen should consider geo-location support for
from the camera. Nonetheless, a dynamic environment
creating location-based IAR applications, and Simultaneous
like a shipyard workshop or a ship that is being built,
Localization and Mapping (SLAM) to map an environment
where the geometrical structure of the place changes
and track movements in order to enable indoor navigation.
through time, can mislead the IAR system.
Examples of this feature can be seen in ARKit or Instant Real-
Finally, it should be mentioned that the vast majority of ity. Cloud services and text recognition features should be
current IAR hardware devices can be still considered exper- also taken into account in scenarios where they are required.
imental developments, what makes it difficult their integra- At the time of writing this article (December 2017), all
tion with existing IAR frameworks and the implementation of the referred SDKs are available. Nevertheless, the IAR
of new features. An open-source framework would make landscape is constantly evolving and many SDKs available
it possible to develop modifications to work with different in the last years have gone away, changed (e.g., different
platforms. license type) due to the shifting commercial priorities or have
been overridden by newer projects (e.g., ARToolkit has been
B. SOFTWARE DEVELOPMENT TOOLS FOR IAR recently acquired by Daqri).
Nowadays, there are many libraries for the development of
IAR applications, each implementing certain functionalities VII. IAR ARCHITECTURE FOR A SHIPYARD 4.0
required in a specific scenario. In general, for the develop- A. TRADITIONAL COMMUNICATIONS ARCHITECTURE
ment of an IAR application it is necessary: The traditional IAR architecture is composed by the three
• 2D and 3D graphic libraries, which should enable real- different layers depicted on the left in Figure 6, which are
time visualization and overlapping of virtual elements in responsible for data acquisition, data transport, and visual-
the field of view. ization and interaction with the user.
• Recognition mechanisms to be able to follow objects or The Visualization and Interaction System (VIS) layer
to superimpose information on them. is composed by HMD, HHD and spatial display devices,
• Speech recognition, which is very useful when the as well as by human-interaction interfaces through which
user is not able to interact with physical controls. the operator, located in the ship under construction,
TABLE 2. Most relevant SDKs for developing IAR applications (first part).
TABLE 2. (Continued.) Most relevant SDKs for developing IAR applications (second part).
the workshop or the shipyard, is able to interact with the Power Line Communication (PLC) technology can be used
system. to transmit information, although it should be noted that the
The Data Transport System (DTS) is responsible for col- network speed could be influenced by the electrical inter-
lecting the information obtained by the Data Acquisition Sys- ference coming from electric circular saws and other tools
tem (DAS) and transmitting it from the cloud infrastructure to that demand high-current peaks. Thus, a PLC system would
the location where operators are using the IAR system. This consist of two modules connected electrically to the same
process involves certain difficulties due to the environmental power phase. The first module would have direct connection
conditions: the shipyard’s structural barriers and the large to the DAS via Ethernet. The second PLC module would be
number of metal parts that impact wireless communications in the area of limited connectivity and would be coupled with
performance. Moreover, it has to be taken into account the a high-speed Wi-Fi access point to which the VIS can be
electrical interference produced by the industrial machinery connected to.
used for the different production processes. Regarding the DAS layer, which is hosted in the shipyard’s
It is worth mentioning that the communications between cloud infrastructure, it can obtain from the Manufacturing
each IAR device and the cloud layer can be performed Execution System (MES) data about work orders and the
through Wi-Fi connections considering that many shipyards elements to be installed.
have already deployed an IEEE 802.11 b/g/n/ac infrastruc- Note that the described IAR architecture, although it
ture. However, note that communications inside a ship sup- has been previously implemented by making use of a cen-
pose a challenge for electro-magnetic propagation due to tral server, a cloud or a PC-cluster [43], [130], [131],
the presence of numerous large metal elements, so other it has certain limitations when applied to IAR systems.
technologies would need to be further studied in such Since IAR devices project information dynamically on a
environments. Considering that electrical wiring is usually display, they require it to be loaded and rendered as fast as
deployed for temporary lighting during the ship construction, possible to provide a good user experience. For this reason,
being built, since electro-magnetic propagation is a challenge tasks. Note that shipbuilding requires handling sophisticated
in such environments with so many metal elements. One 3D CAD models that may be too complex for resource-
alternative that is currently being evaluated by Navantia is constrained IAR devices, so, instead of rendering the objects
PLC, but its application still has to be studied in detail in locally in every IAR device, the tasks are delegated to a
situations where electric interference from saws and other cloudlet that then delivers the resulting image to the IAR
tools occur. device faster than the cloud. In addition, cloudlets can also
The Edge Computing Layer is in the middle of the archi- perform other processing tasks that are too heavy to be per-
tecture, and it is actually divided into two sub-layers that formed in an SBC and that require a fast response.
can interact with each other, but whose objective is different. The cloud is at the top of the architecture. It receives,
The Fog Computing sub-layer is composed by one or several processes and stores data from the Edge Computing Layer.
Single-Board Computers (SBCs) that are installed in fixed The cloud also provides third-party services to IAR devices.
positions throughout the shipyard workshops and in a ship. For instance, in the case of a shipbuilder like Navantia, such
Each SBC acts as a gateway and provides fog services. In the services include the access to the content of the ERP (SAP
case of the IAR service, it supplies IAR devices with localized ERP), the CAD models (through FORAN), the information of
data and responds faster than the cloud, thus acting as a the PLM (Windchill) and to the IIoT platform (ThingWorx).
proxy caching server for IAR data. The second sub-layer is Finally, note that a modern IAR architectures have
composed by Cloudlets, which are local high-end computers also to take security into account. This aspect is often
specialized in rendering and performing compute intensive neglected, but the future dependency on IAR systems make
security essential. This has been addressed recently by some [12] ‘‘Digi-capital augmented/virtual reality report Q4,’’ Digi-Capital LLC,
researchers [17], but further study is required, especially in London, U.K., Tech. Rep. Q4, 2017.
[13] ABI Research. Accessed: Dec. 29, 2017. [Online]. Available:
industrial environments. https://www.abiresearch.com/
[14] ‘‘ABI research’s machine vision in augmented and virtual
VIII. CONCLUSIONS reality markets report,’’ ABI Res., New York, NY, USA, Tech.
This article reviewed the different aspects that influence the Rep. AN-2406, accessed: Dec. 29, 2017. [Online]. Available:
https://www.abiresearch.com/press/ar-vr-and-mobile-device-shipments-
design of an IAR system for the Industry 4.0 shipyard, consid- embedded-vision-/
ering diverse scenarios like workshops and a ship. It was first [15] IDC, Framingham, MA, USA. Accessed: Dec. 29, 2017. [Online]. Avail-
presented a brief overview of the underlying technologies able: https://www.idc.com/home.jsp/
and the main industrial IAR applications. Next, the article [16] N. Anand, N. Iwamoto, M. Kalal, R. Membrila, A. Siviero,
analyzed the most relevant academic and commercial IAR and M. Torchia, ‘‘IDC’s worldwide semiannual augmented and
virtual reality spending guide,’’ IDC, Framingham, MA, USA,
developments for the shipbuilding industry. Then, different Tech. Rep. AN-2406, accessed: Dec. 29, 2017. [Online]. Available:
use cases of IAR for a shipyard were described and a thorough https://www.idc.com/tracker/showproductinfo.jsp?prod_id=1381
review of the main IAR hardware and software solutions was [17] M. Langfinger, M. Schneider, D. Stricker, and H. D. Schotten, ‘‘Address-
ing security challenges in industrial augmented reality systems,’’ in
presented. After such a review, it can be concluded that there Proc. IEEE 15th Int. Conf. Ind. Informat., Emdem, Germany, Jul. 2017,
are many options for developing software IAR developments, pp. 299–304.
but IAR hardware, although it has progressed a great deal in [18] D. Schmalstieg and T. Höllerer, Augmented Reality: Principles and Prac-
the last years, it is still not ready for a massive deployment tice. Reading, MA, USA: Addison-Wesley, 2016.
like the one required by the Industry 4.0 shipyard. [19] A. Nee, S. Ong, G. Chryssolouris, and D. Mourtzis, ‘‘Augmented
reality applications in design and manufacturing,’’ CIRP
Regarding the communications architecture for an IAR Ann., vol. 61, no. 2, pp. 657–679, 2012. [Online]. Available:
system, it was analyzed the traditional version and it was http://www.sciencedirect.com/science/article/pii/S0007850612002090
proposed an enhanced three-layer Edge Computing architec- [20] H. Benko, E. Ofek, F. Zheng, and A. D. Wilson, ‘‘FoveAR: Combining
an optically see-through near-eye display with projector-based spatial
ture based on Cloudlets and on the Fog Computing paradigm, augmented reality,’’ in Proc. 28th Annu. ACM Symp. User Interface Softw.
which allows for supporting physically distributed, low- Technol. (UIST), 2015, pp. 129–135.
latency and QoS-aware applications that decrease the net- [21] M. Uenohara and T. Kanade, ‘‘Vision-based object registration for real-
work traffic and the computational load of traditional cloud time image overlay,’’ Comput. Biol. Med., vol. 25, no. 2, pp. 249–260,
1995.
computing systems. Moreover, considering that fog gateways
[22] U. Neumann and S. You, ‘‘Natural feature tracking for augmented
are usually constraint in terms of computing power, if an IAR reality,’’ IEEE Trans. Multimedia, vol. 1, no. 1, pp. 53–64,
system demands real-time rendering or compute-intensive Mar. 1999.
services, Cloudlets are necessary. [23] T. Okuma, T. Kurata, and K. Sakaue, ‘‘A natural feature-based 3D
object tracking method for wearable augmented reality,’’ in Proc. 8th
IEEE Int. Workshop Adv. Motion Control, Kawasaki, Japan, Mar. 2004,
REFERENCES pp. 451–456.
[1] Goldman Sachs Global Investment Research Technical Report: [24] Z. Chen and X. Li, ‘‘Markless tracking based on natural feature for
Virtual and Augmented Reality—Understanding the Race for the augmented reality,’’ in Proc. IEEE Int. Conf. Edu. Inf. Technol. (ICEIT),
Next Computing Platform. Accessed: Dec. 29, 2017. [Online]. Available: Chongqing, China, Sep. 2010, pp. V2-126–V2-129.
http://www.goldmansachs.com/our-thinking/pages/technology-driving-
[25] J. Yan, Y. Meng, L. Lu, and L. Li, ‘‘Industrial big data in an industry
innovation-folder/virtual-and-augmented-reality/report.pdf
4.0 environment: Challenges, schemes, and applications for predictive
[2] P. Fite-Georgel, ‘‘Is there a reality in industrial augmented
maintenance,’’ IEEE Access, vol. 5, pp. 23484–23491, 2017.
reality?’’ in Proc. 10th IEEE Int. Symp. Mixed Augmented
Reality (ISMAR), Oct. 2011, pp. 201–210. [Online]. Available: [26] Ó. Blanco-Novoa, T. M. Fernández-Caramés, P. Fraga-Lamas, and
http://dx.doi.org/10.1109/ISMAR. 2011.6092387 L. Castedo, ‘‘An electricity price-aware open-source smart socket
[3] R. T. Azuma, ‘‘A survey of augmented reality,’’ Presence, vol. 6, no. 4, for the internet of energy,’’ Sensors, vol. 17, no. 3, p. 643,
pp. 355–385, 1997. Mar. 2017.
[4] I. E. Sutherland, ‘‘The ultimate display,’’ in Proc. IFIP, 1965, [27] T. M. Fernández-Caramés, ‘‘An intelligent power outlet system for the
pp. 506–508. smart home of the Internet of Things,’’ Int. J. Distrib. Sensor Netw.,
[5] I. E. Sutherland, ‘‘A head-mounted three dimensional display,’’ in Proc. vol. 11, no. 11, p. 214805, Nov. 2015.
AFIPS, San Francisco, CA, USA, Dec. 1968, pp. 757–764. [28] M. Suárez-Albela, P. Fraga-Lamas, T. M. Fernández-Caramés,
[6] T. P. Caudell and D. W. Mizell, ‘‘Augmented reality: An applica- A. Dapena, and M. González-López, ‘‘Home automation system
tion of heads-up display technology to manual manufacturing pro- based on intelligent transducer enablers,’’ Sensors, vol. 16, no. 10,
cesses,’’ in Proc. 25th Hawaii Int. Conf. Syst. Sci., vol. 2. Jan. 1992, p. 1595, Sep. 2016.
pp. 659–669. [29] J. Pérez-Expósito, T. M. Fernández-Caramés, P. Fraga-Lamas, and
[7] W. Wohlgemuth and G. Triebfürst, ‘‘ARVIKA: Augmented reality L. Castedo, ‘‘VineSens: An Eco-smart decision-support viticulture sys-
for development, production and service,’’ in Proc. DARE, 2000, tem,’’ Sensors, vol. 17, no. 3, p. 465, Feb. 2017.
pp. 151–152.
[30] T. M. Fernández-Caramés, P. Fraga-Lamas, M. Suárez-Albela, and
[8] W. Friedrich, ‘‘ARVIKA-augmented reality for development, production
L. Castedo, ‘‘Reverse engineering and security evaluation of commercial
and service,’’ in Proc. Int. Symp. Mixed Augmented Reality, 2002, pp. 3–4.
tags for RFID-based IoT applications,’’ Sensors, vol. 17, no. 1, p. 28,
[9] N. Navab, ‘‘Developing killer apps for industrial augmented real-
Dec. 2016.
ity,’’ IEEE Comput. Graph. Appl., vol. 24, no. 3, pp. 16–20,
May 2004. [31] P. Fraga-Lamas and T. M. Fernández-Caramés, ‘‘Reverse engineering the
[10] ARTESAS. (Advanced Augmented Reality Technologies for Industrial communications protocol of an RFID public transportation card,’’ in Proc.
Service Applications) Project. Accessed: Dec. 29, 2017. [Online]. Avail- IEEE Int. Conf. (RFID), May 2017, pp. 30–35.
able: https://www.tib.eu/en/search/id/TIBKAT%3A52755300X/ [32] T. M. Fernández-Caramés, P. Fraga-Lamas, M. Suárez-Albela, and
[11] Google Glasses. Accessed: Dec. 29, 2017. [Online]. Available: L. Castedo, A Methodology for Evaluating Security in Commercial RFID
https://x.company/glass/ Systems, Paulo Crepaldi, Ed. Rijeka, Croatia: InTech, 2017.
[33] P. Fraga-Lamas, ‘‘Enabling technologies and cyber-physical systems for [55] S. Hauswiesner, M. Straka, and G. Reitmayr, ‘‘Virtual try-on through
mission-critical scenarios,’’ Ph.D. dissertation, Dept. Electrónica Sis- image-based rendering,’’ IEEE Trans. Vis. Comput. Graphics, vol. 19,
temas, Univ. Coruña, Galicia, Spain, May 2017. [Online]. Available: no. 9, pp. 1552–1565, Sep. 2013.
http://hdl.handle.net/2183/19143 [56] N. Wiwatwattana, S. Sukaphat, T. Putwanpen, S. Thongnuch, and
[34] M. Suárez-Albela, T. M. Fernández-Caramés, P. Fraga-Lamas, and P. Kanokudomsin, ‘‘Augmenting for purchasing with mobile: Usage and
L. Castedo, ‘‘A practical evaluation of a high-security energy-efficient design scenario for ice dessert,’’ in Proc. 5th Int. Conf. Inf., Intell., Syst.
gateway for IoT fog computing applications,’’ Sensors, vol. 17, no. 9, Appl. (IISA), Jul. 2014, pp. 446–450.
p. 1978, Aug. 2017. [57] N. F. M. El-Firjani and A. M. Maatuk, ‘‘Mobile augmented reality for
[35] P. Fraga-Lamas, T. M. Fernández-Caramés, and L. Castedo, ‘‘Towards the interactive catalogue,’’ in Proc. Int. Conf. Eng. MIS (ICEMIS), Sep. 2016,
Internet of smart trains: A review on industrial IoT-connected railways,’’ pp. 1–4.
Sensors, vol. 17, no. 6, p. 1457, Jun. 2017. [58] BMW’s AR Marketing Campaign for its Electric Vehicles (Accenture’s
[36] P. Fraga-Lamas, T. M. Fernández-Caramés, M. Suárez-Albela, Use Case). Accessed: Dec. 29, 2017. [Online]. Available:
L. Castedo, and M. González-López, ‘‘A review on Internet of Things for https://www.accenture.com/us-en/success-bmw-digital-transformation-
defense and public safety,’’ Sensors, vol. 16, no. 10, p. 1644, Oct. 2016. augmented-reality
[37] P. Fraga-Lamas, L. Castedo-Ribas, A. Morales-Méndez, and [59] F. Lamberti, F. Manuri, A. Sanna, G. Paravati, P. Pezzolla, and
J. M. Camas-Albar, ‘‘Evolving military broadband wireless P. Montuschi, ‘‘Challenges, opportunities, and future trends of emerg-
communication systems: WiMAX, LTE and WLAN,’’ in Proc. Int. ing techniques for augmented reality-based maintenance,’’ IEEE Trans.
Conf. Military Commun. Inf. Syst. (ICMCIS), May 2016, pp. 1–8. Emerg. Topics Comput., vol. 2, no. 4, pp. 411–421, Dec. 2014.
[38] R. Drath and A. Horch, ‘‘Industrie 4.0: Hit or Hype?’’ IEEE Ind. Electron. [60] Z. Zhu et al., ‘‘AR-mentor: Augmented reality based mentoring system,’’
Mag., vol. 8, no. 2, pp. 56–58, Jun. 2014. in Proc. IEEE Int. Symp. Mixed Augmented Reality (ISMAR), Sep. 2014,
[39] L. Da Xu, W. He, and S. Li, ‘‘Internet of Things in industries: A survey,’’ pp. 17–22.
IEEE Trans. Ind. Informat., vol. 10, no. 4, pp. 2233–2243, Nov. 2014. [61] J. Stoyanova, R. Gonçalves, A. Coelhc, and P. Brito, ‘‘Real-time aug-
[40] Made in China 2025: Critical Questions. Center for Strategic and mented reality shopping platform for studying consumer cognitive expe-
International Studies. Accessed: Dec. 29, 2017. [Online]. Available: riences,’’ in Proc. 2nd Experim. Int. Conf., Sep. 2013, pp. 194–195.
https://www.csis.org/analysis/made-china-2025 [62] J. Stoyanova, P. Q. Brito, P. Georgieva, and M. Milanova, ‘‘Comparison of
[41] E. Munera, J. L. Poza-Lujan, J. L. Posadas-Yagüe, J. Simo, J. F. Blanes, consumer purchase intention between interactive and augmented reality
and P. Albertos, ‘‘Control kernel in smart factory environments: Smart shopping platforms through statistical analyses,’’ in Proc. Int. Symp.
resources integration,’’ in Proc. IEEE Int. Conf. Cyber Technol. Autom., Innov. Intell. Syst. Appl. (INISTA), Sep. 2015, pp. 1–8.
Control, Intell. Syst. (CYBER), Jun. 2015, pp. 2002–2005. [63] S. K. Ong, M. L. Yuan, and A. Y. C. Nee, ‘‘Augmented reality appli-
[42] M. Alesky, E. Vartiainen, V. Domova, and M. Naedele, ‘‘Augmented cations in manufacturing: A survey,’’ Int. J. Prod. Res., vol. 46, no. 10,
reality for improved service delivery,’’ in Proc. IEEE 28th Int. Conf. Adv. pp. 2707–2742, 2008.
Inf. Netw. Appl., May 2014, pp. 382–389. [64] H. Naik, F. Tombari, C. Resch, P. Keitler, and N. Navab, ‘‘A step closer
[43] K. Smparounis, D. Mavrikios, M. Pappas, V. Xanthakis, G. P. Viganò, to reality: Closed loop dynamic registration correction in sar,’’ in Proc.
and K. Pentenrieder, ‘‘A virtual and augmented reality approach to col- IEEE Int. Symp. Mixed Augmented Reality, Sep. 2015, pp. 112–115.
laborative product design and demonstration,’’ in Proc. IEEE Int. Technol. [65] H. Wuest, T. Engekle, F. Wientapper, F. Schmitt, and J. Keil, ‘‘From cad to
Manage. Conf. (ICE), Lisbon, Portugal, Jun. 2008, pp. 1–8. 3D tracking: Enhancing scaling model-based tracking for industrial appli-
[44] K. P. K. Reddy, B. Venkitesh, A. Varghese, N. Narendra, G. Chandra, ances,’’ in Proc. IEEE Int. Symp. Mixed Augmented Reality, Sep. 2016,
and P. Balamuralidhar, ‘‘Deformable 3D cad models in mobile aug- pp. 346–347.
mented reality for tele-assistance,’’ in Proc. Asia–Pacific Conf. Multime- [66] M. Vassell, O. Apperson, P. Calyam, J. Gillis, and S. Ahmad, ‘‘Intelli-
dia Broadcast., Apr. 2015, pp. 1–5. gent dashboard for augmented reality based incident command response
[45] M. Schneider, J. Rambach, and D. Stricker, ‘‘Augmented reality based on co-ordination,’’ in Proc. 13th IEEE Annu. Consum. Commun. Netw.
edge computing using the example of remote live support,’’ in Proc. 18th Conf. (CCNC), Jan. 2016, pp. 976–979.
Annu. Int. Conf. Ind. Technol., Mar. 2017, pp. 1277–1282. [67] F. Loch, F. Quint, and I. Brishtel, ‘‘Comparing video and augmented
[46] J. Moloney, ‘‘Augmented reality visualisation of the built environment reality assistance in manual assembly,’’ in Proc. 12th Int. Conf. Intell.
to support design decision making,’’ in Proc. 20th Int. Conf. Inf. Vis., Environ. (IE), Sep. 2016, pp. 147–150.
Jul. 2006, pp. 687–692. [68] C. Shin, B. H. Park, G. M. Jung, and S. H. Hong, ‘‘Mobile augmented
[47] Y. Qi, ‘‘3D modeling and augmented reality,’’ in Proc. 4th Int. Universal reality mashup for furture IoT environment,’’ in Proc. IEEE UIC-ATC-
Commun. Symp., Oct. 2010, pp. 185–192. ScalCom, Dec. 2014, pp. 888–891.
[48] S. Henderson and S. Feiner, ‘‘Exploring the benefits of augmented reality [69] M. A. Frigo, E. C. C. da Silva, and G. F. Barbosa, ‘‘Augmented reality
documentation for maintenance and repair,’’ IEEE Trans. Vis. Comput. in aerospace manufacturing: A review,’’ J. Ind. Intell. Inf., vol. 4, no. 2,
Graphics, vol. 17, no. 10, pp. 1355–1368, Oct. 2011. pp. 125–130, 2016.
[49] S. Zollmann, C. Hoppe, S. Kluckner, C. Poglitsch, H. Bischof, and [70] V. Paelke, ‘‘Augmented reality in the smart factory: Supporting workers
G. Reitmayr, ‘‘Augmented reality for construction site monitoring and in an industry 4.0. environment,’’ in Proc. IEEE Emerg. Technol. Factory
documentation,’’ Proc. IEEE, vol. 102, no. 2, pp. 137–154, Feb. 2014. Autom. (ETFA), Sep. 2014, pp. 1–4.
[50] P. Hořejší, ‘‘Augmented reality system for virtual training of parts assem- [71] M. C. Leu et al., ‘‘Cad model based virtual assembly simulation, planning
bly,’’ in Proc. 25th DAAAM Int. Symp. Intell. Manuf. Autom. (DAAAM), and training,’’ CIRP Ann. Manuf. Technol., vol. 62, no. 2, pp. 799–822,
vol. 100. 2015, pp. 699–706. 2013.
[51] ‘‘Industrial development report 2013. Sustaining employment [72] G. Evans, J. Miller, M. I. Pena, A. MacAllister, and E. Winer, ‘‘Eval-
growth: The role of manufacturing and structural change,’’ uating the Microsoft HoloLens through an augmented reality assem-
United Nations Ind. Develop. Org., Vienna, Austria, Tech. bly application,’’ Proc. SPIE, vol. 10197, p. 101970V, May 2017, doi:
Rep. UNIDO ID/446, accessed: Dec. 29, 2017. [Online]. 10.1117/12.2262626.
Available: https://www.unido.org/sites/default/files/2013-12/UNIDO_ [73] X. Wang, S. K. Ong, and A. Y. C. Nee, ‘‘A comprehensive survey
IDR_2013_main_report_0.pdf of augmented reality assembly research,’’ Adv. Manuf., vol. 4, no. 1,
[52] P. Boulanger, ‘‘Application of augmented reality to industrial tele- pp. 1–22, Mar. 2016.
training,’’ in Proc. 1st Can. Conf. Comput. Robot Vis., May 2004, [74] H. Subakti and J. R. Jiang, ‘‘A marker-based cyber-physical augmented-
pp. 320–328. reality indoor guidance system for smart campuses,’’ in Proc. IEEE 14th
[53] B. Besbes, S. N. Collette, M. Tamaazousti, S. Bourgeois, and Int. Conf. Smart City, Dec. 2016, pp. 1373–1379.
V. Gay-Bellile, ‘‘An interactive augmented reality system: A prototype for [75] R. de Koster, T. Le-Duc, and K. J. Roodbergen, ‘‘Design and control of
industrial maintenance training applications,’’ in Proc. IEEE Int. Symp. warehouse order picking: A literature review,’’ Eur. J. Oper. Res., vol. 182,
Mixed Augmented Reality (ISMAR), Nov. 2012, pp. 269–270. no. 2, pp. 481–501, 2006.
[54] X. Zhang, N. Navab, and S. P. Liou, ‘‘E-commerce direct marketing using [76] D. Aiteanu, B. Hillers, and A. Gräser, ‘‘A step forward in man-
augmented reality,’’ in Proc. Latest Adv. Fast Changing World Multime- ual welding: Demonstration of augmented reality helmet,’’ in Proc.
dia IEEE Int. Conf. Multimedia Expo. (ICME), vol. 1. Jul./Aug. 2000, 2nd IEEE ACM Int. Symp. Mixed Augmented Reality, Oct. 2003,
pp. 88–91. pp. 309–310.
[77] R. S. Andersen, S. Bøgh, T. B. Moeslund, and O. Madsen, ‘‘Task [103] Sony Smarteye Glasses Official Web Page. Accessed:
space HRI for cooperative mobile robots in fit-out operations inside ship Dec. 29, 2017. [Online]. Available: https://developer.sony.com/
superstructures,’’ in Proc. 25th IEEE Int. Symp. Robot Hum. Interact. devices/mobile-accessories/smarteyeglass
Commun. (RO-MAN), Aug. 2016, pp. 880–887. [104] Vuzix Glasses Official Web Page. Accessed: Dec. 29, 2017. [Online].
[78] K. Fast, T. Gifford, and R. Yancey, ‘‘Virtual training for welding,’’ in Available: https://www.vuzix.com/Business-and-Enterprise-Solutions
Proc. 3rd IEEE ACM Int. Symp. Mixed Augmented Reality, Nov. 2004, [105] ‘‘Smart augmented reality glasses,’’ Tractica LLC, Boulder, CO,
pp. 298–299. USA, Tech. Rep., accessed: Dec. 29, 2017. [Online]. Available:
[79] G. A. Lee et al., ‘‘Virtual reality content-based training for spray painting https://www.tractica.com/research/smart-augmented-reality-glasses/
tasks in the shipbuilding industry,’’ ETRI J., vol. 32, no. 5, pp. 695–703, [106] A. Syberfeldt, O. Danielsson, and P. Gustavsson, ‘‘Augmented reality
2010. smart glasses in the smart factory: Product evaluation guidelines and
[80] H. Flatt, N. Koch, C. Röcker, A. Günter, and J. Jasperneite, ‘‘A context- review of available products,’’ IEEE Access, vol. 5, pp. 9118–9130,
aware assistance system for maintenance applications in smart factories 2017.
based on augmented reality and indoor localization,’’ in Proc. IEEE 20th [107] ALVAR Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
Conf. Emerg. Technol. Factory Autom. (ETFA), Sep. 2015, pp. 1–4. http://alvar.erve.vtt.fi
[81] V. Havard, D. Baudry, A. Louis, and B. Mazari, ‘‘Augmented reality [108] ARKit Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
maintenance demonstrator and associated modelling,’’ in Proc. IEEE https://developer.apple.com/arkit/
Virtual Reality (VR), Mar. 2015, pp. 329–330. [109] ARCore Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
https://developers.google.com/ar/
[82] M. Olbrich, H. Wuest, P. Riess, and U. Bockholt, ‘‘Augmented reality
[110] ARLab Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
pipe layout planning in the shipbuilding industry,’’ in Proc. 10th IEEE
http://www.arlab.com
Int. Symp. Mixed Augmented Reality, Oct. 2011, pp. 269–270.
[111] ARmedia Official Web Page. Accessed: Dec. 29, 2017. [Online].
[83] Y.-J. Oh, K.-Y. Park, and E.-K. Kim, ‘‘Mobile augmented reality system
Available: http://www.armedia.it
for Design Drawing visualization,’’ in Proc. 16th Int. Conf. Adv. Commun.
[112] ARToolKit Official Web Page. Accessed: Dec. 29, 2017. [Online].
Technol. (ICACT), Feb. 2014, pp. 1296–1300.
Available: https://artoolkit.org
[84] Newport News Shipbuilding. Accessed: Dec. 29, 2017. [Online]. Avail- [113] ArUco Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
able: http://nns.huntingtoningalls.com/ar/ https://www.uco.es/investiga/grupos/ava/node/26
[85] Index AR Solutions. Accessed: Dec. 29, 2017. [Online]. Available: [114] Augumenta Studio Official Web Page. Accessed: Dec. 29, 2017. [Online].
https://www.indexarsolutions.com/about-us/ Available: http://augumenta.com
[86] Virtual Reality Technology Transforms Design of UK [115] Aurasma Official Web Page. Accessed: Dec. 29, 2017. [Online]. Avail-
Warships. Accessed: Dec. 29, 2017. [Online]. Available: able: https://www.aurasma.com
http://www.baesystems.com/en/article/virtual-reality-technology- [116] BaZar Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
transforms-design-of-uk-warships https://cvlab.epfl.ch/software/bazar
[87] P. Fraga-Lamas, D. Noceda-Davila, T. M. Fernández-Caramés, [117] BeyondAR Official Web Page. Accessed: Dec. 29, 2017. [Online].
M. Díaz-Bouza, and M. Vilar-Montesinos, ‘‘Smart pipe system for Available: http://www.beyondar.com
a shipyard 4.0,’’ Sensors, vol. 16, no. 12, p. 2186, Dec. 2016. [118] Beyond Reality Face Official Web Page. Accessed: Dec. 29, 2017.
[88] P. Fraga-Lamas, T. M. Fernández-Caramés, D. Noceda-Davila, and [Online]. Available: https://www.beyond-reality-face.com
M. Vilar-Montesinos, ‘‘RSS stabilization techniques for a real-time pas- [119] Catchoom Official Web Page. Accessed: Dec. 29, 2017. [Online]. Avail-
sive UHF RFID pipe monitoring system for smart shipyards,’’ in Proc. able: https://catchoom.com
IEEE Int. Conf. RFID (RFID), May 2017, pp. 161–166. [120] IN2AR Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
[89] P. Fraga-Lamas et al., ‘‘Enabling automatic event detection for the pipe http://www.augmentedrealityconcepts.com
workshop of the shipyard 4.0,’’ in Proc. 56th FITCE Congr., Sep. 2017, [121] Instant Reality Official Web Page. Accessed: Dec. 29, 2017. [Online].
pp. 20–27. Available: http://www.instantreality.org
[90] Atheer Air Official Web Page. Accessed: Dec. 29, 2017. [Online]. Avail- [122] Layar Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
able: http://atheerair.com https://www.layar.com
[91] CHIPSIP Sime Glasses Official Web Page. Accessed: Dec. 29, 2017. [123] Mixare Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
[Online]. Available: http://www.chipsip.com/computing/index.php? http://www.mixare.org
mode=data&id=126 [124] Open Space 3D Official Web Page. Accessed: Dec. 29, 2017. [Online].
[92] Daqri Smarthelmet Glasses Official Web Page. Accessed: Dec. 29, 2017. Available: http://www.openspace3d.com
[Online]. Available: https://daqri.com/products/smart-helmet [125] SSTT Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
[93] Epson Moverio BT-200. Accessed: Dec. 29, 2017. [Online]. Available: http://technotecture.com/projects/sstt
https://www.epson.es/products/see-through-mobile-viewer/moverio-bt- [126] UART Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
200 http://uart.gatech.edu
[127] Vuforia Official Web Page. Accessed: Dec. 29, 2017. [Online]. Available:
[94] Epson Moverio BT-300. Accessed: Dec. 29, 2017. [Online]. Available:
https://www.vuforia.com
https://www.epson.es/products/see-through-mobile-viewer/gafas-
[128] Wikitude Official Web Page. Accessed: Dec. 29, 2017. [Online].
moverio-bt-300
Available: https://www.wikitude.com
[95] Epson Moverio Pro BT-2000. Accessed: Dec. 29, 2017. [Online].
[129] ZappWorks Official Web Page. Accessed: Dec. 29, 2017. [Online].
Available: https://www.epson.es/products/see-through-mobile-viewer/
Available: https://zap.works
moverio-pro-bt-2000
[130] G. Schroeder, C. Steinmetz, and C. E. Pereira, ‘‘Visualising the digital
[96] Fujitsu Ubiquitousware HMD Official Web Page. Accessed: twin using web services and augmented reality,’’ in Proc. 14th IEEE Int.
Dec. 29, 2017. [Online]. Available: http://www.fujitsu.com/ Conf. Ind. Inf. (INDIN), Poitiers, France, Jul. 2016, pp. 522–527.
fts/products/computing/peripheral/wearables/hmd-iot001/
[131] C. Matysczok and A. Wojdala, ‘‘Rendering of highly polygonal aug-
[97] Laster WAV∃ Glasses Official Web Page. Accessed: Dec. 29, 2017. mented reality applications on a scalable PC-cluster architecture,’’ in
[Online]. Available: http://laster.fr/en/ Proc. 3rd IEEE ACM Int. Symp. Mixed Augmented Reality, Arlington,
[98] Microsoft HoloLens Official Web Page. Accessed: Dec. 29, 2017. TX, United States, Nov. 2004, pp. 254–255.
[Online]. Available: https://www.microsoft.com/en-us/hololens [132] J. Zhang, S. K. Ong, and A. Y. C. Nee, ‘‘A volumetric model-based CNC
[99] ODG R-7 Glasses Official Web Page. Accessed: Dec. 29, 2017. [Online]. simulation and monitoring system in augmented environments,’’ in Proc.
Available: https://www.osterhoutgroup.com/r-7-glasses-system.html Int. Conf. Cyberworlds, Nov. 2006, pp. 33–42.
[100] Optinvent ORA-2 Glasses Official Web Page. Accessed: Dec. 29, 2017. [133] Q. Rao, C. Grünler, M. Hammori, and S. Chakrabort, ‘‘Design methods
[Online]. Available: http://www.optinvent.com/our_products/ora-2/ for augmented reality in-vehicle infotainment systems,’’ in Proc. 51st
[101] Penny C Wear Glasses Official Web Page. Accessed: Dec. 29, 2017. ACM/EDAC/IEEE Design Autom. Conf. (DAC), Jun. 2014, pp. 1–6.
[Online]. Available: http://www.penny.se/products.html [134] K. Dolui and S. K. Datta, ‘‘Comparison of edge computing implemen-
[102] Recon Jet Glasses Official Web Page. Accessed: Dec. 29, 2017. [Online]. tations: Fog computing, cloudlet and mobile edge computing,’’ in Proc.
Available: https://www.reconinstruments.com/products/jet Global Internet Things Summit (GIoTS), Jun. 2017, pp. 1–6.
PAULA FRAGA-LAMAS (M’17) received the ÓSCAR BLANCO-NOVOA received the B.Sc.
M.Sc. degree in computer science from the Uni- degree in computer science with mention in com-
versity of A Coruña (UDC), in 2008, and the M.Sc. puter engineering and information technology
and Ph.D. degrees in the joint program Mobile from the University of A Coruña (UDC), in 2016,
Network Information and Communication Tech- where he is currently pursuing the master’s degree
nologies from five Spanish universities: the Uni- in computer science with the Group of Electronic
versity of the Basque Country, the University of Technology and Communications, Department of
Cantabria, the University of Zaragoza, the Uni- Computer Engineering. During the last years in
versity of Oviedo, and the UDC, in 2011 and college, he combined his studies with a job as
2017, respectively. Since 2009, she has been with a Software Engineer at a private company. His
the Group of Electronic Technology and Communications, Department of current research interests include energy control smart systems, augmented
Computer Engineering, UDC. She has co-authored over 30 peer-reviewed reality, and Industry 4.0.
indexed journals, international conferences, and book chapters. She has also
been participating in over 20 research projects funded by the regional and
national government as well as well as research and development contracts
with private companies. Her current research interests include wireless com-
munications in mission-critical scenarios, Industry 4.0, Internet of Things
(IoT), augmented reality, RFID, and cyber-physical systems.
Tabletop size Room size World size
Fig 2.2: Sizing the object
Responsive Playspace
If you’re creating a tabletop experience, make sure it fits on every tabletop. Design
the experience size to be responsive. Some users might play on a large banquet table.
Others might set up the experience on a small desk.
Types of movement
In AR experiences, there are four different ways that a user can move.
● Seated, with hands fixed
● Seated, with hands moving
● Standing still, with hands fixed
● Moving around in a real-world space
For each user case, try to:
● Let users know what movements will trigger the app.
● Guide them through the types and range of movement possible.
● Make easy transitions from one pose or movement to another.
● Design for comfort. Try to avoid making the user do anything that’s
physically demanding, uncomfortable, or too sudden.
● Try not to require movement until it’s necessary. Getting users to move is a
great way to engage them, but let them ease into the experience.
Accessibility
When you create 3D objects, create them to be life-size. Full-size objects are easier to
drop straight into your experience. All objects should face the same direction. Use a
right-handed coordinate space, where +Y is up, +X is right, and -Z points forward
from the origin. When you model an object for your scene, make sure to place it on
the ground plane at the geometric center of the object base. Remember, 3D objects
can be viewed from all sides. Use complete objects, and be sure to render all
surfaces, even those that a user might not immediately see, like the back surface of a
curtain, or the bottom of a couch.
Fig 2.3: Modeling of the 3D object in Virtual Environment
UX
Initialization
Use visuals to let users know they’re about to transition from a 2D screen into AR.
You can dim the phone display or use effects to blur the screen when a transition is
about to take place. In some apps, only one part of the experience will take place in
AR. Try to give the user a seamless transition to AR. Let the user launch the
transition from a 2D interface to AR. It’s less jarring when the user is in control. You
can include a button, such as an AR icon, to let users trigger the launch themselves.
Send the user gently into your AR environment.
Audio exploration
Use audio cues to enhance the user’s experience and encourage engagement.
Audio encourages users to engage with the app and explore the 360-degree
environment. Ensure that your audio adds to the experience rather than distracting
from it.
If you’re using audio for 3D objects or the 360 environments, be mindful of a few
things:
● Avoid playing sounds simultaneously
● Add attenuation to moderate sound effects
● Set the audio to fade or stop if the user is not interacting with the object
● Allow users to manually turn off the audio for individual objects
Fig 2.4: User Experience Enhancement
Multiplayer Experience
A multiplayer experience lets different users share the same AR environment. An
object that appears on one user’s device will appear to all users.
● Player 1 detects a surface
● Players 2, 3, and 4 detect the same surface by moving closer to Player 1
● The app recognizes all the players and connects them! Everyone now shares
the same AR environment.
Multiplayer experiences can require more hand-holding than single-user journeys.
Guide your users through each step. Try to make the moment of connection as
seamless as possible.
Fig 2.5: Multiplayer Mode in the virtual space
Interface
Create a world that’s immersive and easy to use Immerse users, don’t distract them.
Try to interrupt your AR world as little as you can. Get users into the experience, and
then get out of the way. Avoid pop-ups and full-screen takeovers unless the user
explicitly selects it. Buttons, 2D alerts, and notifications can distract the user from
the 3D world that you’re creating around them. Instead, let users focus on the scene
itself. Persistent 2D overlays can also disrupt the user’s immersion. It’s a constant
reminder that the world they’re looking at isn’t completely real.
Errors
Disadvantages of AR technology
camera in real-time.
● AR technology could lead to people becoming more dependent on devices
this may cause a large number of health-related issues.
● AR technology is not equipped with security policies. An intruder can hack
the AR-based devices and can manipulate the devices according to their
needs.
● It is expensive to develop AR-based systems and its maintenance is also very
expensive.
Applications
1. AUGMENT
Augment allows its users to see their products in 3D in a real-life environment and in
real-time through tablets or smartphones to drive sales and improve user engagement.
This app is available on both, iOS and Android platforms. This app can be used for
Retail, E-Commerce, Architecture, and other purposes also. Augment allows retailers
and manufacturers to connect with each other and thereby enable online shoppers to
experience the products sitting at home before buying. Customers can view the
images in 3D by rotating them and viewing all the augmented content before
deciding to buy. It has plenty of customers, companies such as Coca-Cola, Siemens,
Nokia, Nestle, and Boeing are using this application.
2. Sun-Seeker
Sun-Seeker is an AR app which provides a flat compass view and a 3D view showing
the solar path, its hour intervals, its equinox, winter and summer solstice paths,
sunrise and sunset times, twilight times, magic hours and also a Map view showing
solar direction for each daylight hour. The app runs on both the mediums i.e.,
Android and iOS. The app has got 3+ ratings from its users
3. ARGON4
This is a fully-featured web browser that has the ability to display augmented reality
content created with the argon.js Javascript framework. argon.js makes it easier for
adding augmented reality content to the web applications in a platform and
technology-independent way and supports the real-time AR capabilities of the
Argon4 Browser. The Argon4 browser is available on both the iTunes App Store and
Google Play Store.
4. AR Browser SDK
This is a browser created by Arabs. This browser allows the users to add an
augmented reality geolocation view to the Android and or iOS application in less
than 5 minutes. With user-friendly API (Application Programming Interface), it can
be fully customized. The framework takes care of all the complex functions of the
augmented reality browser.
Features :
● It provides video support.
● It adds and removes single POIs in real-time.
● It can run on any device.
● It offers great performance and memory management.
● It has an exceptionally light view, smooth and accurate movements.
5. Pokémon Go:
The most popular AR game to date is Pokémon Go which allows users to catch
virtual Pokémon that are hidden throughout the map of the real world. It uses real
locations to encourage players to far and wide in the real world to discover Pokemon.
The game enables the players to search and catch more than a hundred species of
Pokemon as they move in their surroundings. The app works on both the mediums
i.e., Android and iOS.
6. REAL STRIKE
This is a popular shooting AR game which is available only on iOS. The users get a
real-life shooting experience in this game and can record their fights and also create
their own videos. There is a pool that has been polluted by nuclear waste and a group
of pests is just around the corner so players have to stop them infecting the earth.
Users use their phones to scan the mark. The game offers night and thermal vision
goggles to get a clear view even in the evening to complete your mission.
7. AR GPS Drive/Walk Navigation
The application makes use of the smartphone’s GPS and camera to execute a car
navigation system with an augmented reality-powered technology. It is easier and
safer than the normal navigation system for the driver. This application is available
only on Android.
This app guides the drivers directly by the virtual path of the camera preview video
which makes it easy for them to understand. The drivers do not need to map the path
and the road while using this app. The driver can see the real-time camera preview
navigation screen to get driving conditions without hindering his safety.
8. AR GPS Compass Map 3D