Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
Effect of Pulsed Electromagnetic Field Stimulation on Splenomegaly and Immunoglobulin E Levels in 2,4-Dinitrochlorobenzene-Induced Atopic Dermatitis Mouse Model
Next Article in Special Issue
Uncertainty-Aware Federated Reinforcement Learning for Optimizing Accuracy and Energy in Heterogeneous Industrial IoT
Previous Article in Journal
VEPO-S2S: A VEssel Portrait Oriented Trajectory Prediction Model Based on S2S Framework
Previous Article in Special Issue
Context-Aware System for Information Flow Management in Factories of the Future
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 14
Issue 14
10.3390/app14146345
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Enhancing Robot Behavior with EEG, Reinforcement Learning and Beyond: A Review of Techniques in Collaborative Robotics

by
Asier Gonzalez-Santocildes
,
Juan-Ignacio Vazquez
* and
Andoni Eguiluz
Faculty of Engineering, University of Deusto, Avda. Universidades 24, 48007 Bilbao, Spain
*
Author to whom correspondence should be addressed.
Appl. Sci. 2024, 14(14), 6345; https://doi.org/10.3390/app14146345
Submission received: 17 June 2024 / Revised: 10 July 2024 / Accepted: 19 July 2024 / Published: 20 July 2024
(This article belongs to the Special Issue Autonomous Systems in Cyber-Physical Systems and Smart Industry: Innovations and Challenges)
Browse Figures
Figure 1
<p>The various levels of cooperation between a human worker and a robot.</p> "> Figure 2
<p>Classical RL loop [<a href="#B15-applsci-14-06345" class="html-bibr">15</a>].</p> "> Figure 3
<p>Percentage increase in publications across topic clusters over time (2012–2023).</p> "> Figure 4
<p>The five different brain waves: Delta, theta, alpha, beta, and gamma.</p> "> Figure 5
<p>Real experimentation using EEG for an assembly task [<a href="#B33-applsci-14-06345" class="html-bibr">33</a>].</p> "> Figure 6
<p>Different bio-sensors and their positions in the human body [<a href="#B51-applsci-14-06345" class="html-bibr">51</a>].</p> "> Figure 7
<p>Conceptual diagram. Intersection between topics displayed in triplets for future research.</p> ">
Versions Notes

Abstract

:
Collaborative robotics is a major topic in current robotics research, posing new challenges, especially in human–robot interaction. The main aspect in this area of research focuses on understanding the behavior of robots when engaging with humans, where reinforcement learning is a key discipline that allows us to explore sophisticated emerging reactions. This review aims to delve into the relevance of different sensors and techniques, with special attention to EEG (electroencephalography data on brain activity) and its influence on the behavior of robots interacting with humans. In addition, mechanisms available to mitigate potential risks during the experimentation process such as virtual reality are also be addressed. In the final part of the paper, future lines of research combining the areas of collaborative robotics, reinforcement learning, virtual reality, and human factors are explored, as this last aspect is vital to ensuring safe and effective human–robot interactions.

1. Introduction

Today’s industry is advancing at a very fast pace, which is why there is a growing demand to integrate collaborative robots, often called cobots, into work environments. These robots are specifically designed to work alongside humans, sharing their workspace and even collaborating on various tasks [1]. However, the introduction of collaborative robots brings with it some new paradigms and problems that traditional industrial robots do not generate, such as issues related to unintentional human–robot collisions, balancing the workload distribution, and acceptance of collaborative robots, among others [2,3]. That is why it becomes important to take into account aspects such as the work environment, human safety, and people’s emotional well-being [4,5].
In collaborative robotics, it is of great importance to consider the different levels of cooperation between a human worker and a robot. Figure 1 [6] shows a very detailed representation of the different collaborative scenarios. A common industrial scenario is the cell model, where humans and robots are working in clearly differentiated compartments. The first collaborative model is coexistence, where humans and cage-free robots work alongside each other but do not share a workspace, maintaining separate areas of operation. The second model is synchronized interaction, where human workers and robots share a workspace but only one party is present in the workspace at any given time, ensuring a coordinated but separate workflow. The third model, cooperation, takes this a step further by allowing humans and robots to perform tasks simultaneously in the same workspace; however, they do not work concurrently on the same product or component. Finally, the most integrated model is collaboration, where the human worker and robot work in unison, simultaneously on the same product or component, exemplifying the highest level of partnership and synchronization in shared tasks.
Emerging paradigms and problems discussed in the first paragraph that need to be taken into account have increased research efforts in the area of collaborative robotics, with the goal of understanding the impact of humans sharing most—if not all—tasks with robots. It is important to take into account aspects such as efficiency, trust, ergonomics (strain index [7]), and the emotional experiences of humans in environments shared with collaborative robots [8]. Addressing these challenges is key for developing a safe workspace [9]. Ensuring efficiency helps in optimizing task performance, trust is essential for smooth human–robot interaction, ergonomics prevents physical strain, and positive emotional experiences promote better collaboration and acceptance of robots in work environments [9,10].
In the field of collaborative robotics, a leading idea revolves around the application of Reinforcement Learning (RL) to solve some of these challenges [11]. In the context of machine learning, reinforcement learning stands out as an approach in which an agent iteratively improves its behavior by interacting with its environment, receiving feedback through rewards and penalties to fine tune its decision-making process. Through these techniques, cobots can exhibit emergent behavior, and they can learn to handle simple tasks, which can be as basic as lifting a box or assembling a series of components. While it is true that through reinforcement learning, cobots can perform more difficult actions and exhibit more complex behaviors [12], these kinds of training tend to consume a great deal of time and computational resources.
As mentioned in the previous paragraph when presenting reinforcement learning and its uses and strengths, the purpose of applying reinforcement learning to collaborative robotics is to optimize specific routines and potentially discover new behaviors that are otherwise unobtainable. Furthermore, it allows for direct handling of complex procedures, pre-trained behaviors allowing the robot to overcome new, unseen situations, making real-time decisions while interacting with the environment. The ability of a reinforcement learning agent to face unforeseen situations and respond to them is crucial in collaborative robotics [13], where adaptability to evolving scenarios, including unexpected factors such as human reactions, is critical. These key factors, such the ability to adapt to new situations and variables, highlight why reinforcement learning is one of the most appealing areas with respect to shaping the behavior of robots [14] in collaborative environments.
In classical reinforcement learning, the learning process is conceptualized as a loop where an agent interacts with its environment in discrete time steps. Every step, the agent observes the current state of the environment and selects an action based on its policy—a strategy defining its behavior. The action, when executed by the agent in the specific environment, leads to a new state and provides a reward signal to the agent. The agent’s goal is to learn a policy that maximizes the cumulative reward over time. This classical loop of observation, action, reward, and learning proposed by Richard S. Sutton [15] (Figure 2) continues until the agent achieves satisfactory performance or the environment changes significantly.
In the realm of collaborative robotics training, RL plays a central role by enabling robots to learn from interactions within their environment. Typically, this training is conducted in virtual simulation environments, which allow for the execution of hundreds of interactions in a significantly reduced time frame compared to real-world trials. These simulations are instrumental in developing and refining the policies used to make decisions [16]. By utilizing RL, robots can optimize their actions to achieve specific goals, such as improving efficiency, accuracy, or safety in collaborative tasks. Numerous simulation platforms, such as Gymnasium [17] and ROS-Gazebo, are available to the scientific community [18]. These platforms offer diverse tools and frameworks that facilitate the development, testing, and validation of RL algorithms. They enable researchers to simulate complex scenarios, fine tune their models, and ensure that robotic systems can adapt to dynamic environments and interact effectively with human collaborators [12]. This controlled and accelerated training process enhances the reliability and efficiency of collaborative robotic systems before their deployment in real-world scenarios.
Most approaches ignore the human aspect in training reinforcement learning agents when considering the two fields of collaborative robotics and reinforcement learning. As outlined in the opening of this article, factors like human well-being and workspace are significant variables. The approach of integrating reinforcement learning training with the human element via the use of EEG sensors to positively influence the collaborative robot’s behavior has a lot of potential. The interpretation of EEG signals could be incorporated into the observation space (probably with a previous preprocessing phase to extract the relevant features), allowing the collaborative robot to perceive human states more accurately. This integration can significantly influence the reward function, enabling the robot to consider how the human operator is feeling while collaborating in the task. This review seeks to explore the different approaches when merging collaborative robotics, reinforcement learning, and the human element. It is important to note that quantifying human interaction, especially in terms of factors like trust in the robot or task satisfaction, can be challenging [19]. Numerous studies have develop new techniques and methods available for quantifying different variables with measurement devices such as EEG sensors.
EEG sensors are devices used to measure electrical activity in the brain. They are important tools for monitoring neural signals and gaining insights into the cognitive process of human reasoning. The use of these sensors has been increased in the recent stages of human–robot monitoring for different measurements, such as error detection or human comfort [20]. In addition to EEG sensors, other methods are used in other research areas to assess human confidence and related factors [21], including computer vision, eye tracking, and post-experiment human evaluations. Throughout this review article, the sensors and techniques applied in collaborative robotics that have been or can be applied to reinforcement learning to improve human well-being are explored.
It is also worth highlighting the growing trend of using virtual reality in collaborative robotics. Human safety is always needed, and thanks to controlled or virtual environments [22], experiments can be carried out without putting any person at risk. In this paper, different possibilities for carrying out studies using collaborative robots and human cognitive signals are analyzed.
Connecting the exploration of EEG sensors, virtual reality, and collaborative robotics, this article aims to bridge the gap between cutting-edge technologies and key aspects essential for understanding the human aspect in reinforcement learning. The following definitions are used for these relevant terms:
  • Reinforcement Learning: A machine learning approach where an agent refines decision-making skills by interacting with the environment, guided by rewards or penalties for actions, progressively improving decisions [23].
  • Emergent Behavior: For reinforcement learning agents, emergent behaviors relate to unpredictable actions arising from the combination of simpler actions; in collaborative robotics, it is the robot’s ability to exhibit unplanned responses from interactions [12].
  • EEG Sensors: Devices measuring brain electrical activity, which is valuable for monitoring neural signals and cognitive processes, particularly in assessing the mental state of human operators in collaborative robotics [24].
  • Human–Robot Interaction (HRI): Dynamic interactions between humans and robots in collaborative settings, encompassing communication, cooperation, and the study of their collaboration in shared workspaces [25]. In construction, innovative categorizations have been developed, leading to better approaches to human–robot interaction problems [26].
  • Virtual Reality (VR): An immersive technology that transports users to computer-generated environments, offering a multisensory experience that can simulate real-world scenarios, impacting fields such as education, robotics, gaming, and therapy [27,28].

2. Materials and Methods

To conduct this research, a comprehensive literature survey was undertaken, utilizing the most relevant academic platforms. Key resources were meticulously sourced from esteemed databases such as Google Scholar [29], Scopus [30], and Web Of Science [31]. The adopted approach ensures a diverse selection of articles and books, focusing on the latest and most significant research in the field. The integrity of the analysis is supported by the selection of sources from reliable platforms.
Before undertaking all the research, it is relevant to analyze the annual growth in the number of publications, among other factors, since this usually reflects the growing relevance of the selected topics in society. The main purpose is to support the review and future research proposed at the end of the article. Figure 3 shows the increase in publications across topic clusters since 2013. The graph presents a detailed comparison of growth across the following four research clusters: reinforcement learning with collaborative robotics; reinforcement learning with EEG; reinforcement learning combined with both collaborative robotics and virtual reality; and a triad of reinforcement learning, EEG, and collaborative robotics. Each cluster’s trajectory highlights the increasing significance of these integrated research areas over time.
To ensure a systematic and rigorous review of the literature, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology was followed [32]. Initially, approximately 500 papers were identified through a comprehensive search using specific keywords such as “reinforcement learning”, “collaborative robotics”, “EEG”, and “virtual reality”, among others. The search was conducted in the titles, abstracts, and keywords of the papers in the selected databases.
Subsequently, a preliminary selection was made, reducing the number to about 100 papers based on the relevance of their abstracts concerning the intersection of the following four main topics: reinforcement learning, collaborative robotics, EEG, and virtual reality. Finally, a thorough and detailed reading of these 100 papers was performed, evaluating the direct and significant contribution of each one to the intersection of the mentioned topics. As a result, 40 valuable papers were selected, which are discussed throughout this paper and analyzed in the table presented in the Discussion section.
The intersection of robotics with other cutting-edge research areas has experienced a pronounced growth trend beginning in 2019. This trend is especially prominent when considering the combination of robotics with reinforcement learning and virtual reality, as well as the integration of reinforcement learning with EEG technologies. The growing scholarly concentration on these subjects is representative of their increasing impact on the evolution of intelligent systems and on the progression of our understanding in the realm of human–computer interactions.
This documented growth trajectory, consistent across all topic clusters since 2019, suggests an expanding scholarly and developmental interest, reflecting the potential that these interdisciplinary studies hold. The aim of this article is to highlight the continuous and remarkable development in these fields, emphasizing the relevance of this comprehensive review in identifying key studies. In the concluding part of this paper, key directions are outlined, providing actionable steps to foster the integration of these dynamic fields, with the aim of amplifying their collective contribution to future innovations.

3. State of the Art

Most relevant research studies in the field of reinforcement learning applied to collaborative robotics are highlighted herein, with a focus on human factors and brain activity signals. Solely relevant research at the intersection of the aforementioned subjects is analyzed in this section.

3.1. EEG-Based Brain–Computer Interface Approaches in Collaborative Robot Control

In this sub-section, it is intended to analyze different research and technological advances in the area of brain signal capture through EEG applied to collaborative robotics. The purpose is to address the possibility of improving the understanding and execution of robotic systems to make them more intuitive and adaptive to a human being.
One of the most recent studies in the field of collaborative robotics, combining brain activity measurement through EEG and machine learning, is the research published by the University of Pennsylvania in 2022. The paper discusses the use of EEG signals to measure peoples’ trust levels in collaborative construction robots. EEG signals provide valuable information about human brain activity and cognitive states, including trust, during human-robot collaboration [33]. EEG signals are also used for determining mental states, electroencephalography sensors are able to measure brainwave frequencies such as delta, theta, alpha, beta, and gamma waves. Gamma waves (25–45 Hz) provide important insights related to intentional activities, multitasking, and creativity. Beta waves (12–38 Hz) include low beta (12–15 Hz) for idle states, beta (15–23 Hz) for attention, and high beta (23–38 Hz) for stress and difficult activities. Alpha waves (8–12 Hz) are associated with relaxed concentration. Theta waves (4–8 Hz) appear during REM sleep, deep meditation, or flow states. Lastly, delta waves (0.5–4 Hz) signify dreamless sleep [33,34] (See Figure 4).
The incorporation of brain wave signals, specifically alpha and beta waves, into the observation space of a RL agent is meaningful due to their direct relationship with stress and related emotions. Alpha waves are typically associated with states of relaxation and meditation, while beta waves are linked to increased alertness and anxiety [34]. These signals can be processed and subsequently included in the observation space, enabling the RL agent to interpret them and act accordingly [33]. Additionally, integrating these signals into the agent’s reward function can provide valuable insights into behavioral variations, thereby enhancing the agent’s ability to refine its strategies and responses in scenarios involving stress and relaxation.
However, EEG signals can be contaminated by other frequency signals, both intrinsic and extrinsic, resulting in a reduction in the original trust signal quality. In order to address this, some studies used a fixed-gain filtering method to reduce extrinsic components and utilized independent component analysis (ICA) to remove intrinsic components from EEG signals. Once the filtering was done, the results in the EEG measurements were significantly cleaner [33,35].
After the reduction was done, 12 trust related characteristics from the EEG signals that span the temporal and frequency domains were extracted. These features were calculated from segmented EEG data, and this information was then used for machine learning model training.
In order to evaluate the amount of trust levels in collaborative robots several supervised learning algorithms [36] are used, including k-nearest neighbors (k-NN), support vector machine (SVM) [37], and random forest. Among the supervised learning algorithms used, the k-NN outperforms the others showing the highest accuracy at approximately 88%.
Once machine learning algorithms are applied, a test with human participants involving building tasks can be conducted to determine trust levels in different robot collaboration scenarios. The results show that higher levels of trust are achieved while working with semi-autonomous robots. However, working with an autonomous robots lead to lower levels of trust due to the sense of not having any control on the robot. These research findings remark the potential of EEG-based trust measurement in a human-robot collaboration [33,35].
The conclusion is that EEG brainwaves can be used to determine a person’s level of trust in a robot as it performs a task. It’s important to note that these experiments are conducted within a controlled virtual reality environment. Additionally, while this research does not utilize reinforcement learning strategies, highlighting the potential for future application of these signals in training reinforcement learning algorithms for collaborative robot environments is significant [38].
Reinforcement learning in the contexts of collaborative robots and brain signals was not specifically addressed. However, a very recent study also explores the use of reinforcement learning to enhance human-robot collaboration in assembly tasks, focusing on dynamic task allocation, effectively balancing the workload between humans and robots [39]. Building upon this foundation, research in the field continued to evolve modeling discomfort in human-robot collaboration and making the robot meet individual preferences [40]. However, a different investigation secured its status as an innovator in the discipline, constructing the initial steps for the assimilation of EEG signals into reinforcement learning and robotics [41].

3.2. Enhancing EEG in Collaborative Robot Control with Reinforcement Learning

Expanding further on the previous findings, the study [41] explores the application of reinforcement learning algorithms in robotics, specifically in the context of robots learning to solve tasks based on reward signals obtained during task execution. In many other research, these reward signals are either modeled by programmers or provided through human supervision. However, there are situations where encoding these rewards can be challenging, resulting in the suggestion of using EEG-based brain activity as reward signals.
The core idea of this proposed article is to extract rewards from brain activity while observing a robot performing a task, this eliminates the need for an explicit reward model. The paper introduces a new idea for using brain activity signals through EEG sensors to provide correctness feedback to a collaborative robot about an specific task. This demonstrates the ability to identify and classify different error levels based on the brain signals [41].
Brain-computer interfaces (BCI) [42] in robotics have been identified as a hot topic [19]. EEG is highlighted as the recording method of choice due to its portability and high temporal resolution. The research also remarks the use of event-related potentials (ERPs) [43] in error detection and shows how these ERPs may be automatically categorized using machine learning and signal processing methods.
The study also offers a reinforcement learning framework for learning tasks based on reward signals obtained from monitored brain activity. Q-learning [44] is a reinforcement learning method that uses a Q-function to optimize sequential decision-making and was the algorithm of choice to demonstrate learning in real-time tasks in collaborative robotics scenarios. The results of the research suggest that EEG-based reward signals hold great potential in robot learning and task adaptation. However, the study was carried out in 2010, which is why several lines of improvement for future research are presented in the article.
Following the line of research presented where robots learn to adapt their behavior based on error signals generated by brain waves measured by EEG sensors, there is an investigation carried out in 2021 [45] that improves the previous one by proposing an approach in which a robot arm is trained to play a game and then uses the learning to teach different children. The training process involves automatic detection of rewards and penalties based on EEG signals, probability-based action planning, and imitation of human actions for training children.
In this research a specific reinforcement learning scheme is presented. In the case of the research carried out to teach the robot, the planning is not done as in traditional RL. Normally in reinforcement learning, actions are planned based on a partial learning of the environment, which means that agents make decisions based on the partial knowledge they have acquired so far in the environment. In the case of the proposed research, action planning takes place after the RL algorithm has converged (convergence happens when the RL algorithm has reached a state of knowledge where it has learned enough about the environment and the actions).This approach can be very beneficial in situations where fast and accurate decisions are required, one of those could be what the article is describing: the use of RL for training a robot to play a specific game [45].
Regarding the learning approach based on error signals and continuing with the review proposed above, it is necessary to highlight that the error potential-related events (ErrP) [46] signals represent the subjective errors when a subject observes an error either in a robot or even in itself [47]. In the proposed learning case, if no error is detected, a small positive reward will be given, however, if an error is detected, a negative reward will be applied. The rewards will be used to update a table of probabilities that consider states and actions. Once the entire learning phase is completed, the agent will have acquired a behavior based on the probabilities in that table.
In the training phase, the objective is to update the State-Action Probability Matrix (SAPM) to optimize actions for given states. This requires error signal detection and management using classifiers [48]. Unlike traditional BCI systems, training occurs both online and offline. On one hand, offline training involves subjects performing sessions to gather data, with a portion used to train classifiers, using around 12,000 instances of brain signals. On the other hand, online training then adapts the SAPM using reinforcement learning. After training, the agent’s behavior, particularly the robot arm’s action planning, is tested. This process involves data acquisition, offline classifier training, and SAPM adaptation [45].
The study conducts a two-stage training with the Jaco robot arm. The first phase is offline, using EEG data for classifier training with 18,000 instances, including ERD/ERS and ErrP signals. The second phase is online, adapting the SAPM with visual and audio stimuli for learning and correction. The test phase compares the performance of children trained by the robot to those trained by humans [45].
In general, the study is quite innovative since it allows detecting when a user performs an experiment in the wrong way. Then the robot’s behavior will be modified in order to teach the user to perform the experiment correctly. One of the aspects to highlight in this case is that the behavior of the robot is not directly influenced by the EEG signals, rather it will choose an action to be taken when the agent determines, either to replicate one of the movements or to throw the ball again. One of the future lines to point out could be the training of the agent in the task of throwing the ball and then, depending on the degree of error from the user, modify its behavior in a more direct way [45].
Although there are points that could be improved in the previous article, there are other very notable studies that aim to modify the total behavior of a robot based on the measured EEG signals. In addition, the authors only explored the possibility of detecting errors in certain tasks to carry out training of different agents. However, the possibility of detecting different feelings and emotions is something that can be distinctive to modify the behavior of a robot [49].
Other interesting approach was published in 2021 under the name “Emotion-Driven Analysis and Control of Human-Robot Interactions in Collaborative Applications” [33]. The authors focus on the behavior of a robot and the ability of the robot to adapt to different situations depending on the brain signals that are received from an EEG sensor. The research is based on the application of fuzzy logic rules to modify certain critical variables of the robot motion such as speed or motion delay. The rules are created from the beginning by focusing on stress. Nevertheless, a trial-and-error process will be carried out to establish representative relationships between the robot’s speed and the user’s emotions.
The interesting thing about this study is that it is possible to modify the robot’s behavior in a relatively simple way. While it is true that only motion-related variables are modified through the applied rules, the robot can modify its behavior based on measured EEG signals. The experimentation process including the EEG sensor, the collaborative robot and the human can me appreciated in Figure 5. It is also noteworthy that no reinforcement learning is used during this research, the fact that it is possible to use brain signals and relate them directly to emotions such as stress, anxiety, and depression, is a very important aspect of this work that could lead to the use of these signals in reinforcement learning investigations.
Once it is established the possibility to modify the behavior of a robot based on the emotions that the user is feeling, new paradigms and new research areas appear. In a study carried out in June 2022 [49], it is confirmed that it is possible to modify the way a cobot behaves based on the feelings of a human being. The intention is to be able to modify the behavior of the robot to achieve a level of empathy, in this case, the robot acts in a very similar way to the emotions felt by the human. The experiment is carried out in a simple and controlled environment, but it is very useful to demonstrate that it is indeed possible to modify the behavior of a robot based on the brain measurements of a human.
After looking at the most recent studies, brainwave measurement is a reliable method for modifying a robot’s behavior. However, reinforcement learning is not an area where this approach is routinely applied, even though it allows for emergent behaviors and greater adaptability. Although it is true that most of the studies that apply RL focus only on the detection of errors in executions to reward or penalize an agent, it could be very useful to use similar techniques to modify the behavior of a collaborative robot depending on the user’s emotions.
During this review article, the filtering of brain signals has been covered on several occasions. However, it is important to note that other relevant studies propose different techniques to filter the signals and achieve the desired emotion or potential error detection. The use of Convolutional Neural Networks (CNNs) is proposed as a valid method to filter and classify the EEG signals. However, as the classification proposed was done in binary terms considering only if the experiment was done correctly or incorrectly; it’s difficult to relate if this could be useful for more complex experiments where different emotions need to be taken into account [50].
The filtering of EEG signals and their subsequent classification is one of the biggest challenges in this technology [41]. In addition to that, everything related to brain signals is a relatively recent topic that has emerged during the last few years. That is why several techniques have been used in the quantification of signals that can enable Human-Robot Interaction.
In this section it became clear that the brain signals measured by EEG sensors are perfectly valid to carry out different investigations around collaborative robotics and reinforcement learning. In the following section, other relevant techniques related to the capture of different valuable signals for Human-Robot Interaction will be discussed.

3.3. Additional Human State Measuring Techniques for Collaborative Robotics

In this subsection, several techniques related to measuring human states, apart from EEG, are described for their application in human–robot environments. A comprehensive overview is provided in Figure 6, which features a detailed diagram of the human body, indicating the placement of various bio-sensors. These sensors include EOG (Electro-oculography), RSP (Respiration Rate), GSR (Galvanic Skin Response), ECG (Electrocardiography), and BVP (Blood Volume Pulse). This figure serves as a valuable illustration, showcasing the diversity and potential applications of bio-sensors in collaborative robotics.
Further exploration of some bio-sensor devices, as highlighted in [51], is undertaken to provide an overview of their various applications within collaborative robotics and reinforcement learning. This examination aims to illuminate how these sensors can be integrated into robotic systems to enhance human–robot interaction, ensuring more intuitive and effective collaboration.
One of the techniques that has been useful in human–robot interaction is eye tracking. An eye tracker is a device that records and follows a person’s eye movements during human–robot interaction, allowing them to understand how a human focuses attention and responds to visual stimuli. Numerous studies have revealed that the eyes can play a significant role in communication and anticipation of a robot’s actions [52,53].
Eye trackers have been used in different investigations to determine the actions of collaborative robots, as well as in stress detection. The idea behind using an eye tracker is associated with the ease of reading the signals with the right device. The only drawback of this type of technology is the delay between reading and interpreting the signal, which can mean that the human is in a completely different state. Stress detection is one of the most interesting areas behind eye trackers, as it could allow for the training of a RL agent based on the investigations discussed in the previous paragraphs. For the detection of this type of stress-related variable, it is important to take into account pupil diameter and the number of gaze fixations [54].
Another technology that allows for the detection of different emotions and movements is computer vision. Computer vision refers to the application of different algorithms in combination with image and video processing techniques to identify gestures, postures, and facial expressions in real time. The aim of this kind of technique is to understand and analyze human behaviors and associated emotions [55].
One of the most interesting disciplines in computer vision is human activity recognition (HAR). These types of disciplines have wide applications in fields such as human–machine interaction, robotics, and video games, where HAR improves understanding of human intentions and emotions [55,56].
Thanks to human activity recognition, computer vision may have a place in robotics and reinforcement learning applications, as shown in an interesting recent study [57]. However, computer vision research normally targets the use of computer vision in anomaly detection, with insufficient examples of its integration with reinforcement learning. Anomaly detection implies using computer vision techniques to identify unusual behaviors and events in videos or images; this plays a significant role in security surveillance systems [56,58].
Although computer vision is a widely used research technique, it has many drawbacks. Reliability under variable lighting conditions or in blurred images is usually a problem, as these systems often fail under harsh conditions. In addition, interpreting complex images or detecting objects in unusual situations still represents a significant challenge for computer vision. Finally, there is a risk of bias and discrimination in computer vision systems when training datasets are not properly represented [55].
Either eye trackers or adapted cameras for computer vision can be useful to detect variables related to different human factors; however, they depend on many aspects, such as light, to function properly. Another useful technique that was first used years ago to measure stress and fatigue is cardiac rhythm. A cardiological study [59] revealed that arrhythmias and tachycardias are directly correlated with the level of stress and fatigue felt by patients. Correct measurement with the right sensors can determine the level of stress and, therefore, emotions that a person feels when interacting with a robot [59].
The variety of devices capable of measuring heart rate is quite wide—from wearable devices such as smartwatches to body bands or stickers placed on the chest [60]. Among all of them, it is necessary to take into account different limitations, such as battery, connectivity problems, signal quality, accuracy, security when handling data, and device pricing [60].
The studies mentioned above claim that heartbeat sensors are valuable tools for monitoring stress and fatigue [59]. That is because variability in heart rate can provide indications of a person’s emotional and physical state. However, when it comes to determining ErrPsor evaluating complex emotions and cognitive responses, such as those related to EEG, these sensors may not be the best choice due to their limited ability to capture detailed information about specific brain processes.
In the context of exploring methodologies relevant to medical research, several technologies have been identified as effective in evaluating an individual’s stress levels or physiological state in real time. Notably, the measurement of cortisol, a hormone intricately associated with stress response, has emerged as a significant area of interest [61]. The precision offered by cortisol as a metric is high, yet the challenges lie in the actual process of measurement, particularly in achieving real-time data acquisition. Recent advancements have been made in real-time cortisol measurement, signaling a promising yet still emerging area of scientific exploration.
As research in the field advances, there has been a notable increase in the variety of methods being developed to measure human interactions and stress-related emotions. Techniques such as breath rate monitoring [62] and observation of changes in facial complexion, including blushing [63], are indicators of stress levels. These technologies are essential in the realm of human–robot interaction, providing a deeper understanding of human emotional states. Their integration into robotic systems holds the potential to significantly enhance the way robots interpret and respond to human emotions, leading to more empathetic and effective interactions.
Questionnaires and scales are self-reporting tools in which people answer questions about their emotional states and level of stress after performing specific tasks. Some of the most relatable examples of self-report questionnaires are the Beck Depression Inventory (BDI), which is used to assess depression, and the Perceived Stress Questionnaire (PSS), which is used to measure perceived stress. The BDI is a broadly used tool for measuring the severity of depression in individuals. It was developed by psychologist Aaron T. Beck in 1961 [64,65]. The Perceived Stress Questionnaire is used to assess how stressed a person feels in relation to his or her life experiences, circumstances, and tasks [66]. Despite the existence of numerous questionnaires that are consistent and achieve the proposed result, it is always possible to create questionnaires to reveal different variables of interest in a particular experiment.
In conclusion, this subsection highlights various techniques for measuring human states in the context of human–robot interaction. While EEG remains prominent, alternative methods like eye tracking, computer vision, and cardiac rhythm analysis offer valuable insights. Each method has its strengths and limitations, making selection context-dependent.

3.4. Immersive Technologies as a Safe Training Ground for Reinforcement Learning in Human-Interactive Robotics

Given the direct interaction between robots and humans in the field of robotics, which often poses risks, assessing the practicality of virtual reality in experiments related to collaborative robotics is essential. This section seeks to differentiate among virtual reality, mixed reality, and augmented reality to analyze their potential application in future research projects.
Virtual reality refers to a computer-generated environment that immerses the user in a simulated reality, often using a headset or other sensory input devices [67]. In VR, users can interact with and experience a digitally created world that can be entirely different from the physical world [68,69]. Augmented reality (AR) overlays digital information or virtual elements onto the real-world environment [70]. AR enhances the user’s perception of the physical world by providing additional digital content or information, often through a mobile device’s camera or specialized glasses [71,72].
Mixed reality (MR) combines elements of both virtual reality and augmented reality. In MR, digital objects or information are integrated into the real world in a way that allows them to interact with physical objects and the user’s environment [73].
In the field of augmented reality and collaborative robotics, a novel solution was proposed, involving a human–robot collaboration method using augmented reality to enhance construction waste sorting, aiming to improve both efficiency and safety [74]. Mixed reality is also becoming more popular due to some recently released products and some interesting research, ensuring the possibility of applying mixed reality in robotic environments to enhance the interactions between humans and robots [75].
Virtual reality is widely used to simulate real environments, aiming to prevent harm to humans and train users for specific tasks. It is especially prevalent in the medical field, where virtual reality trains medical students to perform complex surgeries [76]. Additionally, this technology can be used to learn how to interact with robots in medical settings or other environments [77].
The intersection between virtual reality and human interaction is also a hot topic. A system that combines robotics with virtual reality to improve welding by interpreting the welder’s intentions for robotic execution was recently introduced. Utilizing virtual reality for intention recognition enables precise and smooth robotic welding. This highlights the potential of integrating robotics and virtual reality in skilled tasks [78].
On the other hand, in order to apply reinforcement learning to train a robotic system to take into account different emotions or states of a human beings, the ideal solution is virtual reality, as these environments meet the safety requirements to guarantee humans are not harmed during experimentation. A recent study [79] describes a method where the virtual reality environment adapts to participant behavior in real time through reinforcement learning, focusing on encouraging helping behavior towards a victim in a simulated scenario of a violent assault.
If robotic training has already been carried out, virtual reality is also a reliable validation technique to test the performance of the system in real-life environments, complying with the required safety conditions. Thanks to the virtual reality glasses that are on the market these days, environments can be generated in a very simple way. One of the examples in which this type of technology can be useful is when interacting with a factory or robotic facility, where the moving parts that can endanger human safety are simulated by the devices [35,80].
In several studies presented in previous sections that implemented virtual reality to provide a safe environment for experimentation, it became clear that the results are very similar to those obtainable in a real environment. Safety in robotic environments is critical, as the integrity of the users is a major concern [80].

4. Discussion

The purpose of this section is to offer a general overview of the aforementioned studies. This is an important contribution to the world of reinforcement learning in collaborative environments where the human is taken into account, and virtual reality is the key for testing.
For this reason, the first part of this section provides a categorization of the most relevant studies and the areas they target. This organizational structure can be useful for future projects where the main objectives are focused on the intersection of the four areas proposed in this article, namely reinforcement learning, virtual reality, collaborative robotics, and human–robot sensors such as EEG. The principal research studies directly related to these topics are presented in Table 1.
In the above table presenting papers in which the four main areas of the review article intersect, clear patterns and trends are observed. One notable finding is the high prevalence of studies combining human-factor sensors with collaborative robotics. This trend suggests a strong synergy and interest in human-centered robotics applications. Furthermore, the integration of reinforcement learning in several of these studies indicates an inclination towards exploring advanced machine learning techniques in the context of human–robot interaction.
Immersive technologies, on the other hand, seem to be a less explored area in comparison. However, when immersive technologies are mentioned, it is often in combination with reinforcement learning. This could suggest an emerging field of interest that integrates augmented or virtual reality with adaptive learning strategies. The absence of certain thematic combinations, such as immersive technologies and human-factor sensing with the inclusion of other topics, highlights possible areas of opportunity for future research [83,84]. The table effectively underscores the significance of the research, emphasizing the key intersections among the various themes. Despite the rising trend in these themes, it is apparent that there is a notable gap in substantial research involving the interaction of all four themes, underscoring their importance and potential for groundbreaking study.
After conducting the analysis, it is critical to investigate potential future directions and challenges that researchers could encounter. This section acts as an essential guide, directing individuals and organizations to well-informed choices and innovative methods that will define the path ahead in several areas of research in collaborative robotics, reinforcement learning, and human factors.
One of the key future directions for the development of high-impact projects involves the ability to adapt a robot’s behavior according to the emotions experienced by a human [41,79]. This approach is of particular importance in the medical field, where robot adaptation to patient stress and anxiety is essential to provide effective and comfortable support [52]. However, this adaptation is not limited to the medical field; in any environment with collaborative robots, controlling and adjusting the level of stress to which the user is subjected is imperative to ensure efficient and respectful interactions in accordance with people’s emotional and psychological needs.
Building on the approach outlined in the previous paragraph, future studies in the field could focus on achieving substantial changes in robot behavior depending on the emotions and stress level of a specific user. This would require parametrization of stress signals to determine when a subject is in a non-optimal emotional state. Once it is possible to detect whether a user is under high levels of stress or not in real time [33], reinforcement learning can be applied to achieve emergent behaviors in robots. For training, different strategies can be considered—either real-time training with human subjects or offline training [50,81] with models that replicate previously recorded human behaviors and emotions.
For the recording of a user’s emotions, the various techniques discussed throughout this review are available. Although it is true that all of them can be applied successfully, there are pros and cons to each of them. That is why it is necessary to consider which one is the most suitable for each investigation.
The feelings and emotions of a human when interacting with a robot can be recorded in a real environment or in a virtual environment. In terms of human safety, a virtual environment can be considered, which can also easily be adapted to the needs of a given experiment [80].
In addition, for the results to be truly relevant, the experimental phase must be carried out with enough subjects to reaffirm a valid theory and behavior. That is the reason why a large number of users should be considered in order to train a robot. However, it is necessary to test the robot’s behavior with a different set of subjects than those used during training to verify the independence of the obtained results and the training process.
Future research projects in the area of robotics, reinforcement learning, and human factors can follow the guidelines mentioned above. However, there are still some unresolved questions that can be addressed in order to carry out a much more promising project. For example, can multiple collaborative robots, under a single controlling brain, work alongside a user while considering the user’s emotional state?
One of the most interesting prospects for the future of reinforcement learning is the possibility of conducting training in real environments using the technique known as “sim2real” (simulation to real world). This promising direction looks to apply knowledge acquired in virtual environments, where iteration and learning are safer and more efficient, to real robots operating in the physical world [85]. This can open up new opportunities for automation and robotics in a wide range of applications, from manufacturing to space exploration, while addressing the challenges inherent in transferring skills from a simulation to the real world. This sim2real transition has the potential to transform the way robots learn and adapt to real-world environments.
The integration of collaborative robotics and reinforcement learning in the workplace holds the promise to significantly impact economic and social aspects. These technologies offer the promise of automating repetitive and laborious tasks, thereby enhancing worker well-being and productivity [86]. However, this shift towards automation also presents challenges and opportunities, particularly in terms of its effects on job markets and the emergence of new types of employment [87]. Understanding the implications of these changes for job markets is noteworthy, as it can shed light on the potential effects on job satisfaction and workplace injuries [88]. By focusing on the human aspect of technological integration, it is possible to pave the way for a future where technology complements human capabilities, fostering collaboration and well-being [89]. Future investigations and projects may focus on such human–robot collaboration to improve human welfare and achieve better productivity. An interesting idea could be to achieve the proposed outcome using reinforcement learning and different measurement devices, as discussed throughout this paper.
Another hot topic is the concept of Human Digital Twins (HDTs), which represents a relatively new and powerful approach in the field of human–robot interaction. This means creating detailed digital representations of human beings, offering a new opportunity to enhance how humans interact with robotic systems. The technology is in its early stages; however, it holds the promise of facilitating more intuitive, efficient, and personalized interactions between humans and robots [82].
As we venture deeper into the realm of collaborative robotics, reinforcement learning, and human-robot interaction, it is imperative to address the ethical considerations that come with these technological advancements. The integration of robots into human-centric environments raises important questions about privacy, autonomy, and the potential for unintended consequences [90]. How do we ensure that these technologies respect individual privacy and autonomy? What measures do we take to prevent misuse or abuse of such technologies, particularly in sensitive areas like health care and personal assistance [91]? Furthermore, the potential emotional impact on humans interacting with robots, especially those designed to mimic human behaviors and emotions, must be carefully considered [49]. It is essential to develop and adhere to ethical guidelines and standards that prioritize human well-being, ensuring that the deployment of these technologies enhances rather than diminishes human experiences and values.
The application of collaborative robotics with reinforcement learning presents a very large horizon of possibilities. As further research continues to advance in this area, new paradigms and applications will emerge, aiming to transform a variety of sectors, from manufacturing and logistics to health care and space exploration. This article has explored some of the most exciting possibilities for the future of these technologies. In this context, we propose a series of actions that can serve as a guide for professional researchers interested in harnessing the full potential of human factors, collaborative robotics, and reinforcement learning.

5. Conclusions

In conclusion, this comprehensive analysis of reinforcement learning applied to collaborative robots shows the current state and potential of the intersection between robotics, reinforcement learning, virtual reality, and human factors. Numerous investigations have probed the intricate task of integrating user emotions and feelings into the design of collaborative artificial intelligence systems. It is important to note that a variety of studies and research journals were examined and classified in the previous section. Effort was focused on providing a cohesive and focused view of the topic.
This closing section intends to highlight the critical need for ongoing research and development in the realm of collaborative technologies, particularly those capable of comprehensively understanding and responding to human emotions. The vision for the future is one where human–machine interactions transcend the mechanical, moving towards a relationship characterized by intuitive and empathetic connections. As such, the next frontier in research should target the creation of collaborative robots that are not only aware of but can also adapt their behavior in real time to emotional states such as anxiety or stress. This adaptability, powered by reinforcement learning, promises to revolutionize human–robot interactions by making them dynamically responsive and deeply empathetic.
The importance of selecting accurate and effective methods for capturing human-related variables cannot be overstated. Whether these variables are recorded in real-life settings or through high-fidelity simulations, ensuring the integrity and applicability of research findings is paramount. This step is foundational in advancing the field and ensuring that the development of collaborative technologies is grounded in valid, actionable insights.
In this context, the aim is to facilitate the design and development of new devices capable of measuring different emotions and human-related variables more accurately. These advanced devices, leveraging emerging technologies and innovative methodologies, could offer an unprecedented window into the complexity of human experience, surpassing the limitations of existing instruments [92]. The ability to accurately measure complex emotional states and psychological variables in real time opens new avenues for understanding human–machine interaction, allowing collaborative robots and other technologies to adapt and respond more effectively and sensitively to human needs.
A testing process for these devices is necessary, requiring rigorous protocols to validate their efficacy, accuracy, and reliability across a variety of scenarios and populations. Thorough testing in controlled environments, followed by real-life situational trials, can ensure that these devices are not only innovative but also practical and applicable across a broad spectrum of applications, from medical to educational fields and beyond.
Exploring future directions and the challenges that lie at the intersection of collaborative robotics, reinforcement learning, and human interaction opens up opportunities for innovation. One particularly promising avenue is in the medical field, where adapting robot behavior to patient emotions could significantly enhance patient care. This application underscores the broader necessity of recording and analyzing user emotions across various contexts, whether they occur in the tangible world or within virtual environments.
For instance, in medical care, the ability to adapt to different patients’ emotions during ongoing treatment is essential. This adaptability can be particularly beneficial in diverse scenarios such as treating elderly patients, who may require a more comforting and reassuring approach, or in pediatric care, where children might need a more engaging and gentle interaction to feel safe and cooperative [93]. Additionally, in high-stress situations such as critical operations, where patients may feel anxious or scared, having technology that can detect and respond to these emotional states could allow for adjustments in treatment approaches, communication strategies, or environmental settings to alleviate stress and improve the overall care experience.
The implementation of emotionally aware robots in these scenarios not only has the potential to improve patient outcomes by providing care that is responsive to their emotional states but also to enhance the efficiency of healthcare providers [94]. By automating the adaptation to patient needs based on emotional cues, healthcare professionals can focus more on clinical tasks, knowing that the emotional and comfort needs of their patients are being addressed in a nuanced and responsive manner.
The importance of interdisciplinary research is noteworthy, combining insights from psychology, medical science, artificial intelligence, and robotics to create a holistic solution that addresses both the emotional and physical needs of patients [95]. The development of such technologies will require significant challenges to be overcome, including ensuring privacy, managing the complexity of emotional expressions across different cultures and individuals, and developing robust algorithms that can accurately interpret a wide range of emotional states and adjust behaviors in a helpful, non-intrusive way [96].
The integration of collaborative robotics, reinforcement learning, virtual reality, and EEG technologies represents a frontier in interdisciplinary research, promising innovative solutions across various domains. Before delving into the intricacies of this integration, it is essential to grasp the overarching depiction presented in the forthcoming diagram (Figure 7). This conceptual map illustrates the intersections between EEG, virtual reality (VR), reinforcement learning (RL), and collaborative robotics, delineating distinct research domains where these technologies converge. Each triplet within the diagram signifies a unique area of study, offering insights into potential research directions and pathways for further exploration.
In the first triplet (A), EEG + VR + RL, researchers could explore neuroadaptive environments, where virtual reality setups are enhanced by EEG and reinforcement learning. These environments dynamically adjust content based on users’ neural responses, offering personalized and effective learning experiences. Brain-driven virtual reality learning utilizes EEG and VR technologies to shape learning experiences, promoting immersive and adaptive learning processes.
In the second triplet (B), EEG + VR + Robotics, researchers can explore cognitive robotics interfaces, where EEG, VR, and robotics converge to enable direct control of robots within virtual environments using brain signals. This integration facilitates seamless human–robot interaction and task execution. Brain-controlled robots in virtual settings play a central role, utilizing EEG technology to empower users to control robots within virtual environments through brain signals. This innovation paves the way for intuitive and immersive human–robot interaction scenarios.
In parallel, the VR + RL + Robotics triplet (C) introduces immersive learning automation, where researchers integrate VR, reinforcement learning, and robotics to create immersive training environments where robots learn and adapt their behavior through interactions within virtual worlds, enhancing learning outcomes and skill acquisition. Simultaneously, learning-driven robotic behavior in virtual environments combines VR, reinforcement learning, and robotics to simulate learning scenarios where robots autonomously acquire new skills and behaviors, offering novel opportunities for training and experimentation.
The last triplet (D), formed by EEG, RL, and robotics, introduces brain-guided automation, which integrates EEG feedback and reinforcement learning into robotic systems, enabling adaptive behavior and decision making based on real-time cognitive signals, thus enhancing efficiency and adaptability in automated tasks. This integration not only optimizes the performance of the system but also promotes a higher level of interaction between human operators and robotic systems, making the technology more intuitive and responsive to human states and intentions.
In the previous paragraphs, the intersections of different technologies were grouped into triplets, examining how their combined capabilities can lead to more innovative and efficient systems. Although there is no existing research that clearly integrates all four technologies together, the potential benefits of such a comprehensive integration are significant. Given the rapid growth in each of these fields, combining them could pave the way for the development of robotic systems that are not only efficient but also acutely aware of their human operators and surroundings. Proposing robotic systems that can be operated through cognitive signals from humans to conduct training and exhibit behaviors generated through reinforcement learning in a virtual world seems like a promising direction. Such systems could revolutionize various sectors, including industrial, environmental, and even domestic settings, by introducing robots that are more efficient and more attuned to the nuances of human and environmental interactions.
However, it is important to acknowledge the limitations and potential points of criticism of these existing methods. Technically, integrating multiple advanced technologies presents several challenges. Issues such as data synchronization, real-time processing, and the development of robust and adaptive algorithms are significant obstacles. For instance, ensuring that EEG signals are accurately interpreted and translated into meaningful commands in real time requires sophisticated signal processing techniques and powerful computational resources. Similarly, combining VR and RL necessitates creating highly responsive virtual environments that can adapt to both the robot’s learning processes and the user’s interactions without noticeable latency.
The use of cognitive signals and bio-sensors raises several ethical concerns. Privacy is a major issue, as EEG data can potentially reveal sensitive information about a person’s mental state, health, and emotions. There is a risk of misuse of or unauthorized access to these data, which could lead to ethical and legal complications. Ensuring informed consent and implementing stringent data protection measures are essential to address these concerns. Moreover, the psychological impact of long-term interaction with robots and immersive virtual environments on users should be carefully studied to prevent any adverse effects.
From an implementation perspective, deploying such sophisticated systems in real-world environments demands substantial financial investment, extensive training for users, and rigorous testing to ensure safety and reliability. The initial costs of setting up advanced robotic systems integrated with EEG, VR, and RL can be prohibitive for many organizations. Additionally, users need to be adequately trained to interact with these systems effectively, which requires time and resources. Furthermore, rigorous testing under various conditions is necessary to ensure that these systems can operate safely and reliably in different environments, which can be a time-consuming and complex process.
Addressing these challenges is essential for the successful adoption and integration of these innovative technologies in collaborative robotics. Future research should focus on developing more efficient algorithms, enhancing data security, and creating cost-effective solutions to make these advanced systems more accessible and practical for widespread use. By overcoming these obstacles, the full potential of integrating EEG, VR, RL, and robotics can be unlocked, leading to advancements in various fields. Ultimately, the synergy of these technologies promises to significantly improve human–robot collaboration, fostering a future where machines not only assist but empathize and intuitively interact with their human counterparts, making technology more humane and effective. The path forward is promising, and with continued interdisciplinary efforts, the vision of emotionally aware, responsive, and adaptive collaborative robots can become a reality, revolutionizing industries and enhancing the quality of human life.

Author Contributions

Conceptualization, A.G.-S. and J.-I.V.; Methodology, A.G.-S. and J.-I.V.; Data curation, A.G.-S.; Validation, J.-I.V. and A.E.; Writing—original draft, A.G.-S.; Writing—review and editing, A.E., J.-I.V. and A.G.-S.; Supervision, J.-I.V. and A.E. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the project ACROBA, which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 101017284, and the project EGIA which has received funding from the ELKARTEK programme from the Basque Government.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not Applicable.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
ARAugmented Reality
BCIBrain–Computer Interface
BDIBeck Depression Inventory
BVPBlood Volume Pulse
CNNConvolutional Neural Network
ECGElectrocardiography
EEGElectroencephalography
EOGElectro-oculograph
ErrPError Potential-Related Event
ERPError-Related Potential
GSRGalvanic Skin Response
HARHuman–Robot Recognition
HDTHuman Digital Twin
HRIHuman–Robot Interaction
ICAIndependent Component Analysis
KNNK-Nearest Neighbor
MRMixed Reality
PSSPerceived Stress Questionnaire
RLReinforcement Learning
RSPRespiration Rate
SVMSupport Vector Machine
VRVirtual Reality

References

  1. Weiss, A.; Wortmeier, A.K.; Kubicek, B. Cobots in industry 4.0: A roadmap for future practice studies on human–robot collaboration. IEEE Trans. Hum. Mach. Syst. 2021, 51, 335–345. [Google Scholar] [CrossRef]
  2. Krüger, J.; Lien, T.; Verl, A. Cooperation of human and machines in assembly lines. CIRP Ann. 2009, 58, 628–646. [Google Scholar] [CrossRef]
  3. Baumgartner, M.; Kopp, T.; Kinkel, S. Analysing Factory Workers’ Acceptance of Collaborative Robots: A Web-Based Tool for Company Representatives. Electronics 2022, 11, 145. [Google Scholar] [CrossRef]
  4. Sherwani, F.; Asad, M.M.; Ibrahim, B. Collaborative Robots and Industrial Revolution 4.0 (IR 4.0). In Proceedings of the 2020 International Conference on Emerging Trends in Smart Technologies (ICETST), Karachi, Pakistan, 26–27 March 2020. [Google Scholar] [CrossRef]
  5. Parsons, H. Human factors in industrial robot safety. J. Occup. Accid. 1986, 8, 25–47. [Google Scholar] [CrossRef]
  6. Bauer, W.; Bender, M.; Braun, M.; Rally, P.; Scholtz, O. Lightweight Robots in Manual Assembly—Best to Start Simply! Examining Companies’ Initial Experiences with Lightweight Robots; Frauenhofer-Institut für Arbeitswirtschaft und Organisation IAO: Stuttgart, Germany, 2016. [Google Scholar]
  7. Pearce, M.; Mutlu, B.; Shah, J.; Radwin, R. Optimizing makespan and ergonomics in integrating collaborative robots into manufacturing processes. IEEE Trans. Autom. Sci. Eng. 2018, 15, 1772–1784. [Google Scholar] [CrossRef]
  8. Simone, V.D.; Pasquale, V.D.; Giubileo, V.; Miranda, S. Human-Robot Collaboration: An analysis of worker’s performance. Procedia Comput. Sci. 2022, 200, 1540–1549. [Google Scholar] [CrossRef]
  9. Kragic, D.; Gustafson, J.; Karaoguz, H.; Jensfelt, P.; Krug, R. Interactive, Collaborative Robots: Challenges and Opportunities. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 13–19 July 2018. [Google Scholar]
  10. Monari, E.; Avallone, G.; Valori, M.; Agostini, L.; Chen, Y.; Palazzi, E.; Vertechy, R. Physical Ergonomics Monitoring in Human–Robot Collaboration: A Standard-Based Approach for Hand-Guiding Applications. Machines 2024, 12, 231. [Google Scholar] [CrossRef]
  11. Sheridan, T.B. Human-robot interaction: Status and challenges. Hum. Factors 2016, 58, 525–532. [Google Scholar] [CrossRef]
  12. Kober, J.; Bagnell, J.A.; Peters, J. Reinforcement learning in robotics: A survey. Int. J. Robot. Res. 2013, 32, 1238–1274. [Google Scholar] [CrossRef]
  13. Kormushev, P.; Calinon, S.; Caldwell, D. Reinforcement Learning in Robotics: Applications and Real-World Challenges. Robotics 2013, 2, 122–148. [Google Scholar] [CrossRef]
  14. Brunke, L.; Greeff, M.; Hall, A.W.; Yuan, Z.; Zhou, S.; Panerati, J.; Schoellig, A.P. Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning. Annu. Rev. Control Robot. Auton. Syst. 2022, 5, 411–444. [Google Scholar] [CrossRef]
  15. Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction; MIT Press: Cambridge, MA, USA, 2018. [Google Scholar]
  16. Kiran, B.R.; Sobh, I.; Talpaert, V.; Mannion, P.; Sallab, A.A.; Yogamani, S.; P’erez, P. Deep Reinforcement Learning for Autonomous Driving: A Survey. IEEE Trans. Intell. Transp. Syst. 2020, 23, 4909–4926. [Google Scholar] [CrossRef]
  17. Towers, M.; Terry, J.K.; Kwiatkowski, A.; Balis, J.U.; de Cola, G.; Deleu, T.; Goulão, M.; Kallinteris, A.; KG, A.; Krimmel, M.; et al. Gymnasium, v0.28.1; Zenodo: Geneva, Switzerland, 2023. [CrossRef]
  18. Koenig, N.; Howard, A. Design and use paradigms for Gazebo, an open-source multi-robot simulator. In Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), Sendai, Japan, 28 September–2 October 2004; Volume 3, pp. 2149–2154. [Google Scholar] [CrossRef]
  19. Maurtua, I.; Ibarguren, A.; Kildal, J.; Susperregi, L.; Sierra, B. Human–robot collaboration in industrial applications: Safety, interaction and trust. Int. J. Adv. Robot. Syst. 2017, 14, 172988141771601. [Google Scholar] [CrossRef]
  20. Wang, W.; Chen, Y.; Li, R.; Jia, Y. Learning and comfort in human–robot interaction: A review. Appl. Sci. 2019, 9, 5152. [Google Scholar] [CrossRef]
  21. Sawangjai, P.; Hompoonsup, S.; Leelaarporn, P.; Kongwudhikunakorn, S.; Wilaiprasitporn, T. Consumer Grade EEG Measuring Sensors as Research Tools: A Review. IEEE Sens. J. 2020, 20, 3996–4024. [Google Scholar] [CrossRef]
  22. Burdea, G. Invited review: The synergy between virtual reality and robotics. IEEE Trans. Robot. Autom. 1999, 15, 400–410. [Google Scholar] [CrossRef]
  23. Kaelbling, L.P.; Littman, M.L.; Moore, A.W. Reinforcement learning: A survey. J. Artif. Intell. Res. 1996, 4, 237–285. [Google Scholar] [CrossRef]
  24. Salazar-Gomez, A.F.; DelPreto, J.; Gil, S.; Guenther, F.H.; Rus, D. Correcting robot mistakes in real time using EEG signals. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017. [Google Scholar]
  25. Goodrich, M.A.; Schultz, A.C. Human-Robot Interaction: A Survey. Found. Trends Hum.-Comput. Interact. 2007, 1, 203–275. [Google Scholar] [CrossRef]
  26. Rodrigues, P.B.; Singh, R.; Oytun, M.; Adami, P.; Woods, P.J.; Becerik-Gerber, B.; Soibelman, L.; Copur-Gencturk, Y.; Lucas, G.M. A multidimensional taxonomy for human-robot interaction in construction. Autom. Constr. 2023, 150, 104845. [Google Scholar] [CrossRef]
  27. Slater, M.; Sanchez-Vives, M.V. Enhancing our lives with immersive virtual reality. Front. Robot. AI 2016, 3, 74. [Google Scholar] [CrossRef]
  28. Freeman, D.; Reeve, S.; Robinson, A.; Ehlers, A.; Clark, D.; Spanlang, B.; Slater, M. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol. Med. 2017, 47, 2393–2400. [Google Scholar] [CrossRef] [PubMed]
  29. Google Scholar Search Engine. Available online: https://scholar.google.com (accessed on 11 April 2024).
  30. Scopus Database. Available online: https://www.scopus.com (accessed on 11 April 2024).
  31. Web of Science. Available online: https://www.webofscience.com/wos (accessed on 11 April 2024).
  32. Urrútia, G.; Bonfill, X. Declaración PRISMA: Una propuesta para mejorar la publicación de revisiones sistemáticas y metaanálisis. Med. Clín. 2010, 135, 507–511. [Google Scholar] [CrossRef] [PubMed]
  33. Toichoa Eyam, A.; Mohammed, W.M.; Martinez Lastra, J.L. Emotion-driven analysis and control of human-robot interactions in collaborative applications. Sensors 2021, 21, 4626. [Google Scholar] [CrossRef]
  34. Alarcão, S.M.; Fonseca, M.J. Emotions Recognition Using EEG Signals: A Survey. IEEE Trans. Affect. Comput. 2019, 10, 374–393. [Google Scholar] [CrossRef]
  35. Shayesteh, S.; Ojha, A.; Jebelli, H. Workers’ trust in collaborative construction robots: EEG-based trust recognition in an immersive environment. In Automation and Robotics in the Architecture, Engineering, and Construction Industry; Springer International Publishing: Cham, Switzerland, 2022; pp. 201–215. [Google Scholar]
  36. Caruana, R.; Niculescu-Mizil, A. An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA, 25–29 June 2006. [Google Scholar] [CrossRef]
  37. Pontil, M.; Verri, A. Properties of Support Vector Machines. Neural Comput. 1998, 10, 955–974. [Google Scholar] [CrossRef] [PubMed]
  38. Akinola, I.; Wang, Z.; Shi, J.; He, X.; Lapborisuth, P.; Xu, J.; Watkins-Valls, D.; Sajda, P.; Allen, P. Accelerated Robot Learning via Human Brain Signals. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation, Paris, France, 31 May–31 August 2019. [Google Scholar] [CrossRef]
  39. Zhang, R.; Lv, Q.; Li, J.; Bao, J.; Liu, T.; Liu, S. A reinforcement learning method for human-robot collaboration in assembly tasks. Robot. Comput. Integr. Manuf. 2022, 73, 102227. [Google Scholar] [CrossRef]
  40. Lagomarsino, M.; Lorenzini, M.; Constable, M.D.; De Momi, E.; Becchio, C.; Ajoudani, A. Maximising Coefficiency of Human-Robot Handovers through Reinforcement Learning. IEEE Robot. Autom. Lett. 2023, 8, 4378–4385. [Google Scholar] [CrossRef]
  41. Iturrate, I.; Montesano, L.; Minguez, J. Robot reinforcement learning using EEG-based reward signals. In Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, AK, USA, 3–7 May 2010. [Google Scholar]
  42. Millán, J.D.R.; Rupp, R.; Müller-Putz, G.R.; Murray-Smith, R.; Giugliemma, C.; Tangermann, M.; Vidaurre, C.; Cincotti, F.; Kübler, A.; Leeb, R.; et al. Combining brain-computer interfaces and assistive technologies: State-of-the-art and challenges. Front. Neurosci. 2010, 4, 161. [Google Scholar] [CrossRef] [PubMed]
  43. Gehring, W.J.; Goss, B.; Coles, M.G.H.; Meyer, D.E.; Donchin, E. A Neural System for Error Detection and Compensation. Psychol. Sci. 1993, 4, 385–390. [Google Scholar] [CrossRef]
  44. Watkins, C.J.C.H.; Dayan, P. Q-learning. Mach. Learn. 1992, 8, 279–292. [Google Scholar] [CrossRef]
  45. Kar, R.; Ghosh, L.; Konar, A.; Chakraborty, A.; Nagar, A.K. EEG-induced autonomous game-teaching to a robot arm by human trainers using reinforcement learning. IEEE Trans. Games 2022, 14, 610–622. [Google Scholar] [CrossRef]
  46. Yeung, N.; Botvinick, M.M.; Cohen, J.D. The neural basis of error detection: Conflict monitoring and the error-related negativity. Psychol. Rev. 2004, 111, 931–959. [Google Scholar] [CrossRef] [PubMed]
  47. Ferrez, P.W.; del R Millan, J. Error-related EEG potentials generated during simulated brain-computer interaction. IEEE Trans. Biomed. Eng. 2008, 55, 923–929. [Google Scholar] [CrossRef] [PubMed]
  48. Lotte, F.; Congedo, M.; Lécuyer, A.; Lamarche, F.; Arnaldi, B. A Review of Classification Algorithms for EEG-based Brain–computer Interfaces. J. Neural Eng. 2007, 4, R1. [Google Scholar] [CrossRef] [PubMed]
  49. Borboni, A.; Elamvazuthi, I.; Cusano, N. EEG-based empathic safe cobot. Machines 2022, 10, 603. [Google Scholar] [CrossRef]
  50. Luo, T.J.; Fan, Y.C.; Lv, J.T.; Zhou, C.L. Deep reinforcement learning from error-related potentials via an EEG-based brain-computer interface. In Proceedings of the 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Madrid, Spain, 3–6 December 2018. [Google Scholar]
  51. Shu, L.; Xie, J.; Yang, M.; Li, Z.; Li, Z.; Liao, D.; Xu, X.; Yang, X. A Review of Emotion Recognition Using Physiological Signals. Sensors 2018, 18, 2074. [Google Scholar] [CrossRef] [PubMed]
  52. Onose, G.; Grozea, C.; Anghelescu, A.; Daia, C.; Sinescu, C.J.; Ciurea, A.V.; Spircu, T.; Mirea, A.; Andone, I.; Spânu, A.; et al. On the feasibility of using motor imagery EEG-based brain-computer interface in chronic tetraplegics for assistive robotic arm control: A clinical test and long-term post-trial follow-up. Spinal Cord 2012, 50, 599–608. [Google Scholar] [CrossRef] [PubMed]
  53. Onnasch, L.; Schweidler, P.; Schmidt, H. The potential of robot eyes as predictive cues in HRI-an eye-tracking study. Front. Robot. AI 2023, 10, 1178433. [Google Scholar] [CrossRef]
  54. Mariscal, M.A.; Ortiz Barcina, S.; García Herrero, S.; López Perea, E.M. Working with collaborative robots and its influence on levels of working stress. Int. J. Comput. Integr. Manuf. 2024, 37, 900–919. [Google Scholar] [CrossRef]
  55. Pérez, L.; Rodríguez, Í.; Rodríguez, N.; Usamentiaga, R.; García, D.F. Robot guidance using machine vision techniques in industrial environments: A comparative review. Sensors 2016, 16, 335. [Google Scholar] [CrossRef]
  56. Beddiar, D.R.; Nini, B.; Sabokrou, M.; Hadid, A. Vision-based human activity recognition: A survey. Multimed. Tools Appl. 2020, 79, 30509–30555. [Google Scholar] [CrossRef]
  57. Zhu, X.; Liang, Y.; Sun, H.; Wang, X.; Ren, B. Robot obstacle avoidance system using deep reinforcement learning. Ind. Robot 2022, 49, 301–310. [Google Scholar] [CrossRef]
  58. Mohindru, V.; Singla, S. A review of anomaly detection techniques using computer vision. In Recent Innovations in Computing: Proceedings of ICRIC 2020, Jammu, India, 13–14 May 2022; Lecture Notes in Electrical Engineering; Springer: Singapore, 2021; pp. 669–677. [Google Scholar]
  59. Stamler, J.S.; Goldman, M.E.; Gomes, J.; Matza, D.; Horowitz, S.F. The effect of stress and fatigue on cardiac rhythm in medical interns. J. Electrocardiol. 1992, 25, 333–338. [Google Scholar] [CrossRef]
  60. Xintarakou, A.; Sousonis, V.; Asvestas, D.; Vardas, P.E.; Tzeis, S. Remote cardiac rhythm monitoring in the era of smart wearables: Present assets and future perspectives. Front. Cardiovasc. Med. 2022, 9, 853614. [Google Scholar] [CrossRef]
  61. Hellhammer, D.H.; Wüst, S.; Kudielka, B.M. Salivary cortisol as a biomarker in stress research. Psychoneuroendocrinology 2009, 34, 163–171. [Google Scholar] [CrossRef]
  62. Carere, C.; Van Oers, K. Shy and bold great tits (Parus major): Body temperature and breath rate in response to handling stress. Physiol. Behav. 2004, 82, 905–912. [Google Scholar] [CrossRef]
  63. Leary, M.R.; Britt, T.W.; Cutlip, W.D.; Templeton, J.L. Social blushing. Psychol. Bull. 1992, 112, 446–460. [Google Scholar] [CrossRef]
  64. Jackson-Koku, G. Beck depression inventory. Occup. Med. 2016, 66, 174–175. [Google Scholar] [CrossRef]
  65. Beck, A.T.; Steer, R.A.; Brown, G. Beck Depression Inventory–II. In Psychological Assessment; Psychological Corporation: San Antonio, TX, USA, 1996. [Google Scholar] [CrossRef]
  66. Jumani, A.K.; Siddique, W.A.; Laghari, A.A.; Abro, A.; Khan, A.A. Virtual reality and augmented reality for education. In Multimedia Computing Systems and Virtual Reality; CRC Press: Boca Raton, FL, USA, 2022; pp. 189–210. [Google Scholar]
  67. LaValle, S.M. Virtual Reality; Cambridge University Press: Cambridge, UK, 2023. [Google Scholar]
  68. Parong, J.; Mayer, R.E. Learning science in immersive virtual reality. J. Educ. Psychol. 2018, 110, 785–797. [Google Scholar] [CrossRef]
  69. Brenneis, D.J.A.; Parker, A.S.; Johanson, M.B.; Butcher, A.; Davoodi, E.; Acker, L.; Botvinick, M.M.; Modayil, J.; White, A.; Pilarski, P.M. Assessing Human Interaction in Virtual Reality With Continually Learning Prediction Agents Based on Reinforcement Learning Algorithms: A Pilot Study. arXiv 2021, arXiv:2112.07774. [Google Scholar] [CrossRef]
  70. Caudell, T.P.; Mizell, D.W. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences, Kauai, HI, USA, 7–10 January 1992. [Google Scholar]
  71. Craig, A.B. Understanding Augmented Reality: Concepts and Applications; Morgan Kaufmann: Burlington, MA, USA, 2013. [Google Scholar]
  72. Berryman, D.R. Augmented reality: A review. Med. Ref. Serv. Q. 2012, 31, 212–218. [Google Scholar] [CrossRef]
  73. Hughes, C.; Stapleton, C.; Hughes, D.; Smith, E. Mixed reality in education, entertainment, and training. IEEE Comput. Graph. Appl. 2005, 25, 24–30. [Google Scholar] [CrossRef]
  74. Chen, J.; Fu, Y.; Lu, W.; Pan, Y. Augmented reality-enabled human-robot collaboration to balance construction waste sorting efficiency and occupational safety and health. J. Environ. Manag. 2023, 348, 119341. [Google Scholar] [CrossRef]
  75. Szczurek, K.A.; Cittadini, R.; Prades, R.M.; Matheson, E.; Di Castro, M. Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality. IEEE Access 2023, 11, 39555–39576. [Google Scholar] [CrossRef]
  76. Covaciu, F.; Crisan, N.; Vaida, C.; Andras, I.; Pusca, A.; Gherman, B.; Radu, C.; Tucan, P.; Al Hajjar, N.; Pisla, D. Integration of Virtual Reality in the Control System of an Innovative Medical Robot for Single-Incision Laparoscopic Surgery. Sensors 2023, 23, 5400. [Google Scholar] [CrossRef]
  77. Lee, J.Y.; Mucksavage, P.; Kerbl, D.C.; Huynh, V.B.; Etafy, M.; McDougall, E.M. Validation Study of a Virtual Reality Robotic Simulator—Role as an Assessment Tool? J. Urol. 2012, 187, 998–1002. [Google Scholar] [CrossRef]
  78. Wang, Q.; Jiao, W.; Yu, R.; Johnson, M.T.; Zhang, Y. Virtual Reality Robot-Assisted Welding Based on Human Intention Recognition. IEEE Trans. Autom. Sci. Eng. 2020, 17, 799–808. [Google Scholar] [CrossRef]
  79. Rovira, A.; Slater, M. Encouraging bystander helping behaviour in a violent incident: A virtual reality study using reinforcement learning. Sci. Rep. 2022, 12, 3843. [Google Scholar] [CrossRef]
  80. Badia, S.B.i.; Silva, P.A.; Branco, D.; Pinto, A.; Carvalho, C.; Menezes, P.; Almeida, J.; Pilacinski, A. Virtual reality for safe testing and development in collaborative robotics: Challenges and perspectives. Electronics 2022, 11, 1726. [Google Scholar] [CrossRef]
  81. Ghadirzadeh, A.; Chen, X.; Yin, W.; Yi, Z.; Björkman, M.; Kragic, D. Human-centered collaborative robots with deep reinforcement learning. IEEE Robot. Autom. Lett. 2020, 6, 566–571. [Google Scholar] [CrossRef]
  82. Wang, B.; Zhou, H.; Li, X.; Yang, G.; Zheng, P.; Song, C.; Yuan, Y.; Wuest, T.; Yang, H.; Wang, L. Human Digital Twin in the context of Industry 5.0. Robot. Comput. Integr. Manuf. 2024, 85, 102626. [Google Scholar] [CrossRef]
  83. Saghafian, M.; Sitompul, T.; Laumann, K.; Sundnes, K.; Lindell, R. Application of Human Factors in the Development Process of Immersive Visual Technologies: Challenges and Future Improvements. Front. Psychol. 2021, 12, 634352. [Google Scholar] [CrossRef]
  84. Farias, M.C.Q. Quality of Experience of Immersive Media—New Challenges. In Proceedings of the 29th Brazilian Symposium on Multimedia and the Web, Ribeirão Preto, Brazil, 23–27 October 2023. [Google Scholar] [CrossRef]
  85. Zhao, W.; Queralta, J.P.; Westerlund, T. Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: A Survey. In Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, ACT, Australia, 1–4 December 2020. [Google Scholar] [CrossRef]
  86. Marmpena, M.; Garcia, F.; Lim, A.; Hemion, N.; Wennekers, T. Data-driven emotional body language generation for social robotics. arXiv 2022, arXiv:2205.00763. [Google Scholar]
  87. Paolillo, A.; Colella, F.; Nosengo, N.; Schiano, F.; Stewart, W.; Zambrano, D.; Chappuis, I.; Lalive, R.; Floreano, D. How to compete with robots by assessing job automation risks and resilient alternatives. Sci. Robot. 2022, 7, eabg5561. [Google Scholar] [CrossRef]
  88. Koster, S.; Brunori, C. What to do when the robots come? Non-formal education in jobs affected by automation. Int. J. Manpow. 2021, 42, 1397–1419. [Google Scholar] [CrossRef]
  89. Dunstan, B.J.; Koh, J.T.K.V. A cognitive model for human willingness in human-robot interaction development. In Proceedings of the SIGGRAPH Asia 2014 Designing Tools For Crafting Interactive Artifacts, Shenzhen, China, 3–6 December 2014. [Google Scholar]
  90. van Maris, A.; Zook, N.; Caleb-Solly, P.; Studley, M.; Winfield, A.; Dogramadzi, S. Designing ethical social robots—A longitudinal field study with older adults. Front. Robot. AI 2020, 7, 1. [Google Scholar] [CrossRef]
  91. Draper, H.; Sorell, T. Ethical values and social care robots for older people: An international qualitative study. Ethics Inf. Technol. 2017, 19, 49–68. [Google Scholar] [CrossRef]
  92. Pal, S.; Mukhopadhyay, S.; Suryadevara, N. Development and Progress in Sensors and Technologies for Human Emotion Recognition. Sensors 2021, 21, 5554. [Google Scholar] [CrossRef]
  93. Logan, D.; Breazeal, C.; Goodwin, M.; Jeong, S.; O’Connell, B.; Smith-Freedman, D.; Heathers, J.A.J.; Weinstock, P. Social Robots for Hospitalized Children. Pediatrics 2019, 144, e20181511. [Google Scholar] [CrossRef]
  94. Swangnetr, M.; Kaber, D. Emotional State Classification in Patient–Robot Interaction Using Wavelet Analysis and Statistics-Based Feature Selection. IEEE Trans. Hum.-Mach. Syst. 2013, 43, 63–75. [Google Scholar] [CrossRef]
  95. Rudd, I. Leveraging Artificial Intelligence and Robotics to Improve Mental Health. Intellect. Arch. 2022, 11, 3. [Google Scholar] [CrossRef]
  96. Esposito, A.; Esposito, A.; Vogel, C. Needs and challenges in human computer interaction for processing social emotional information. Pattern Recognit. Lett. 2015, 66, 41–51. [Google Scholar] [CrossRef]
Applsci 14 06345 g001
Figure 1. The various levels of cooperation between a human worker and a robot.
Figure 1. The various levels of cooperation between a human worker and a robot.
Applsci 14 06345 g001
Applsci 14 06345 g002
Figure 2. Classical RL loop [15].
Figure 2. Classical RL loop [15].
Applsci 14 06345 g002
Applsci 14 06345 g003
Figure 3. Percentage increase in publications across topic clusters over time (2012–2023).
Figure 3. Percentage increase in publications across topic clusters over time (2012–2023).
Applsci 14 06345 g003
Applsci 14 06345 g004
Figure 4. The five different brain waves: Delta, theta, alpha, beta, and gamma.
Figure 4. The five different brain waves: Delta, theta, alpha, beta, and gamma.
Applsci 14 06345 g004
Applsci 14 06345 g005
Figure 5. Real experimentation using EEG for an assembly task [33].
Figure 5. Real experimentation using EEG for an assembly task [33].
Applsci 14 06345 g005
Applsci 14 06345 g006
Figure 6. Different bio-sensors and their positions in the human body [51].
Figure 6. Different bio-sensors and their positions in the human body [51].
Applsci 14 06345 g006
Applsci 14 06345 g007
Figure 7. Conceptual diagram. Intersection between topics displayed in triplets for future research.
Figure 7. Conceptual diagram. Intersection between topics displayed in triplets for future research.
Applsci 14 06345 g007
Table 1. Categorical classification of relevant documents at the intersection of the four main areas of the review article. The documents are listed and organized by publication year from earliest to latest.
Table 1. Categorical classification of relevant documents at the intersection of the four main areas of the review article. The documents are listed and organized by publication year from earliest to latest.
Document ReferenceReinforcement LearningImmersive TechnologiesHuman-Factor SensorsCollaborative Robotics
Iturrate, I. et al. (2010) [41]✓✓ ✓
Onose, G. et al. (2012) [52] ✓✓
Kragic, D. et al. (2016) [9] ✓✓
Sheridan, T.B. et al. (2016) [11] ✓✓
Salazar-Gomez, A.F. et al. (2017) [24] ✓✓
Luo, T.J. et al. (2018) [50]✓ ✓✓
Pearce, M. et al. (2018) [7] ✓
Ghadirzadeh, A. et al. (2020) [81]✓ ✓✓
Brenneis, D.J.A. et al. (2021) [69]✓✓✓
Toichoa Eyam, A. et al. (2021) [33] ✓✓
Borboni, A. et al. (2022) [49] ✓✓
Shayesteh, S. et al. (2022) [35] ✓✓✓
Zhang, R. et al. (2022) [39]✓ ✓✓
Rovira, A. et al. (2022) [79]✓✓
Kar, R. et al. (2022) [45]✓ ✓✓
Badia, S.B.i. et al. (2022) [80] ✓ ✓
Simone, V.D. et al. (2022) [8] ✓
Zhu, X. et al. (2022) [57]✓ ✓✓
Wang, B. et al. (2022) [82]✓ ✓✓
Lagomarsino, M. et al. (2023) [40]✓ ✓
Note: ✓ indicates that the topic is relevant throughout the document.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Gonzalez-Santocildes, A.; Vazquez, J.-I.; Eguiluz, A. Enhancing Robot Behavior with EEG, Reinforcement Learning and Beyond: A Review of Techniques in Collaborative Robotics. Appl. Sci. 2024, 14, 6345. https://doi.org/10.3390/app14146345

AMA Style

Gonzalez-Santocildes A, Vazquez J-I, Eguiluz A. Enhancing Robot Behavior with EEG, Reinforcement Learning and Beyond: A Review of Techniques in Collaborative Robotics. Applied Sciences. 2024; 14(14):6345. https://doi.org/10.3390/app14146345

Chicago/Turabian Style

Gonzalez-Santocildes, Asier, Juan-Ignacio Vazquez, and Andoni Eguiluz. 2024. "Enhancing Robot Behavior with EEG, Reinforcement Learning and Beyond: A Review of Techniques in Collaborative Robotics" Applied Sciences 14, no. 14: 6345. https://doi.org/10.3390/app14146345

APA Style

Gonzalez-Santocildes, A., Vazquez, J. -I., & Eguiluz, A. (2024). Enhancing Robot Behavior with EEG, Reinforcement Learning and Beyond: A Review of Techniques in Collaborative Robotics. Applied Sciences, 14(14), 6345. https://doi.org/10.3390/app14146345

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop