research-article

Open access

Planning to Minimize the Human Muscular Effort during Forceful Human-Robot Collaboration

Authors:

Luis F. C. Figueredo,

Rafael De Castro Aguiar,

Lipeng Chen,

Thomas C. Richards,

Samit Chakrabarty,

Mehmet DogarAuthors Info & Claims

ACM Transactions on Human-Robot Interaction (THRI), Volume 11, Issue 1

Article No.: 10, Pages 1 - 27

https://doi.org/10.1145/3481587

Published: 18 October 2021 Publication History

All formats PDF

Abstract

This work addresses the problem of planning a robot configuration and grasp to position a shared object during forceful human-robot collaboration, such as a puncturing or a cutting task. Particularly, our goal is to find a robot configuration that positions the jointly manipulated object such that the muscular effort of the human, operating on the same object, is minimized while also ensuring the stability of the interaction for the robot. This raises three challenges. First, we predict the human muscular effort given a human-robot combined kinematic configuration and the interaction forces of a task. To do this, we perform task-space to muscle-space mapping for two different musculoskeletal models of the human arm. Second, we predict the human body kinematic configuration given a robot configuration and the resulting object pose in the workspace. To do this, we assume that the human prefers the body configuration that minimizes the muscular effort. And third, we ensure that, under the forces applied by the human, the robot grasp on the object is stable and the robot joint torques are within limits. Addressing these three challenges, we build a planner that, given a forceful task description, can output the robot grasp on an object and the robot configuration to position the shared object in space. We quantitatively analyze the performance of the planner and the validity of our assumptions. We conduct experiments with human subjects to measure their kinematic configurations, muscular activity, and force output during collaborative puncturing and cutting tasks. The results illustrate the effectiveness of our planner in reducing the human muscular load. For instance, for the puncturing task, our planner is able to reduce muscular load by $69.5\%$ compared to a user-based selection of object poses.

1 Introduction

When a human and a robot jointly manipulate an object, the robot must take into account the human kinematic and biomechanical response to different possible robot actions. If the robot can predict such response, then it can plan actions that reduce the biomechanical load on the human, leading to more efficient and comfortable human-robot collaboration.

Fig. 1.

In this work, we present a method that simultaneously predicts the kinematic and biomechanical response of the human during forceful human-robot collaboration (fHRC). We then use these predictions to plan robot grasps and configurations that reduce the load on the human biomechanical system, particularly by minimizing the predicted human muscular effort.

Take for example the collaborative tasks in Figure 1. We show a ‘puncturing’ task (top left) where the human forcefully inserts a pointed tool into a hard foam board at a designated point on the board. In the top row (i.e., Figure 1(A), (C), and (E)), we show three different human-robot configurations to perform the same puncturing task. Notice that, given the same task (e.g., puncturing), the human body takes a different kinematic configuration, and exerts different amounts of muscular effort, depending on where the robot holds the object (the board). In this work, we explore this interaction. We present a planner that, given a forceful task like puncturing or cutting, intelligently decides on the robot configuration, grasp, and position of the object to proactively instruct humans to configurations that demand less physical muscular effort. For example, in Figure 1, our planner produces the robot configuration in (A) for puncturing and (B) for cutting. After the robot attains the planned configuration, the human performs the task. ((C) and (D) are randomized configurations; (E) and (F) are human-designed configurations.)

To produce such configurations, our planner solves three problems: (i) predicting the human muscular effort, (ii) predicting the human kinematic configuration, and (iii) ensuring the stability of the robot grasp and joints.

Predicting the human muscular effort. Given a particular human biomechanical model, kinematic configuration (i.e., the joint angles of a kinematic model of the human body), the object pose, and an estimate of the tooltip forces required to complete a task (e.g., puncturing), our planner predicts the human muscular effort required.

To do this, we transform cooperative task-space forces, to human muscle-space—where human motor-control signals are applied and muscle-tendon forces generate the output joint torques. The muscle-space actuation is nonetheless highly redundant (take for instance, the seven-joint kinematic model in the work of Saul et al. [45] that is actuated by a set of 50 muscles). Consequently, a given forceful human-robot collaborative operation often has infinite solutions in the muscle-space. To solve this indeterminacy, we use two well-known approaches for dimensionality reduction in muscle-space. These produce two fundamentally different muscle activity prediction paradigms—and therefore two different planners. Our main objective in using both methods is to understand the influence of redundancy resolution techniques over the actual minimization of muscle effort. We present the details of how we predict the muscular effort in Section 4.

Predicting the human kinematic configuration. In the preceding discussion, to predict the muscular effort, we assumed that we know the kinematic configuration of the human body. However, there is also kinematic redundancy in the human body. Given an object pose, the human can choose among many different body configurations to perform the task. Therefore, when the robot is choosing an object pose, before it can predict the muscular effort, it first needs to predict the likely kinematic configuration the human will choose. In this work, we focus on a seven-joint model of the human shoulder, elbow, and wrist. In most literature [5, 26, 27, 52], it is assumed that humans resolve this redundancy in a consistent manner: choosing a limb configuration that minimizes energy expenditure in terms of muscle activity. Therefore, in this work, we assume that the human chooses the kinematic configuration that results in the minimum muscle effort. We name this the kinematic minimum effort assumption (KMEA). We present the details of how we predict the human kinematic configuration through this assumption in Section 5.

Ensuring the stability of the robot grasp and joints. Finally, it is also important for the planner to choose (i) grasp points on the object such that the external human applied force can be resisted in a stable manner (e.g., the object does not slip through the fingers) and (ii) robot arm configurations such that the predicted torque load on the robot joints are within limits. These constraints must be satisfied while the predicted human muscle effort, as explained earlier, is minimized, therefore tightly coupling these different constraints and requiring a simultaneous solution to the human- and robot-related constraints. We explain the details of how we model the stability of the robot grasp and robot joints in Section 6. To do this, we mostly rely on an existing formulation of ours from previous work [15] but integrate it here with human-related constraints.

To evaluate the performance of the proposed planner, we conducted fHRC experiments in $\text{puncturing}$ and cutting tasks, examples of which are shown in Figure 1. These were performed with different participants and conditions—measuring task-space forces, joint-space kinematics (using a motion-capture system), muscle-space biofeedback (using nine electromyography (EMG) sensors), and participant perception. We particularly analyze (i) whether the human muscular effort, as indicated by EMG readings, is indeed reduced when we use the proposed planners and (ii) how well the predictions of KMEA agrees with the actual human kinematic configurations, as indicated by the motion-capture readings. The results show a reduction in the muscle effort when our planner is used and show KMEA to be effective, although open to improvement.¹

We call our proposed planner the comfort-based planner since achieving comfortable human-robot collaboration is our overall goal and since muscular effort plays an important part in a human’s comfort perception. However, we also acknowledge that muscular effort is probably not the only factor, and other psychological, spatial, and postural factors may also be contributing to a subject’s comfort perception. To evaluate how well a decrease in muscle effort translates into an increase in comfort perception, we conduct a psychometric questionnaire.

Building human-robot collaboration systems that are more comfortable for humans is important. In particular, a comfortable configuration can improve performance, alleviate and reduce stress, and lower incidence of musculoskeletal disorders—musculoskeletal strains are the largest cause of work-related injuries in many industrial countries [6, 30, 34]. The proposed formulation of muscle effort minimization and planning contributes to the understanding of human muscular response to daily activities, and it is a building block to developing comfortable human-robot collaboration systems.

We have previously presented part of the formulation here in a conference paper [14]. The current work, however, extends it significantly. Most importantly, here (i) we use an additional and more realistic higher-dimensional muscular model; (ii) we discuss the redundancy-resolution techniques that the higher-dimensional model requires; (iii) we use EMG, kinematic, and force data to analyze the performance of our planners during human-robot experiments (while our previous set of experiments only used questionnaires to evaluate the planners); and (iv) we conduct experiments to analyze KMEA, a key aspect of our formulation.

The rest of this article is structured as follows. Section 2 presents the related work. Section 3 formalizes the overall planning problem we aim to solve. Sections 4, 5, and 6 detail the key building blocks of the system as explained earlier: the muscle effort prediction, human configuration prediction, and robot stability, respectively. Section 7 presents our simulation experiments, and Section 8 presents our human subject experiments. Section 9 further analyzes the human experiment results in terms of muscle selection and discusses the contribution of individual muscles to the task. Section 10 concludes the article and discusses future work.

2 Related Work

Designing actions around physical human-robot collaboration (pHRC) is a complex procedure that involves planning and control over the tightly coupled constrained dynamical system [2, 16, 23, 56]. Among the many challenges pHRC poses, this work mainly focuses in two areas that have often been studied separately: human motion synthesis and prediction and bio-informed pHRC [1, 25, 31, 44].

Human motion synthesis has long been a topic of investigation in physiology, anatomy, biomechanics, and neuroscience. In recent years, it also became a topic of robotics research that produced novel insights into human motion mechanisms [22, 32]. In particular, with improved simulation tools [18, 21] and musculoskeletal models [45, 46, 51], task-space-oriented approaches extended to muscle-space have successfully leveraged robotics techniques to movement sequence reconstruction [27], tracking and/or performance assessment/filtering [7, 27], and instantaneous task- to muscle-nullspace assessment [49, 50]. These methods nonetheless fail to cover sensory-motor planning and long-term prediction (e.g., how to predict reaching and forceful interaction prior to the start of the motion). This work fills this gap and assesses the performance of the planning prediction during fHRC, which represents a largely unexplored problem that falls both into the categories of human posture prediction and pHRC. Bio-informed pHRC is a recent topic in robotics that integrates biomechanics methods to improve human-robot interaction through a better understanding of the influence of forces over the musculoskeletal system (e.g., [37, 45, 51]). In our previous work [14], we devised an optimization-based planning strategy taking the muscle activation estimation from Tanaka et al. [51], and peripersonal safety from Roncone et al. [43]. From a more comprehensive musculoskeletal model described in Saul et al. [45], Peternel et al. [40] focused on fatigue estimation—extending techniques from biomechanics literature [4] to the fHRC framework. More recently, this work was extended in Peternel et al. [41] to allow control of the task force direction (i.e., the relative pose attitude). The robot-tool attitude varied accordingly to muscle activity based fatigue estimation w.r.t. a set of predefined task-dependent muscle groups. The robot pose was however planned by assessing an ergonomics cost function based on the overloading joint-torque profile from center of pressure and ground reaction forces (e.g., [29, 42]). Similar optimization-based planning strategies are also found in other works [28, 30].

However, our work presents an optimization-based planning strategy fully based on the 6-DoF wrench mapping to muscle activation. In this sense, this work is similar to our previous efforts [14]. Yet, herein we unfold a more general comfort paradigm—with different musculoskeletal models. We also take a thorough experimental assessment addressing different comfort metrics and biosignals from EMGs. Furthermore, another novelty presented is the experimental assessment of KMEA that underlies our biomechanics-based results during fHRC tasks and defines the redundancy resolution for most pHRC literature. To the best of our knowledge, there are no other studies capable of both predicting and planning for human comfort, especially if we take forceful interaction into account. This is also the most complete pHRC work in terms of assessing muscle activity with nine simulatenous biofeedback readings from EMGs.

Other works on ergonomics and comfort for HRC. Ergonomic concepts as RULA (rapid upper limb assessment) and REBA (rapid entire body assessment) were also exploited and extended to assess human-robot interaction comfort level in other works [9, 10]. Nonetheless, although ergonomic concepts are well-posed for high-level rapid task-planning, they fail to address the impact and magnitude of larger forces and dynamic constraints in fHRC.

There has also been a considerable amount of research on comfort analysis concerning human-robot interaction for object hand-over (e.g., [3, 11, 12]), yet solutions neglect any forceful interaction and mostly rely on the geometry of interaction, physiological evaluation [20], object attitude [38], or at most learning transfer stability and corresponding wrist wrenches [13, 36].

Fig. 2.

3 Problem Formulation

To better illustrate the interaction context, consider the forceful task of drilling on an object held by the robot in a shared environment, as shown in Figure 2(C). The proposed planning strategy takes as prior knowledge the input set $(\boldsymbol {x}_{_{\mathcal {\text{h}}}},\,\boldsymbol {g}_{_{\mathcal {\text{h}}}},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}},\,m_{t},\,^{\text{o}\negthinspace }\boldsymbol {f})$ (shown in Figure 2(A)) given by the following:²

•

Human-positioning, $\boldsymbol {x}_{_{\mathcal {\text{h}}}}$, more specifically, their shoulder pose w.r.t. the robot frame $\sum _{\text{r}}$;

•

Human grasp on the tool $\boldsymbol {g}_{_{\mathcal {\text{h}}}}$, which also gives the tooltip pose $^{\text{w}\negthinspace }\boldsymbol {x}_{t}$ w.r.t. the wrist frame $\sum _{\text{w}}$;

•

The pose $^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}}$ of the tooltip w.r.t. the object frame $\sum _{\text{o}}$ (e.g., the drill point on the board);

•

The mass (and center of mass) of the tool $m_{\text{t}}$; and

•

The generalized applied force $^{\text{o}\negthinspace }\boldsymbol {f}$ at the tooltip w.r.t. to the object (e.g., drilling force).

With this input, the robot then plans its arms configuration in joint-space, $\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}}$, and the robot grippers pose, $^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}$, to grasp the object, given in the object frame $\sum _{\text{o}}$. These define the object pose and, consequently, where in the cooperative-task space the forceful interaction takes place.

Given the preceding inputs, our human-comfort-based planner aims at finding the robot configuration and grasp that minimizes muscular effort while satisfying robot kinematic and grasp stability constraints (as shown in Figure 2 B). We assume biological comfort is related to the generated forces and muscle activity during the execution of motor tasks—that is, we aim at lightening the human muscular load during interaction. More specifically, we aim to minimize the ratio of the applied muscle-tendon forces w.r.t. the maximum force capability per muscle involved in the motion/force output. More details about muscle activity are given in Section 4. Using this comfort metric, we define the comfort-based planningas the following optimization problem:

\begin{align} \boldsymbol {q}_{_{_{\mathcal {{{\bf r}}}_{}}}}^{\text{opt}},\negthinspace \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}^{\text{opt}} & =\arg \negthickspace \min _{\boldsymbol {q}_{_{_{\mathcal {{{\bf r}}}_{}}},}\negthinspace \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}}\left\lbrace \text{$\text{muscleAct}$}\left(\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}},\,^{\text{o}\negthinspace }\boldsymbol {f}\right)\right\rbrace \end{align}

(1a)

\begin{align} \negthickspace \negthickspace \negthickspace \negthickspace \negthickspace \negthickspace \negthickspace s.t.\, & \text{biomecConstraints}\left(\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}},\,^{\text{o}\negthinspace }\boldsymbol {f}\right), \end{align}

(1b)

\begin{align} & \text{graspStability}\left(\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}},\,^{\text{o}\negthinspace }\boldsymbol {f}\right), \end{align}

(1c)

\begin{align} & \text{geomConstraints}\left(\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}},\,\boldsymbol {x}_{_{\mathcal {\text{h}}}},\,^{\text{w}\negthinspace }\boldsymbol {x}_{t},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}}\right), \end{align}

(1d)

where muscleAct is a comfort metric based functional that computes the summed muscle activity of different muscles involved in a task. Details on the functional and the biomechanics constraints, biomecConstraints, mapping task-space variables to predicted muscular-activity are given in Section 4. Additionally, the graspStability summarizes the robot grasp stability conditions (described in Section 6) for the given force operation and robot configuration, whereas the geomConstraints is described next.

geomConstraints: checks geometric and kinematic constraints ensuring the tightly coupled human-robot configuration is within human and the robot joint-limits range $(\mathbb {Q}_{\text{h,lim}},\mathbb {Q}_{\text{$\boldsymbol {r}$,lim}})$. Furthermore, assuming quasi-static human-robot interaction, the geomConstraints defines where the interaction will take place by means of the tooltip pose $\boldsymbol {x}_{\text{t}}$,

\begin{align} \boldsymbol {x}_{_{\mathcal {\text{h}}}}\odot \,^{\text{h}\negthinspace }\boldsymbol {x}_{\text{w}}\odot \,^{\text{w}\negthinspace }\boldsymbol {x}_{t} & =\boldsymbol {x}_{\text{t}}, \end{align}

(2a)

\begin{align} \boldsymbol {x}_{\text{rg}}\odot \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}^{-1}\odot \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}} & =\boldsymbol {x}_{\text{t}}, \end{align}

(2b)

where $\odot$ is the rigid-body group operation, $\boldsymbol {x}_{\text{rg}}\triangleq \text{RFKM}(\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}})$ are the robot end-effectors (grippers) pose based on the robot forward kinematics model (RFKM), $^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}^{-1}$ is the pose of the object w.r.t. to the grippers (the inverse transformation of $^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}$), and $^{\text{h}\negthinspace }\boldsymbol {x}_{\text{w}}$ is the human wrist pose w.r.t. to human arm base (shoulder) given by the human forward kinematics model (HFKM):

\begin{align} ^{\text{h}\negthinspace }\boldsymbol {x}_{\text{w}} & \triangleq \text{{HFKM}}(\boldsymbol {q}_{_{\mathcal {\text{h}}}}). \end{align}

(3)

From (2) and (3), the robot can tailor human response to complete the task with the least amount of force and muscle activity (1a) based on human biomechanics models (1b) while satisfying robot kinematic and stability constraints.

Nonetheless, there is often an infinite number of arm configurations that satisfy (3). In this manner, the robot through (2) in fact shapes the set of all feasible human arm configurations $\mathcal {Q}_{_{\mathcal {\text{h}}}}(\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}},\,\boldsymbol {x}_{_{\mathcal {\text{h}}}},\,^{\text{w}\negthinspace }\boldsymbol {x}_{t},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}})$. This reflects human flexibility (redundancy) to choose different configurations given kinematics constraints and raises the question of how the human will choose their arm configuration based on (2a) and (3). To solve this redundancy in a consistent manner, in the rest of this work, we assume the existence of a unique mapping function that predicts the human arm configuration $\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}}\in \mathcal {Q}_{_{\mathcal {\text{h}}}}$ satisfying (2) and (3)—throughout this work, we will always use $\hat{}$ to distinguish predicted from real variables. Further details on this redundancy resolution strategy are given in Section 5.

4 Predicting the Muscle Effort: Task- to Muscle-space Mapping

This section provides model descriptions for the human kinematics and biomechanics, as well as different methodologies for predicting muscular activity, human comfort, and arm configuration. These concepts shape the backbone of this work and are crucial for the automatic reasoning and planning over human biological motor properties. In Section 4.1, we begin by defining the comfort metric used in $\text{\emph {muscleAct}}$ in Equation (1a). We then describe the transformation that takes human task-space force and torques (see Figure 2(A)) to human joint-space torques in Section 4.2. The torque values are thereafter mapped to muscle-space actuation by means of two fundamentally different paradigms in Sections 4.3 and 4.4.

To keep the presentation simple, in this section, we assume we know the human arm configuration at joint level, $\boldsymbol {q}_{_{\mathcal {\text{h}}}}$, that the human will take during task execution. Nonetheless, as stressed previously, arm configuration can only be predicted by our planner. Details of this prediction will be explained later in Section 5.

4.1 Muscular Effort

To minimize the summed activation of all muscles involved with the task-space force application, we need to assess human effort from muscle-space variables. Similarly to other authors [14, 27, 51], we adopt the concept of muscle activity effort ($\left\Vert \hat{\boldsymbol {\alpha }}\right\Vert _{\mathbb {M}^{m}}$) and explicitly define the comfort metric functional in (1) as

\begin{equation} \left\Vert \hat{\boldsymbol {\alpha }}\right\Vert _{\mathbb {M}^{m}}^{2}\triangleq \underset{\quad \quad \boldsymbol {\alpha }\in \mathbb {M}^{m}}{{\text{muscleAct}}}\, {=}\, \boldsymbol {\alpha }^{T}\boldsymbol {\alpha }, \end{equation}

(4)

where $\boldsymbol {\alpha }$ is the vector of activation level of the $m-$muscles involved in the force operation. The activation denotes the muscle-tendon force normalized by the maximum force a muscle can generate—that is, $\alpha _{i}\in [0,1]$, $i\, {=}\, \lbrace 1,\dots ,m\rbrace$. Hence,

\begin{equation} \boldsymbol {\alpha }\in \mathbb {M}^{m},\text{ with }\mathbb {M}\, {=}\, \lbrace \alpha \in \mathbb {R}\,|\,0\, {\le }\, \alpha \, {\le }\, 1\rbrace . \end{equation}

(5)

We predict muscle activation and forces indirectly, first estimating the related joint variables (i.e., human joint torques) and then mapping the joint torques to muscle activation. In Section 4.2, we present the mapping from task- to joint-space wrenches. One additional challenge is that there are many more DoFs in a musculoskeletal system than in joint-space (i.e., redundancy) [19], which leads to an indeterminacy w.r.t. muscle activation and forces. To solve this redundancy, we estimate $\hat{\boldsymbol {\alpha }}$ from biomechanics models (biomecConstraints), which often relies on either musculoskeletal-model-based optimization strategies or model-reduction techniques based on simulation or human-based experiments [40, 51]. In Sections 4.3 and 4.4, we present these two different estimation paradigms for the biomecConstraints, which shapes different comfort criteria with (1).

4.2 Task-to Joint-Space Modeling for the Human Arm

Fig. 3.

To reason over human biomechanics, we begin by recalling that similar to a robot manipulator, human limbs can be represented by kinematic chains. Indeed, the upper limb is often modeled³ as a 7-DoF serial chain system with the shoulder defined as the base frame, as shown in Figure 3. From the human kinematics and the geometric constraints, the pose of the tooltip $\boldsymbol {x}_{\text{t}}$ can be computed as in (2a). Consequently, generalized velocities and wrenches exerted at this tooltip can be locally mapped to the joint-space manifold, at configuration $\boldsymbol {q}_{_{\mathcal {\text{h}}}}$, through the geometric Jacobian $\boldsymbol {J}_{_{\mathcal {\text{h}}}}$—that is,

\begin{equation} \boldsymbol {\tau }_{_{\mathcal {\text{h}}}}=\boldsymbol {J}^{T}_{_{\mathcal {\text{h}}}}(\boldsymbol {f}+\boldsymbol {\rho }_{\text{t}})+\boldsymbol {\tau }_{\rho }, \end{equation}

(6)

where $\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}$ is the required joint-space torque and $\boldsymbol {f}$ is the exerted wrench represented in world frame—note that, for the sake of clarity, we are omitting arguments whenever possible. The wrench $\boldsymbol {f}$ is computed by means of norm-preserving object to tooltip and tool to world-frame transformations (i.e., $\negthinspace \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}}$ and $\boldsymbol {x}_{\text{t}}(\boldsymbol {q}_{_{\mathcal {\text{r}}}},\negthinspace \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}})$) applied to the known wrench at the tooltip in object-frame $\negthinspace \,^{\text{o}\negthinspace }\boldsymbol {f}$. In (6), we can also account for joint-torque contributions from the arm and tool weights. The gravitational torques from the arm are given by $\boldsymbol {\tau }_{\rho }(\boldsymbol {q}_{_{\mathcal {\text{h}}}})$, whereas $\boldsymbol {\rho }_{\text{t}}(\negthinspace \,^{\text{h}\negthinspace }\boldsymbol {x}_{\text{w}},m_{\text{t}})$ denotes the wrenches at end-pose stemming from the tool weight.⁴

4.3 Muscular Activity Prediction Based on Static Optimization ($\hat{\boldsymbol {\alpha }}_{_{\text{SO}}}$)

To address the muscle redundancy and estimate the muscle activation level $\hat{\boldsymbol {\alpha }}(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})$, biomechanics solutions often rely on the observation that humans resolve this redundancy in a relatively consistent [33] and optimal manner [5, 50]. Optimality, in this case, refers to minimizing energy expenditure by means of the motor control system optimization of the constrained norm of the muscular activity (5).

Existing muscle-effort optimization techniques are usually classified into static or dynamic—w.r.t. muscle activation and contraction dynamics. Static optimization neglects the nonlinear activation/contraction muscle dynamics which in turn eliminates the time-dependency from optimization and yields a bounded least-square problem [39, 53]. Compared to dynamic methods, they are less complex and time-consuming. Among classes of dynamic optimizers, direct shooting [19] and computed muscle control [52] are the most widely used strategies that concern neural excitation and contraction dynamics while integrating muscle activation. Nonetheless, despite requiring more elaborate solvers, dynamic methods have only shown improvements in very narrow cases (e.g., long-term activity or fatigue estimation [4]), which are not the case in this work. We deploy the static optimization strategy since we are focused on highly forceful but short interactions. In this sense, joint torques are defined by muscle-tendon force and the moment-arm connecting the joint to the antagonistic pair of muscles. The optimization then focuses in providing the muscle activation vector that yields least muscular activity effort satisfying biomechanics constraints—that is, biomecConstraints:

\begin{align} \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}=\arg \min _{\boldsymbol {\alpha }\in \mathbb {M}^{m}} & \lbrace \boldsymbol {\alpha }^{T}\boldsymbol {\alpha }\rbrace \end{align}

(7a)

\begin{align} s.t.\, & \boldsymbol {\varGamma }_{\text{h}}(\boldsymbol {q}_{_{\mathcal {\text{h}}}})\boldsymbol {F}_{_{\mathcal {\text{h}}}}(\boldsymbol {q}_{_{\mathcal {\text{h}}}})\boldsymbol {\alpha }=\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}, \end{align}

(7b)

where $\hat{\boldsymbol {\alpha }}_{_{\text{SO}}}$, obtained from static optimization, gives the predicted muscle activity and provides an equality constraint to (1b). The diagonal matrix $\boldsymbol {F}_{_{\mathcal {\text{h}}}}=\textrm {diag}({f_{1}^{\text{max}},\dots ,f_{m}^{\text{max}}})\in \mathbb {R}^{m\times m}$ depicts the maximum isometric muscle forces, and $\boldsymbol {\varGamma }_{\text{h}}\in \mathbb {R}^{7\times m}$ is a matrix transformation that maps muscle forces to joint torques. $\boldsymbol {\varGamma }_{\text{h}}$ contains $\gamma _{ij}\in \mathbb {R}$ elements representing the moments-arms between the i-th muscle and the j-th joint. Hence, each $i$-th muscle contributes to the torque generated at the $j$-th joint by means of the muscle force generated (activation level $\alpha _{i}$ times its maximum isometric force $f_{i}^{\text{max}}$) and its moment-arm about the joint $\gamma _{ij}$—that is, $\tau _{j}=\sum _{i}^{m}\gamma _{ij}(f_{i}^{\text{max}}\alpha _{i})$.

The matrices $\boldsymbol {F}_{_{\mathcal {\text{h}}}}$ and $\boldsymbol {\varGamma }_{\text{h}}$ depend on the human joint configuration $\boldsymbol {q}_{_{\mathcal {\text{h}}}}$ (we explain how to predict $\boldsymbol {q}_{_{\mathcal {\text{h}}}}$ in Section 5) and the corresponding musculoskeletal model. For the model, we take parameters from the seminal work of Saul et al. [45], through the OpenSim simulator [21], where $m=50$.

Remark 1.

The optimization (7) as well as the resulting $\boldsymbol {\alpha }_{_{\text{SO}}}$ defining the biomecConstraintsin (1b) satisfies the muscle-activation set definition in (5) (i.e., $\boldsymbol {\alpha }\in \mathbb {M}^{m}$). This positive constraint makes it impossible to take an unconstrained optimization directly over the induced norm (as $\mathbb {M}^{m}$ is not a vector space) nor to device standard performance characterization metrics as in other works [27, 51]:

\begin{align*} E & =\hat{\boldsymbol {\alpha }}^{T}\hat{\boldsymbol {\alpha }}=\boldsymbol {\tau }^{T}_{_{\mathcal {\text{h}}}}\left(\boldsymbol {\varGamma }_{\text{h}}\boldsymbol {F}_{\text{h}}^{2}\boldsymbol {\varGamma }_{\text{h}}^{T}\right)^{\negthinspace {-}1}\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}\\ & =\big (\boldsymbol {J}^{T}_{_{\mathcal {\text{h}}}}\negthinspace (\boldsymbol {f}{+}\boldsymbol {\rho }_{\text{t}})+\boldsymbol {\tau }_{\rho }\big)^{T}\negthickspace (\boldsymbol {\varGamma }_{\text{h}}\boldsymbol {F}_{\text{h}}^{2}\boldsymbol {\varGamma }_{\text{h}}^{T})^{\negthinspace {-}1} \big (\boldsymbol {J}^{T}_{_{\mathcal {\text{h}}}}\negthinspace (\boldsymbol {f}{+}\boldsymbol {\rho }_{\text{t}})+\boldsymbol {\tau }_{\rho }\big). \end{align*}

The preceding solution is ill-conditioned as it neglects $\boldsymbol {\alpha }$ positiveness—that is, it assumes the possibility of negative activation of muscles, which is physiologically unfeasible.

4.4 Muscular Activity Prediction Based on Model Reduction ($\text{$ \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}$}$)

In addition to musculoskeletal optimization techniques, an alternative approach proposed by Tanaka et al. [51] relies on directly measuring human upper-limb joint torques resulting from task-space forces. In other words, it experimentally evaluates the transformation between end-point force capabilities w.r.t. joint torques bypassing the muscle- to joint-torques transformation. To reduce the dimensionality of the original problem, Tanaka et al. [51] constrained the study on the most relevant agonist muscles per joint motion dynamics. Recording muscle activation and joint torques, generated by both uniarticular movements (positive and negative motions) at different joint-vector ($\boldsymbol {q}_{_{\mathcal {\text{h}}}}$) configurations, the authors devised a method to compute the activation level of the agonist (most relevant) muscles given the joint posture, direction of motion (e.g., flexion or extension), and torque. Hence, for a given $\boldsymbol {q}_{_{\mathcal {\text{h}}}}$, a unique muscular activation is associated for each joint torque (given its uniarticular direction). In this sense, the redundancy concerning muscle and joint torques is removed.

With this mapping, for a given $\boldsymbol {q}_{_{\mathcal {\text{h}}}}$ and joint direction, we can compute the maximum possible torque at the $i$-th joint, $\tau _{i}^{max}$ (based on maximum absolute values under maximum voluntary contraction). This leads to a bijective mapping based on the matrix transformation $\boldsymbol {\varPi }(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})=\textrm {diag}({\tau _{1}^{max}(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}),\dots ,\tau _{7}^{max}(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})})$—a diagonal matrix composed of human upper-limb maximum torque elements—that can be used to compute the muscle activation as follows

biomecConstraints ($\boldsymbol {\alpha }_{_{\text{MR}}}$):

\begin{equation} \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}=\boldsymbol {\varPi }^{-1}(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}, \end{equation}

(8)

where $\hat{\boldsymbol {\alpha }}_{_{\text{MR}}}=[\alpha _{1},\dots ,\alpha _{7}]^{T}\in \mathbb {M}^{7}$ provides the predicted muscle activation and an equality constraint to (1b).

Note however that $\boldsymbol {\varPi }(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})$ has a nonlinear dependency to the joints positions and torques, particularly torque direction. This is because each functional $\tau _{i}^{max}(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})$ approximates the muscle dynamics linearized at given joint posture and expected motion/torque direction. That is to say, the maximum torque for each joint depends not only on its position but also on its motion and torque direction (in particular, the torque that implicitly defines extension/flexion of antagonistic muscles). Hence, $\hat{\boldsymbol {\alpha }}^{T}_{_{\mathcal {\text{h}}}}\hat{\boldsymbol {\alpha }}_{_{\mathcal {\text{h}}}}$ is non-symmetrical w.r.t. $\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}$.⁵

Equipped with joint configurations and sign/directions of the torques, we can build the matrix of maximum torque elements $\boldsymbol {\varPi }(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})$ and compute the muscular activation (8). The main advantage of the solution (8) compared to (7) is the reduced computational complexity obtained from removing the redundancy.

Other Methods for Muscular Activity Estimation. In addition to optimization and model-reduction-based strategies, it is worth mentioning the work of Peternel et al. [40], which similarly to (8) also presents an efficient and fast solution to output muscular activity estimation based on the user configuration and external forces. In contrast to previous methods, the work in Peternel et al. [40] addresses the problem from a machine learning point of view by means of a GPR-based learning algorithm combined with a simulated dataset of arm joint configurations, end-point forces, and resulting muscular forces from the musculoskeletal model of Saul et al. [45] and the OpenSim simulator [21]. In this context, the resulting metric provides approximate but faster results compared to (7) and could be straightforwardly used with our comfort-based planning without loss of generality. Notice that, if well-trained, any similar strategy will lead to the same results as in (7).

5 Predicting the Human Kinematic Configuration: the KMEA

For a given forceful task, our goal is to plan robot joints and grasp configurations, and an object pose that optimizes human comfort according to muscleAct in (4). However, as stressed in the problem definition in Section 3, the robot can only shape the set of all feasible human configurations $\mathcal {Q}_{_{\mathcal {\text{h}}}}(\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}},\,\boldsymbol {x}_{_{\mathcal {\text{h}}}},\,^{\text{w}\negthinspace }\boldsymbol {x}_{t},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}})$, which in turn raises the question of how the human will choose their configuration and if that will satisfy our comfort metric.

In biomechanics literature (e.g., [5, 26, 27, 52]), it is often assumed that humans resolve this redundancy consistently in the most efficient manner in terms of minimizing the muscle activation required to transmit a given task-space force. We call this KMEA and adopt this assumption in this work.

In other words, KMEA relies on a human’s innate self-motion optimization scheme in the integrated joint-muscle nullspace—that is, the nullspace projection from task-space to muscle-space. The self-motion optimization can therefore be described by

\begin{equation} \begin{aligned}\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}}= & \arg \min _{\boldsymbol {q}_{_{\mathcal {\text{h}}}}\in \mathbb {Q}_{\text{h,lim}}}\left\lbrace \hat{\boldsymbol {\alpha }}^{T}\left(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}(\boldsymbol {q}_{_{\mathcal {\text{h}}}})\right)\hat{\boldsymbol {\alpha }}\left(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}(\boldsymbol {q}_{_{\mathcal {\text{h}}}})\right)\right\rbrace \\ & \begin{aligned}s.t.\quad & \hat{\boldsymbol {\alpha }}\leftarrow \text{biomecConstraints}(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\,^{\text{o}\negthinspace }\boldsymbol {f}),\\ & \boldsymbol {q}_{_{\mathcal {\text{h}}}}\in \mathcal {Q}_{_{\mathcal {\text{h}}}}\left(\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}},\,\boldsymbol {x}_{_{\mathcal {\text{h}}}},\,^{\text{w}\negthinspace }\boldsymbol {x}_{t},\,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}}\right),\\ & \hat{\boldsymbol {\alpha }}\in \mathbb {M}^{m},\end{aligned}\end{aligned} \end{equation}

(9)

where biomecConstraints refers to the equality constraint defined either by (7), for static optimization solutions (i.e., $\hat{\boldsymbol {\alpha }}_{_{\text{SO}}}$), or by (8), for model-reduction approaches (i.e., $\hat{\boldsymbol {\alpha }}_{_{\text{MR}}}$). The constraint $\boldsymbol {\alpha }\in \mathbb {M}^{m}$ also ensures the optimal result is well-posed. The set $\mathcal {Q}_{_{\mathcal {\text{h}}}}$ is defined over (2a) and $\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}$ is obtained in (6) based on the joint configuration.

To better understand KMEA, take for example a task of cutting off a circular piece of a board, shown in Figure 4. The board configuration is pre-set, and the circle is divided into 16 discrete (also) pre-set points and forces (15 N along the cutting direction). We computed feasible IK solutions for such task satisfying the constraints in (9) with (7b). The dark blue curve depicts the optimization following (9) (i.e., when a human is at the most comfortable configuration (KMEA)), whereas the light blue depicts the worst (most uncomfortable) configuration (i.e., the $\arg \max$ from (9) that yields the highest muscular activity). For two possible configurations (Figure 4(A) and (B)), our experience is that configuration [B] is much more comfortable than [A], which coincides with the comfort metric assessment and experiments in Section 8. Moreover, at the proximity of joint limits (position $f_{11}$ has no feasible IK solution), we observe the soaring muscular activity cost (degradation in comfort) as expected. In addition, note the consistency of the metric at the circular closing (positions $f_{16}$ to $f_{1}$). Figure 4 also shows the importance of planning and control strategies in shaping the human feasible response to comfortable configurations.

Fig. 4.

6 Ensuring Grasp and Robot Stability

From the robot’s point of view, quasi-static forceful interaction imposes constraints over reachability in terms of coupled kinematics and stability in terms of static equilibrium. The former ensures the existence of a joint configuration $\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}}$ satisfying (2) within the joint limits—that is, $\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}}\in \mathbb {Q}_{\text{$\boldsymbol {r}$,lim}}$. However, the grasp stability constraint describes the relationship between the force applied on the object and its counterpart at joint-space w.r.t. the robot capability to resist it. Given our assumption of a bimanual cooperative robot, $\boldsymbol {q}_{_{\mathcal {{{\bf r}}}_{}}}=[\boldsymbol {q}^{T}_{_{\mathcal {\text{r}}}}[1]\ \ \boldsymbol {q}^{T}_{_{\mathcal {\text{r}}}}[2]]^{T}\in \mathbb {R}^{n}$, where $\boldsymbol {q}_{_{\mathcal {\text{r}}}}[1],\boldsymbol {q}_{_{\mathcal {\text{r}}}}[2]$ refer to the left and right arm configuration, the grasp stability can be computed from the symmetric formulation from Uchiyama and Dauchez [54]. This formulation, however, leaves the forces at the grip contacts unconstrained. Herein, we are also interested in checking the stability against insufficient forces at the gripper contacts (e.g., if the frictional forces between fingers can prevent the object from sliding) and deformation due to contact compliance. Imposing this additional constraint onto the symmetric formulation [54], the robot stability against human exerted forces is checked by

graspStability:

\begin{equation} \begin{array}{l} \boldsymbol {W}_{_{\mathcal {{{\bf r}}}_{}}}\boldsymbol {f}_{_{\mathcal {{{\bf r}}}_{}}}=-\boldsymbol {f},\\ \boldsymbol {J}^{T}_{_{\mathcal {{{\bf r}}}_{}}}\boldsymbol {f}_{_{\mathcal {{{\bf r}}}_{}}}=\boldsymbol {\tau }_{_{\mathcal {{{\bf r}}}_{}}},\\ |\boldsymbol {\tau }{_{_{\mathcal {\text{r}}}}}|\le \boldsymbol {\tau }_{_{\mathcal {\text{r}}}}^{\max },\\ |\boldsymbol {f}_{r}|\le \boldsymbol {f}_{r}^{max},\end{array} \end{equation}

(10)

where $\boldsymbol {f}_{_{\mathcal {{{\bf r}}}_{}}}\, {=}\, {[\boldsymbol {f}^{T}_{_{\mathcal {\text{r}}}}[1],\boldsymbol {f}^{T}_{_{\mathcal {\text{r}}}}[2]]^{T}}\in \mathbb {R}^{12}$ is wrench vector at each robot gripper, and $\boldsymbol {W}_{_{\mathcal {\text{r}}}}\in \mathbb {R}^{6\times 12}$ is the grasp matrix that maps the wrenches at the grippers to the reactive resultant task-space force and torques on the object—that is, $\boldsymbol {f}$—this refers to the human applied force in the world frame, which depends on the input force task $^{\text{o}\negthinspace }\boldsymbol {f}$ and the planned object pose $\boldsymbol {x}_{\text{o}}$.

Similarly, $\boldsymbol {J}_{_{\mathcal {{{\bf r}}}_{}}}\in \mathbb {R}^{12\times n}$ is a block diagonal matrix with both arms’ geometric Jacobians used to map the wrenches at robot grippers into the joint-space manifold yielding the augmented joint-torque value $\boldsymbol {\tau }_{_{\mathcal {{{\bf r}}}_{}}}\, {=}\, [\boldsymbol {\tau }^{T}_{_{\mathcal {\text{r}}}}[1],\boldsymbol {\tau }_{_{\mathcal {\text{r}}}}[2]^{T}]^{T}\in \mathbb {R}^{n}$. The robot cooperative stability is thus secured if the required joint torque, $\boldsymbol {\tau }_{_{\mathcal {{{\bf r}}}_{}}},$ and wrench at gripper point, $\boldsymbol {f}_{r}$, are respectively within the torque limits $\boldsymbol {\tau }_{_{\mathcal {\text{r}}}}^{\max }\in \mathbb {R}^{n}$ and gripper force limits $\boldsymbol {f}_{r}^{max}=[\boldsymbol {f}_{r_{1}}^{max},\boldsymbol {f}_{r_{2}}^{max}]^{T}\in \mathbb {R}^{12}$. We skip further details on the mathematical formulation as they lie beyond the scope of this work, yet they can be found in our previous work [16].

7 Planning: Quantitative Analysis

This section explores quantitative aspects of the proposed comfort-based planning strategies under different scenarios (exerted force ($^{\text{o}\negthinspace }\boldsymbol {f}$), human-robot positioning ($\boldsymbol {x}_{_{\mathcal {\text{h}}}}$), tool transformation ($^{\text{w}\negthinspace }\boldsymbol {x}_{t}$), weight ($m_{\text{t}}$) and tooltip position on the object, $^{\text{o}\negthinspace }\boldsymbol {x}_{\text{t}}$). In all scenarios, the simulated Baxter robot needs to plan to grasp and to position an object in the human’s workspace to perform a fHRC task, as shown in Figure 2. Planning decisions were made according to the following planners:

•

Random-planner: Given a forceful operation, the robot randomly picks an object pose within the reachable human-robot space (2) and only checks for static stability (10) and cooperative geometric constraints (2)

. This planner acts as a baseline.

•

SO-Comf-Planner: Our comfort-based planner from (1) with muscular activity effort ($\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$) computed from static optimization (7).

•

MR-Comf-Planner: Similarly, this planner optimizes (1) but with muscular activity effort ($\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert$) computed from the model-reduction strategy (8).

In addition to the preceding planners, we also defined a fixed-pose configuration in between human and robot (more specifically, $40\,$cm in front of the shoulder)—and the robot plans grasps such that both geometric and stability constraints are satisfied for all scenarios.

The scenarios included the human performing $\text{puncturing}$, pulling, and cutting operations with the aid of the robot. The force directions for $\text{puncturing}$ and pulling were taken normal to the board in opposite directions, whereas for the cutting task, it was aligned with the cutting direction. For each forceful task, we also designed different simulations based on varying the force magnitude ($^{\text{o}\negthinspace }\boldsymbol {f}$), human-robot positioning ($\boldsymbol {x}_{_{\mathcal {\text{h}}}}$), and tool weight ($m_{\text{t}}$) and length (which gives way different poses of the tooltip w.r.t. the wrist frame (i.e., $^{\text{w}\negthinspace }\boldsymbol {x}_{t}$)). To obtain a realistic representation of the required forces during the planning stage, before the experiments, we estimated force magnitudes using a force-torque sensor. For quantitative purposes, planning algorithms were executed in Matlab using multistart SQP-solver and scatter-search-based surrogate [55] for global optimization. For the muscular effort assessment with $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$ and computing the biomechanics constraints (7), we also exported musculoskeletal model parameters from Saul et al. [45] using the OpenSim simulator [21].

7.1 Optimization Performance

This section illustrates the comfort gains, in terms of reduced muscle activity, from both comfort-based planners in contrast to random and fixed-pose configurations. To this aim, we devised 135 different scenarios for each planner—involving 45 $\text{puncturing}$, 45 cutting, and 45 pulling tasks with

•

Three different force magnitudes $\left\lbrace 7.5,15,30\right\rbrace$$\,$N;

•

Three human-robot positioning ($\boldsymbol {x}_{_{\mathcal {\text{h}}}}$): ${\left\lbrace [0,1.2],\,[0.2,1.4],\,[-0.2,1.0]\right\rbrace }\,$m along the floor plane with constant height and orientation (human-robot facing each other);

•

Five tools (weight (kg), length (m) $\in \left\lbrace [.35,.1],\,[.1,.1],\,[.7,.1],\,[.35,.5],\,[.35,.15]\right\rbrace$).

Table 1 shows the statistics of the three planners and the fixed-pose results for the $\text{puncturing}$, pulling, and cutting forceful tasks. For each output, we computed the muscular activity effort from $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$ and $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert$ using KMEA based on the corresponding muscle activity prediction criteria.

Table 1.

	Musc. Act. Effort ($\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert$)			Musc. Act. Effort ($\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$)
	RndP	Fixed-Pose	MR-Comf-Planner	RndP	Fixed-Pose	SO-Comf-Planner
Puncturing$\phantom{\int }$	1	$.920\pm .310$	$\boldsymbol {.606\pm .342}$	1	$1.279\pm .517$	$\boldsymbol {.341\pm .145}$
Pulling	1	$1.189\pm .445$	$\boldsymbol {.371\pm .141}$	1	$1.073\pm .336$	$\boldsymbol {.262\pm .087}$
Cutting	1	$1.164\pm .434$	$\boldsymbol {.817\pm .205}$	1	$2.340\pm .953$	$\boldsymbol {.610\pm .193}$

Table 1. Average Muscular Activity Effort (the Lower the Better) Normalized with Random-Planner (RndP) Results for 45 Puncturing, 45 Pulling, and 45 Cutting Operations

For the first part of the table (first three columns), we computed the human kinematics that minimize (9) according to $\hat{\boldsymbol {\alpha }}_{_{\text{MR}}}$ (i.e., biomecConstraints defined in (8)) and compute the corresponding muscular activity effort according to $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert$. The same steps are taken for the second part (last three columns)—that is, we computed KMEA and muscular activity effort according to $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$. To improve readability, results from the Random-Planner were used as baseline to normalize the results of other planners.

As expected, SO-Comf-Planner outperforms others for comfort-metric based on $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$, whereas MR-Comf-Planner yields superior results when using $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert$. In a practical application, this implies that, for $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$, the Random-Planner and fixed-pose configuration compared to our comfort-planner respectively demand in average up to $2.9\times$ and $3.8\times$ more muscular activity for $\text{puncturing}$, $3.8\times$ and $4.1\times$ more for pulling, and $1.6\times$ and $3.8\times$ for cutting tasks. This corroborates with our argument and validates the performance both comfort-based planners. Another point to highlight is that the improvements from the SO-Comf-Planner are more noticeable w.r.t. to other planners.

On the other end, the fixed position often yields worst results with large uncertainty—reflected in the large standard deviation. This is due to the fact that a simple pose, as intuitive as it is (a board positioned in front of a user), cannot cover all different interaction possibilities comfortably, which calls for an optimization method that guarantees reduced muscular requirements.

The consistency of our approach and results, particularly for the static optimization, is reflected by the lower standard deviation. One limitation is the required planning time. To compute one solution, the SO-Comf-Planner took 32.2 seconds on average, whereas the MR-Comf-Planner took 23.9 seconds.

7.2 Optimization Performance: Multiple Tasks

We also assessed planning performance to address multiple tasks producing a configuration that minimizes the worst muscular activity effort among those tasks. This is done by extending the comfort-based planner from one task in (1) to a min-max optimization-based planner—that is,

\begin{align*} \boldsymbol {q}_{_{_{\mathcal {{{\bf r}}}_{}}}}^{\text{opt}},\negthinspace \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}^{\text{opt}} & =\arg \negthickspace \min _{\boldsymbol {q}_{_{_{\mathcal {{{\bf r}}}_{}}},}\negthinspace \,^{\text{o}\negthinspace }\boldsymbol {x}_{\text{rg}}}\max _{\ell {\in }\lbrace 1, \ldots ,|\ell |\rbrace }\left\lbrace \underset{\ell }{\text{muscleAct}}\left(\hat{\boldsymbol {q}}_{\text{h,$\ell $}},\,^{\text{o}\negthinspace }\boldsymbol {f}_{\ell }\right)\right\rbrace \\ \negthickspace \negthickspace s.t.\, & \text{same constraints from}(1b)-(1d)\text{ for each task }\ell . \end{align*}

Here, we considered 15 sequential tasks (i.e., $|\ell |{=}15$) involving 5 $\text{puncturing}$ operations followed by a straight line cutting divided in 10 discrete operations, as shown in the top-left figure in Table 2.

Table 2.

Similarly to the previous analysis, both comfort-based planners produced a robot response capable of reducing the muscular activity of the human counterpart. In particular, SOComf-Planner outperforms others for the $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$ comfort metric, whereas MRComf-Planner outperforms others for $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert$. The values in grey correspond to one comfort-based planner with a different assessment metric (e.g., evaluating the SOComf-Planner with $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert$). The Random-Planner yields the worst results and demands up to $3.3\times$ more muscular activity compared to SOComf-Planner. The fixed-pose scheme, in this case, provides better results compared to the purely Random-Planner, yet it demands at least $1.7\times$ more muscle activation than our comfort-based planners. This highlights that it is possible to plan for multiple operations with human comfort in mind, ensuring no operation demands too much muscular activity. This result paves the way for integrating more elaborate motion/task planners that consider continuous and/or sequential forceful interactions (e.g., [15, 17]) while minimizing the total muscular activity required from the human co-worker.

8 Human-Robot Experiments

This section presents a series of human-robot experiments focused on validating comfort assumptions and evaluating the performance of the proposed comfort-based planning scheme during forceful cooperative motor tasks in unconstrained real-world scenarios with the Baxter robot. The experimental setup⁶ used in our study is illustrated in Figure 5.

Fig. 5.

Healthy participants (A to E, 5, 1 female) of an average age of (30.5 $\pm$ 6.5) were recruited. All participants but one were right hand dominant. Participant A was used to conduct the pilot studies, and the data is not included here. To reduce stress of interacting with the robot, participants, on arrival, were suggested to actively engage with the Baxter robot to familiarize them with it. This commonly involved the participants manipulating Baxter’s arms and exploring its gravity compensation mode. Participants were then briefed on the two fHRC tasks they would conduct:

•

Puncturing (Figures 1 and 5):

— A hard foam board (2.5 cm depth), designed for resisting puncturing actions, was prepared for each of the participants.

— Equidistant locations are marked on each board, representing puncturing targets.

— The participant begins the task by moving their dominant arm toward the target while holding the puncturing tool in a power grip.

— The participant then proceeds to puncture the target, and subsequently removing the tool from the board, lowering their arm to a neutral position close to the body.

•

Cutting (Figures 1 and 5):

◦

— A craft foam board (0.5 cm depth) is prepared for the participants to cut through.

◦

— Similar to the puncturing task, a target cut pattern is designed for each trial and board, by preparing the board with two hollow squares per trial. Each square indicates the start and finish location for the cut line. For safety, participants do not cut across the board.

◦

— With a precision grip, the participants initiate the task by raising their dominant arm towards the “Start” square of the indicated trial.

◦

— The participant then proceeds to cut the board, within the space between the hollow squares, until he/she reaches the “End” square.

◦

— The participant then removes the cutting tool from the board and returns their arm to a neutral, close to body position.

For planning purposes, we assumed a $\text{puncturing}$ force, normal into the board, with magnitude $15\,$N, and a cutting force, aligned with the cutting direction, with maximum magnitude $10\,$N.

Prior to beginning the task, the participants positioned themselves in the work space in their desired position. This position was marked on the floor, to ensure in each trial a quasi equal positioning of the participant. To mitigate any possible effects of fatigue, adequate rest periods was ensured between trials. The tasks were designed to measure different levels of comfort in three distinct conditions as defined by the robot planner strategy. Each participant completed a minimum of 42 trials in all (7 trials per condition). The three different configurations were offered in a random order to avoid any bias:

•

Comfort-Planner: Our comfort-based planner from (1) with muscular activity effort ($\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$) was computed from static optimization (7).⁷ The planned object pose and robot configuration are shown in Figure 5 (left) for the two operations.

•

Random-planner: Similar to the Random-Planner defined in Section 7. Yet, all users had the same human to board pose transformation, which was selected randomly prior to the start of the experiments. The robot planned grasps and configurations accordingly. This was done to maintain consistency between participants’ performance across the experiments. This object pose and robot configuration can be seen in Figure 5 (center) for the two operations.

•

UserDef-Pose: Similar to the fixed-pose in Section 7 but different for $\text{puncturing}$ and cutting. In addition, instead of taking a heuristic-based approach, the configurations were selected based on previous human-robot experiments where six different participants were asked to identify preferred (comfortable) positions for both operations. The userDef-Pose was built on the mean pose of the board w.r.t. the human, after outliers removal. Similar to the Random-Planner, for consistency, all users had the same human to board pose transformation. This object pose and robot configuration can be seen in Figure 5 (right) for the two operations.

In addition to the preceding operations, participants performed similar tasks at maximum voluntary contraction (MVC) before and after the experiments, which served as a baseline assessment of forces and to confirm that there were no changes in the force capability during the experiment. For the human kinematics tracking, we used the Optitrack system (at $120\,$Hz) synchronized with the force acquisition (at $1\,$KHz) from the robot and the EMG recording system (Noraxon wireless DTS sensors, model 546 with 10-Hz high-pass filter and low-pass at $1.5\,$KHz). Recorded data were further processed with Spike2 and Matlab

. Each individual muscle activity was normalized against the summed muscle activity and generated force of each trial. The resulting database was also manually scrutinized to ensure quality and experiments with data loss were removed (e.g., camera occlusion and EMG noise).

The wireless EMG sensors were anatomically placed over nine different muscles selected to include the main muscular actuators for the corresponding motor tasks across the shoulder, elbow, and wrist joints. Tables 4 and 5, both in Section 9, detail the muscles description, body functions, and relevance to the forceful tasks.

Finally, for clarity, the following definitions will be used hereafter:

•

Predicted muscle activation level refers to the estimated muscle activation level $\hat{\boldsymbol {\alpha }}\in \mathbb {M}^{m}$, for the musculoskeletal model with 50 muscles from Saul et al. [45].

•

Predicted muscle force refers to the estimated muscle force $\hat{f}_{i,m}\, {=}\, \hat{\alpha }_{i}F_{i}$—that is, the force at the i-th muscle given by the predicted activation $\hat{\alpha }_{i}$ scaled by its maximum isometric force $F_{i}$.

•

Measured muscle activity refers to the muscle activity measured from EMG recordings from muscles, which is related to the force generated by selected muscles.

8.1 Comfort Performance from EMGs Biofeedback

As expected, the different configurations—established by the Comfort-Planner, Random-Planner, and UserDef-Pose—invoked different muscle recruitment patterns from the subjects, generating varying levels of force and muscle activity for each trial and condition. In this section, our objective is to analyze if, during the preceding experiments, the comfort-planning strategy consistently reduced muscular effort when compared with the Random-Planner and the UserDef-Pose strategies.

We present the data we collected in our experiments in Figure 6 and in Table 3.

Fig. 6.

Table 3.

		Subject B	Subject C	Subject D	Subject E
Puncturing	UserDef	$495.58\pm 132.76$	$1,105.00\pm 475.87$	$1,639.67\pm 430.74$	$526.72\pm 116.17$
	Comfort	$\boldsymbol {361.67\pm 54.81}$	$\boldsymbol {813.66\pm 173.86}$	$\boldsymbol {633.52\pm 304.06}$	$\boldsymbol {345.12\pm 102.02}$
	Random	$577.52\pm 182.18$	$883.16\pm 231.48$	$1,675.83\pm 210.15$	$\boldsymbol {336.83\pm 123.44}$
Cutting	UserDef	$\boldsymbol {696.31\pm 242.30}$	$1,037.05\pm 114.04$	$\boldsymbol {1,698.91\pm 209.35}$	$942.67\pm 251.24$
	Comfort	$\boldsymbol {696.43\pm 194.50}$	$1,568.48\pm 280.71$	$1,925.61\pm 600.57$	$\boldsymbol {574.36\pm 53.29}$
	Random	$725.10\pm 124.84$	$\boldsymbol {713.24\pm 110.90}$	$1,882.23\pm 508.40$	$1,380.04\pm 100.56$

Table 3. Summed Muscle Activity of Each Subject Presented as the Mean $\pm$ Standard Deviation for Both Puncturing and Cutting Tasks

Comparison across planners is thus possible and highlights subjects’ preference to the planners. The values for Comfort-Planner for the puncturing task were consistently the lowest, highlighting its role in reducing muscle effort to complete the task. For the cutting task, such a relationship is not as clear, with cases like Subjects C and D presenting higher values for the Comfort-Planner.

Figure 6 presents all the trials by all the subjects. Each point is a trial. We plot the measured summed muscle activity and their respective task-space force values for $\text{puncturing}$ (left) and cutting (right). Each point is labeled with the human subject (B to E) performing that trial. Each point is colored according to the planner used for the trial: the Comfort-Planner (blue), Random-Planner (grey), or UserDef-Pose (red). To make visual comparison between planners easier, a $\text{condition}$ area is drawn for each planner using the boundary data points from trials of that planner, excluding outliers (i.e., those over three scaled median absolute deviations from the remaining points).

Table 3 presents the data focusing on each subject separately. For each subject (B to E), we present the mean and standard deviation of the more than seven trials under each planner, for puncturing and cutting. The lowest mean muscle activity of each subject is emphasized in bold.

In the following, we discuss the data separately for puncturing and cutting. We also perform statistical significance tests to test for our claim that the Comfort-Planner produces consistently lower muscle activity when compared to the other planners.

Puncturing. For the puncturing task, the data indicates a reduction in the muscular effort when the Comfort-Planner is used. In Figure 6, the resulting blue $\text{condition}$ area presents the lowest summed muscle activity and force values. It also presents a narrower area than the remaining conditions, as expected. The Random-Planner yields the worst results (least comfortable), whereas the highly scattered UserDef-Pose results reveal that general (one-fits-all) solutions do not sufficiently constrain the forceful operation to a comfort area. This leads to a larger variability among participants, which in turn yields both comfortable and uncomfortable executions (even for the same participant on different trials). The smaller variation in the Comfort-planner trials (when compared with the variations of the UserDef and Random-Planners) may be suggesting that while the UserDef-Pose and Random-Planner data points are sampled from a larger underlying distribution, the Comfort-Planner results are sampled from the smaller subset that requires less muscular activity. Overall, the centroid of the UserDef-Pose and Random-Planner $\text{condition}$ areas respectively yield muscular activity $69.5\%$ and $72.5\%$ larger compared to the Comfort-Planner $\text{condition}$ area, and similarly $29.8\%$ and $60.2\%$ larger forces.

Table 3 also supports our claim that the Comfort-Planner produces lower muscle activity at the subject level. The Comfort-Planner has the lowest mean muscle activity for three of the four subjects. For the fourth subject (E), although the Random-Planner mean muscle activity is lower, the Comfort-Planner value is extremely close.

To test our hypothesis, we have performed a repeated measures ANOVA to compare the effect of planner type on each subject’s performance in terms of muscle activity. Significant difference in means was assessed post hoc, and a Bonferroni-Holm correction was applied ($P_h$). This was all performed using the JASP software (JASP 0.14.1.0, University of Amsterdam). Overall, the mean measured muscle activity was different for the puncturing task (F = 18.45, p < .001, df = 2), significantly different between Comfort-planner (F = 10.07, df = 3, p < .001) and UserDef-Pose ($P_h$ < .001), and also with Random ($P_h$ < .001). The means of UserDef-Pose and Random-Planner were not ($P_h = 0.39$) significantly different. In other words, our hypothesis that the Comfort-Planner produces consistently lower muscle activity is indeed valid and statistically significant.

Cutting. Results for the cutting task are less clear. In Figure 6, different planners reveal smaller differences when looking at the overall area per condition.

The subject-specific evaluation in Table 3 is also not completely convincing, but for two of the four subjects (B and E), the Comfort-Planner produces minimal muscle activity. An exceptional case was Subject C’s response for the cutting task. Subject C data shows lowest mean muscle activity for the Random-Planner, whereas the Comfort-Planner performs worst (this result agrees with the responses of Subject C to the questionnaire, which we will present in Section 8.2). There are two possible scenarios to explain this: either the musculoskeletal model did not properly cope with the real muscle activity model from Subject C or the muscle-nullspace response was not optimized for the specific task-space. For the second possibility, it is relevant to highlight that Subject C has expressed in the questionnaire additional comments to be unfamiliar with DIY tasks. In this sense, and taking note that the cutting was a harder task to understand and perform compared to the $\text{puncturing}$, it is possible that the unfamiliarity prevented a proper optimization in the nullspace. In other words, the motor control system was not aware of which forces to expect and therefore did not respond as predicted (shown in Figure 7). From this perspective, additional executions would likely improve the overall expected response. Yet, it is worth noting that this effect could also be mitigated by taking uncertainties related to human response into consideration.

There is another important difference between the puncturing task and cutting task: the puncturing task was much shorter (both in terms of the time it takes and the distance the tool needs to travel) than the cutting task. This has negative consequences for the cutting results in two ways. First, during cutting, there were more opportunities for the subject to change body posture, or the grasp on the tool. Since our model assumes a fixed human body posture, this represented a larger deviation from our assumptions, when compared with the puncturing case. Second, the length of the cutting tasks also meant more dynamic activity of the muscles, which in general results in less reliable and more noisy readings with the EMG biosensors. These aspects of the task might have contributed to the weaker results we observe with cutting.

Statistical significance tests (including Subject C) produce inconclusive results for the cutting task. The mean of the measured muscle activity was not significantly different for the planners across subjects for cutting (F = 0.39, p = 0.68, df = 2). There was no difference in means across the three planners—Comfort-planner to UserDef-Pose ($P_h$ = 1) and with Random-Planner ($P_h$ = 1). In addition, means of UserDef-Pose and Random-Planner were not ($P_h$ = 1) significantly different. Still, we note that, if we exclude Subject C, UserDef-Pose and Random-Planner $\text{condition}$ areas in Figure 6 respectively yield muscular activity $26.6\%$ and $58.6\%$ larger compared to the Comfort-Planner $\text{condition}$ area.

Muscle-activity prediction.

We also analyzed the prediction capabilities of the comfort metric $\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }$. Figure 7 presents the predicted summed muscle forces from the musculoskeletal model [45] (i.e., the norm of the force generated by the $m=50$ muscles. The values were computed by taking the measured human kinematics—that is, $\boldsymbol {q}_{_{\mathcal {\text{h}}}}$ and measured task-space force $^{\text{o}\negthinspace }\boldsymbol {f}$ (vertical axis in Figure 7).

Fig. 7.

The predicted $\text{condition}$ areas are consistent with the measured values in Figure 6, particularly for the $\text{puncturing}$ task. The cutting task presented larger differences w.r.t. the measured data. This reflects participants’ perception of $\text{puncturing}$ being an easier to understand/implement task (see Section 8.2).

The $\text{puncturing}$ task also presented a linear behavior w.r.t. to the task-space force to predicted muscle activation and forces, as expected by combining (6) with (7b). In other words, a linear increase in task-space force should reflect a linear increase in muscular activation. Similar behavior is also observed for the cutting task based on the Comfort-Planner scenario.

8.2 Participants’ Perception of Comfort

Fig. 8.

After the experiments, for each forceful task and robot configuration, participants were asked to complete a psychometric questionnaire. Participants were asked to grade (1 to 5) the following questions (in addition to providing further comments on the experiments):

(1)

How safe have you felt during the robot interaction?

(2)

How difficult was the task for you, in terms of the kinematics (reaching for the task)?

(3)

How difficult was the task for you, in terms of the required force?

(4)

How natural/fluid was the interaction for you? (How likely were you to choose the board/robot pose?)

(5)

Overall, how comfortable was the task for you?

The scores are summarized in Figure 8. For questions 1, 4 and 5, the higher the grade, the better the user perception about the interaction, whereas questions 2 and 3 work the other way around.

From Figure 8(a) and (b), it is clear that participants agreed the $\text{puncturing}$ task was more comfortable, safer, and easier to execute (both in terms of their body posture as well as in terms of the required forces) compared with the cutting task. For the $\text{puncturing}$ task, participants’ perception of the Comfort-Planner and UserDef-Pose configurations were similar with a small improved comfort perception to our planner. Performance for the Random-Planner was considerably worse, as predicted by both the musculoskeletal model and the measured muscle activity shown in Figures 6 and 7.

For the cutting task, participants expressed the robot behavior w.r.t. the Comfort-Planner to be more comfortable and safe yet less intuitive than the UserDef-Pose. Participants graded the UserDef-Pose response to be the less difficult (w.r.t. the kinematics), which corroborates with participants’ general comments on intuitiveness. Curiously, participants also graded the UserDef-Pose to be less force demanding, which contradicts the readings from task-space force sensors and EMG data, shown in Figure 6.

An exceptional case was Subject C’ s responses for the cutting task, as shown in Figure 8(c). The participant expressed the Random-Planner resulting configuration as the best both in terms of safety and comfort, as well as in terms of being less difficult, whereas the Comfort-Planner was considered the worst in all questions. The results for this participant differed completely from the other participants in this task, yet the participant’s consideration was also captured from the EMG response as discussed previously.

All participants reported that the cutting task was more difficult to execute than the $\text{puncturing}$— which agrees with the general response in Figure 8. Subjects C and D also commented they needed to increase their force during task as the task was not as fluid. Subject C has additionally expressed to have never done DIY tasks. Subject D was the only participant to have performed human-robot interaction experiments prior to this work, yet all participants reported to have enjoyed the experience and have found the fHRC to be safe.

8.3 Quantitative Analysis of KMEA

Fig. 9.

During fHRC experiments, participants were free to perform the tasks in the way they felt more comfortable, which in turn implied slightly different human kinematic configurations compared to the ones predicted and therefore different musculoskeletal response.

To quantitatively analyze KMEA, for each trial, we computed the KMEA kinematics and corresponding predicted muscle activity effort and compared the outputs with participants’ results. The difference between predicted and real data is shown in Figure 9. The y-axis shows the kinematics error norm in terms of joint values from predicted KMEA ($\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}}$) and real measured joints ($\boldsymbol {q}_{_{\mathcal {\text{h}}}}$)—that is, $\vert \vert {\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}}-\boldsymbol {q}_{_{\mathcal {\text{h}}}}}\vert \vert$. The x-axis depicts the predicted muscle activity effort from all trials $\hat{\boldsymbol {\alpha }}(\boldsymbol {q}_{_{\mathcal {\text{h}}}})$ (computed from (7)) compared to the values associated with the KMEA configuration—that is, the muscle activity prediction error norm due to the KMEA, $\vert \vert {\hat{\boldsymbol {\alpha }}(\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}})-\hat{\boldsymbol {\alpha }}(\boldsymbol {q}_{_{\mathcal {\text{h}}}})}\vert \vert$. In this sense, a zero kinematics error (i.e., $\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}}=\boldsymbol {q}_{_{\mathcal {\text{h}}}}$) implies zero predicted muscle activity error (i.e., $\hat{\boldsymbol {\alpha }}(\hat{\boldsymbol {q}}_{_{\mathcal {\text{h}}}})=\hat{\boldsymbol {\alpha }}(\boldsymbol {q}_{_{\mathcal {\text{h}}}})$). The closer to 0, the better is the KMEA prediction.

In both $\text{puncturing}$ (left) and cutting (right) tasks, the muscular effort, as shown in Figure 9, remained close to predicted muscular minimum effort value. Yet, the $\text{puncturing}$ task allowed for a larger kinematic redundancy from participants while maintaining small variations of the muscular effort value. This is particularly clear for configurations based on the Comfort-Planner (blue cluster), which were able to accommodate larger joint deviations without undermining muscle activity. Configurations based on the Random-Planner forced participants to larger deviations from the KMEA assumption for both $\text{puncturing}$ and cutting scenarios. Such behavior may be associated with the difficulty of visualizing the task from such non-intuitive random configurations (as highlighted by participants). The muscular effort variation is also higher in the cutting task compared to the $\text{puncturing}$ task and more liable to increase even with small kinematic changes.

9 Individual Muscle Selection and Assessment

Table 4.

Code	Name	Function
PM	Pectoralis Major	Flex and medially rotate shoulder joint, horizontally adduct humerus toward opposite shoulder
AD	Anterior Deltoid	Flex, medially rotate and horizontally adduct the shoulder
PD	Posterior Deltoid	Extend, laterally rotate and horizontally abduct the shoulder
LD	Latissimus Dorsi	Extend, medially rotate and adduct the shoulder joint
BB	Biceps Brachii	Flex the elbow, supinate the forearm, assist to flex the shoulder
LT	Triceps B. (Lateral head)	Extend the elbow
MT	Triceps B. (Medial head)	Extend the elbow, extend the shoulder, assist to adduct the shoulder
FR	Flexor Carpi Radialis	Extend the wrist, abduct the wrist, assist to flex the elbow
ER	Extensor Carpi Radialis	Flex the wrist, abduct the wrist, may assist to flex the elbow

Table 4. Selection of Muscles and Their Respective Functions Associated with the Upper Limb [8]

Table 5.

	Comfort-Planner	Random-Planner	UserDef-Pose
$\text{Puncturing}$	0.780	0.896	0.855
Cutting	0.654	0.609	0.537

Table 5. Ratio between the Predicted Least Summed Muscular Force $\Vert {\hat{ \boldsymbol {f}}_{m}}\Vert$ from the Selected Muscles Compared to the 50 Muscle Segments in Saul et al. [45]

This section presents an additional discussion about the muscle selection, relevance to the task, and a thorough assessment of measured activity of individual muscles.

9.1 Muscle Selection

For all experiments in Section 8, the wireless sensors were placed in nine different anatomical positions selected to include the main actuators for the corresponding motor tasks across the shoulder, elbow, and wrist joints. Table 4 details the muscle code used and their body functions. It is important to highlight that the nine EMG sensors are in fact covering 16 individual muscle segments from the model in Saul et al. [45]. This is due to the fact that it is difficult to fully isolate muscle segments activities from certain muscles in real-world unconstrained applications—for example, although BIClong and BICshort presented in Saul et al. [45] have different contributions, anatomically, they belong and are considered just segments of the BB. The 16 corresponding muscle segments from the musculoskeletal system [45] are PECM{1,2,3} DELT{1,3}, LAT{1,2,3}, BIClong, BICshort, TRIlat, TRIlong, TRImed, FCR, ECRB, and ECRL—muscle code from OpenSim [21].

To better understand the relevance of the muscle selection—and validate their importance prior to the real experiments—to the planning results, we also computed the expected muscular force for the 50-muscles model in Saul et al. [45]. The ratio between the predicted muscle force stemming from the selected muscles compared to the total muscle force of the upper limb (generated by the 50 muscle segments) is shown in Table 5. As expected, the selected muscles cover most of the total force generated—about $78\%$ to $90\%$ of the total muscle force for $\text{puncturing}$ and $54\%$ to $65\%$ for cutting.

9.2 Individual Muscle Assessment

In this section, we explore specificities of the muscle activation and analyze whether the musculoskeletal model can predict which muscles are being activated during the fHRC.

Fig. 10.

Figure 10 presents the individual muscle recruitments against the total generated force, for all $\text{puncturing}$ trials (different vertical lines) performed under the comfort-planner strategy. From the results, it is clear how each participant employs different muscle configurations to complete the tasks, with AD, ECR, PD, and BB generally being the most active muscles for the $\text{puncturing}$ task (for the cutting task, the most activated muscles were PM, PD, MT, and FCR, yet for brevity, we will focus on $\text{puncturing}$). The muscle codes are described in Table 4.

The experiment was designed with a fixed human grasp per operation, yet there was no constraint on the human while executing the task. An unforeseen DoF rose from the hand grip the subjects were adopting while holding the tools. Although in the $\text{puncturing}$ task there is a fairly limited number of possible grips (typically a diagonal volar grip is used), for the cutting we have noticed two distinct grips (internal precision grip and the diagonal volar grip). The immediately noticeable effect of these distinct grip modes is on the FCR and ECR muscles, altering the subject’ s grasp force on the tool itself. This grasp force does not contribute directly toward the generated force at the end-effector of the tool. At this stage, the impact toward our algorithm development and model verification is minimal, and the individual measured muscle force normalized by the summed force is in most cases well distributed. Still, these different grips pose an interesting new question to solve at a later stage.

10 Conclusion and Future Work

This work provides a human-comfort-based planner that improves robot decision-making capabilities in shaping human musculoskeletal response during an fHRC. Our planner relies on predictions of the human-arm configuration and muscle to task-space mapping to deploy robot grasps and configurations that tailor human collaborative action toward a less muscular demanding interaction area.

The proposed comfort planning framework, agnostic to muscular activity estimation paradigms, enables the robot to shape human kinematics and musculoskeletal response. Effectiveness of our algorithm were first analyzed through different simulation scenarios. Furthermore, extensive experiments on two distinct motor tasks ($\text{puncturing}$ and cutting) have validated our comfort and kinematic response hypotheses. Most importantly, results demonstrated the efficiency of the proposed Comfort-Planner, which is able to identify and plan collaborative actions leading humans to exert smaller muscular forces. Experimental results have also shown that both random and user-defined poses fail to ensure satisfactory behavior for different participants, which highlights the importance of user-tailored planning solutions to improve fHRC acceptance. Results also reveal the practical potential of the proposed strategy to reduce muscular load in manufacturing and other industries through intelligent collaborative robots.

Among possible directions the proposed framework can lead to future findings, we highlight a few. First, possible improvements w.r.t. muscle activity prediction . Possible research focus would either include online learning strategies, relying on musculoskeletal model calibration, or adding robustness to the optimization to mitigate model uncertainties. The latter is more suitable for practical applications due to the constraints in measuring biofeedback signals outside a laboratory environment. In future works, we aim to assess additional human factors (e.g., safety perception [14, 35]) and task-specific execution (e.g., visibility criterion) to improve even further human experience during a collaborative action. We are also currently investigating methods based on precomputing a comfort map from KMEA for the human reachable space to speed up the comfort assessment, particularly for real-time control applications. Finally, we will also explore fHRC positioning influence over the human grasp. For instance, taking the human grasp clustering in other works [24, 47], we may explore if the robot can modify the human grasp for a given task.

Footnotes

The compiled experimental data will be available at data.leeds.ac.uk.

We use top-left superscripts to distinguish the coordinate frames w.r.t. which variable is expressed. For simplicity, superscripts will be omitted for world (robot) coordinate frame $\sum _{_{\mathcal {\text{r}}}}$.

The HFKM of the human upper limb deployed in Section 4.4 is defined in accordance to Figure 3, and to the musculoskeletal model of Holzbaur et al. [46], whereas in Section 4.4, a standard spherical-revolute-spherical model is employed to satisfy the non-redundant transformation used in other works [14, 51]. In general, the choice of HFKM will be dependent on the musculoskeletal model deployed for analysis as the torque to muscle mapping inherently relies on the HFKM and the mapping in (6).

⁴

We assume quasi-static interaction, yet extension to dynamics can be obtained directly, such as by means of the Lagrangian method [48], $\boldsymbol {M}_{_{\mathcal {\text{h}}}}\ddot{\boldsymbol {q}}_{_{\mathcal {\text{h}}}}{+}\boldsymbol {C}_{_{\mathcal {\text{h}}}}\dot{\boldsymbol {q}}_{_{\mathcal {\text{h}}}}=\boldsymbol {\tau }_{_{\mathcal {\text{h}}}}-\boldsymbol {J}^{T}_{_{\mathcal {\text{h}}}}(\boldsymbol {f}{+}\boldsymbol {\rho }_{\text{t}}){-}\boldsymbol {\tau }_{\rho },$ where $\boldsymbol {f}$ is the end-pose wrench, and $\boldsymbol {M}_{_{\mathcal {\text{h}}}}$ and $\boldsymbol {C}_{_{\mathcal {\text{h}}}}$ are the inertia and centrifugal-Coriolis forces.

⁵

Experimental data to compute the maximum torque matrix $\boldsymbol {\varPi }(\boldsymbol {q}_{_{\mathcal {\text{h}}}},\boldsymbol {\tau }_{_{\mathcal {\text{h}}}})$ in (8) is found in Tanaka et al. [51]. Proportions for maximum joint torques follow a similar pattern across individuals and, even for different body weights, differ only in magnitude.

⁶

Experiments were approved by the ethics committee of the University of Leeds (ID: MEEC 17-034) respecting the Declaration of Helsinki. An exclusion criteria was applied, ensuring that people with known neurophysiological disorders or injuries were not selected. All participants were in good physical condition, followed experimental guidelines (e.g., use of sports gear, avoidance of caffeine or exercise in both days prior to the experiment), and provided their full consent toward the experiments.

⁷

Given the forceful nature of the tasks, it was seen in the pilot experiments that participants were fatigued when performing too many trials. Hence, only one comfort-based planner was implemented which suffices to illustrate its advantages.

References

[1]

Arash Ajoudani, Andrea Maria Zanchettin, Serena Ivaldi, Alin Albu-Schäffer, Kazuhiro Kosuge, and Oussama Khatib. 2018. Progress and prospects of the human–robot collaboration. Autonomous Robots 42, 5 (June 2018), 957–975. https://doi.org/10.1007/s10514-017-9677-2

	Musc. Act. Effort (\(\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}\right\Vert\))			Musc. Act. Effort (\(\left\Vert \hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\right\Vert _{ }\))
	RndP	Fixed-Pose	MR-Comf-Planner	RndP	Fixed-Pose	SO-Comf-Planner
Puncturing\(\phantom{\int }\)	1	\(.920\pm .310\)	\(\boldsymbol {.606\pm .342}\)	1	\(1.279\pm .517\)	\(\boldsymbol {.341\pm .145}\)
Pulling	1	\(1.189\pm .445\)	\(\boldsymbol {.371\pm .141}\)	1	\(1.073\pm .336\)	\(\boldsymbol {.262\pm .087}\)
Cutting	1	\(1.164\pm .434\)	\(\boldsymbol {.817\pm .205}\)	1	\(2.340\pm .953\)	\(\boldsymbol {.610\pm .193}\)

		Subject B	Subject C	Subject D	Subject E
Puncturing	UserDef	\(495.58\pm 132.76\)	\(1,105.00\pm 475.87\)	\(1,639.67\pm 430.74\)	\(526.72\pm 116.17\)
	Comfort	\(\boldsymbol {361.67\pm 54.81}\)	\(\boldsymbol {813.66\pm 173.86}\)	\(\boldsymbol {633.52\pm 304.06}\)	\(\boldsymbol {345.12\pm 102.02}\)
	Random	\(577.52\pm 182.18\)	\(883.16\pm 231.48\)	\(1,675.83\pm 210.15\)	\(\boldsymbol {336.83\pm 123.44}\)
Cutting	UserDef	\(\boldsymbol {696.31\pm 242.30}\)	\(1,037.05\pm 114.04\)	\(\boldsymbol {1,698.91\pm 209.35}\)	\(942.67\pm 251.24\)
	Comfort	\(\boldsymbol {696.43\pm 194.50}\)	\(1,568.48\pm 280.71\)	\(1,925.61\pm 600.57\)	\(\boldsymbol {574.36\pm 53.29}\)
	Random	\(725.10\pm 124.84\)	\(\boldsymbol {713.24\pm 110.90}\)	\(1,882.23\pm 508.40\)	\(1,380.04\pm 100.56\)

Abstract

1 Introduction

2 Related Work

3 Problem Formulation

4 Predicting the Muscle Effort: Task- to Muscle-space Mapping

4.1 Muscular Effort

4.2 Task-to Joint-Space Modeling for the Human Arm

4.3 Muscular Activity Prediction Based on Static Optimization (\(\hat{\boldsymbol {\alpha }}_{_{\text{SO}}}\))

4.4 Muscular Activity Prediction Based on Model Reduction (\(\text{$ \hat{\boldsymbol {\alpha }}_{_{\text{MR}}}$}\))

5 Predicting the Human Kinematic Configuration: the KMEA

6 Ensuring Grasp and Robot Stability

7 Planning: Quantitative Analysis

7.1 Optimization Performance

7.2 Optimization Performance: Multiple Tasks

8 Human-Robot Experiments

8.1 Comfort Performance from EMGs Biofeedback

Muscle-activity prediction.

8.2 Participants’ Perception of Comfort

8.3 Quantitative Analysis of KMEA

9 Individual Muscle Selection and Assessment

9.1 Muscle Selection

9.2 Individual Muscle Assessment

10 Conclusion and Future Work

Footnotes

References

Cited By

Index Terms

Recommendations

A selective muscle fatigue management approach to ergonomic human-robot co-manipulation

Manipulation Planning for Forceful Human-Robot-Collaboration

Operational space analysis of human muscular effort in robot assisted reaching tasks

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations