Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
A Gamma Process with Three Sources of Variability
Next Article in Special Issue
Event-Triggered Sliding Mode Impulsive Control for Lower Limb Rehabilitation Exoskeleton Robot Gait Tracking
Previous Article in Journal
Statistical Analysis of Alpha Power Inverse Weibull Distribution under Hybrid Censored Scheme with Applications to Ball Bearings Technology and Biomedical Data
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Symmetry
Volume 15
Issue 1
10.3390/sym15010163
symmetry-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Gait Phase Classification of Lower Limb Exoskeleton Based on a Compound Network Model

by
Yuxuan Xia
1,
Jiaqian Li
2,*,
Dong Yang
1 and
Wei Wei
1,*
1
College of Optoelectronics Science and Engineering, Soochow University, Suzhou 215000, China
2
Suzhou Industrial Park Institute of Vocational Technology, Suzhou 215123, China
*
Authors to whom correspondence should be addressed.
Symmetry 2023, 15(1), 163; https://doi.org/10.3390/sym15010163
Submission received: 22 November 2022 / Revised: 13 December 2022 / Accepted: 3 January 2023 / Published: 5 January 2023
(This article belongs to the Special Issue Biomechanics and Energetics of Human Motion When Walking with Lower Limb Orthoses, Prosthesis, and Exoskeletons)
Browse Figures
Figure 1
<p>Block diagram of the gait acquisition system.</p> "> Figure 2
<p>Relationship between the foot and ground of the lower limb exoskeleton robot: foot lift (L), foot hang (H), and foot support (S).</p> "> Figure 3
<p>Time-varying curve of gait phase at 6 km/h.</p> "> Figure 4
<p>Positions of optical markers and inertial measurement units (IMUs).</p> "> Figure 5
<p>Overall Structure of the Network Model.</p> "> Figure 6
<p>Structure diagram of long short-term memory (LSTM) cycle unit.</p> "> Figure 7
<p>Bidirectional long short-term memory (BiLSTM) neural network.</p> "> Figure 8
<p>Classification accuracy under CNN-BiLSTM network model.</p> "> Figure 9
<p>Classification accuracy under LSTM network model.</p> "> Figure 10
<p>Classification accuracy under gated recurrent unit (GRU) network model.</p> "> Figure 11
<p>Sample size of each gait phase.</p> "> Figure 12
<p>Classification accuracy at 50 Hz.</p> ">
Versions Notes

Abstract

:
The classification of lower limb gait phase is very important for the control of exoskeleton robots. In order to enable the exoskeleton to determine gait phase and provide appropriate assistance to the wearer, we propose a compound network based on CNN-BiLSTM. The method uses data from inertial measurement units placed on the leg and pressure sensor arrays placed on the sole as inputs to the model. The convolutional neural network (CNN) is used to obtain the local key features of gait data, and then the bidirectional long short-term memory (BiLSTM) network is used to extract the serialized gait phase information from the local key features to obtain the high-level feature expression. Finally, the seven phases of both feet were obtained through the classification of the softmax layer. We designed a gait acquisition system and collected the gait data from seven subjects at varying walking speeds. In the test set, the highest gait phase classification accuracy can reach 95.09%. We compared the proposed model with the long short-term memory (LSTM) network and gated recurrent unit (GRU) network. The experimental results show that the average accuracy of CNN-BiLSTM network from seven subjects is 0.417% higher than that of the LSTM network and 0.596% higher than that of the GRU network. Therefore, the ability of the CNN-BiLSTM network to classify gait phases can be applied in designing exoskeleton controllers that can better assist for different gait phases correctly to assist the wearer to walk.

1. Introduction

In the past few decades, lower limb exoskeleton robots have been continuously researched. They have shown great application potential in numerous military, medical and industrial fields [1]. According to different expected functions, lower limb exoskeleton robots can be divided into power-assisted and rehabilitation types. Power-assisted lower limb exoskeletons are mainly used to enhance the speed, strength and endurance of normal humans, such as BLEEX [2], HULC [3] and HAL [4]. Rehabilitation lower limb exoskeletons are mainly used to assist disabled people to walk and help patients perform rehabilitation training, such as Indego [5], ReWalk [6] and ALEX [7]. With the continuous expansion to different application scenarios, lower limb exoskeleton robots need to adapt to more and more complex environments and the active movement of the wearer. The lower limb exoskeleton is a highly coupled human-machine system, and its walking gait is highly consistent with human gait. Human joints and exoskeleton joints have similar movements under the same gait phase. Gait phase recognition is of great significance for the human-machine coordination control of lower limb exoskeleton robots. First of all, gait phase is widely used in the lower limb exoskeleton gait trajectory generation, gait control and gait evaluation tasks, which can be used as a starting point for better human-machine coordination control. Secondly, accurate recognition of gait phase is the key to achieve efficient metabolic control of the lower limb exoskeleton. Inaccurate gait phase information often increases the user’s effort or causes the motor to impose inappropriate joint torque. Moreover, in some lower limb exoskeleton devices based on phase control, gait phase often determines the control mode required by the control system at present, which is essential information for realizing the control of lower limb exoskeleton robot. For example, Kazerooni et al. [2] designed the lower limb exoskeleton BLEEX and divided the human walking cycle into three gait phases: single support, double support and double support with one redundancy, and built corresponding motion models and control algorithms for different gait phases, thus realizing the control of lower limb exoskeleton.
In order to recognize the characteristics of human gait phase, scholars at home and abroad have carried out a significant amount of research, mainly on machine vision recognition and sensor recognition. The requirements of recognition based on machine vision are much higher than those of sensor recognition, and the process is complex, so it is limited to experimental scenes. The sensor-based recognition not only has a low cost, small size, simple and convenient installation, but is also not easily affected by the external environment. Therefore, most human motion recognition based on lower limb exoskeleton uses sensors. At present, the mainstream sensors include the plantar pressure sensor, inertial measurement unit (IMU), surface electromyogram (SEMG) and electroencephalogram (EEG). Lim et al. [8] developed an algorithm to detect gait phase with a small number of sensors by using center of pressure (COP). Liu et al. [9] used inertial sensors to collect real-time motion data, calculated the group correlation coefficient between motion data and template data, and used the hidden Markov model (HMM) to identify the final motion state. Nguyen et al. [10] adopted distributed plantar pressure sensors to obtain human motion information, and used K-nearest neighbor (KNN) classification method to realize recognition of five motion patterns, including flat walking, upstairs, downstairs, uphill and downhill. Hsu et al. [11] used a non-parametric weighted feature extraction (NWFE) algorithm and principal component analysis (PCA) method to reduce the feature size of acceleration signals, and used a least square support vector machine (LS- BiLSTM) motion pattern based on one-to-one strategy for classification and recognition. Wu et al. [12] designed a continuous gait phase estimator based on an adaptive oscillator network, including a gait task classifier, a gait target reset, a peak detector, and a model-based transitional gait phase estimator to improve the performance of the gait target network. Qian et al. [13] proposed a gait phase estimation method based on an adaptive oscillator, which can accurately estimate the gait phase of users when they move on various terrain. Wei et al. [14] used SEMG and EEG to compare the performance of linear discriminant analysis (LDA), KNN, and a kernel support vector machine (KSVM). Qin et al. [15] proposed a human gait phase recognition algorithm based on fuzzy theory to identify the gait phase at the next moment of human lower limb movement.
Based on a large amount of data, machine learning methods are not affected by biomechanical models and cost functions [16], and gradually become a feasible method for gait phase classification. Zeng et al. [17] proposed a back propagation (BP) neural network algorithm model based on support vector machine (SVM), which improved the average recognition accuracy by 9.3552% compared with the SVM algorithm. Lee et al. [18] estimated the knee angle as well as the angles of the talocrural and subtalar joints of the ankle by applying neural networks to two IMUs attached to the thigh and shank. Qiu et al. [19] used intrinsic mode functions (IMFs) extracted from the original gait signals by ensemble empirical mode decomposition (EEMD) as the input of the classification algorithm to give 14 experimental results of normal human gait phase recognition. Lu et al. [20] used a machine learning classifier to select feature subsets of EMG, IMU and foot switch signals from a series of time-domain features of window parameters, and realized the classification of different stages of human jumping. This team [21] also used SEMG as an input to improve the recognition accuracy of the human lower limb jumping stage by bidirectional long short-term memory (BiLSTM) and convolution long short-term memory (ConvLSTM). Huang et al. [22] proposed an online gait detection method based on distance and multi-sensor information fusion to solve the problem of overfitting in the case of limited data. Yunas et al. [23] used the deep learning model to automatically extract the ground reaction force characteristics and the body movements attached to three different positions of the lower body, making up for the decline in the spatiotemporal accuracy of the individual model. Wang et al. [24] proposed a deep learning-based approach to map multi-channel SEMG signals to human lower limb movements involving four different gait phases and three flexion and extension joint angles. Kang et al. [25] utilized a gait phase estimator based on convolutional neural network (CNN) that can adapt to different locomotion mode settings to modulate the exoskeleton assistance. Wu et al. [26] also proposed a graph convolution network model (GCNM) for the gait phase classification of lower extremity exoskeletons, which can solve the problem of gait phase classification in non-Euclidean domains based on exoskeleton diagram mechanism.
Many existing studies based on deep learning place too much faith in the processing power of models, and the feature extraction of limited data is not comprehensive enough. The above research [25] uses the CNN-based model to inherently extract the feature information in the sensor data in the neural network architecture. However, lower limb walking is a continuous process, and its characteristics in the time series are also important. In the above study [21], BiLSTM and ConvLSTM were used to identify the jump stages, and SEMG and IMU data were used as nodes. However, this method only considers the adjacency relation between nodes, ignoring the physical significance of nodes themselves. These studies can obtain a better recognition rate under normal conditions, but may reduce the accuracy of gait phase recognition under complex conditions. In order to make the exoskeleton adapt continuously to varying walking speeds, we designed the gait acquisition system and established the movement pattern data set. We recruited seven healthy subjects and collected lower limb movement data and plantar pressure data at continuously varying walking speed. In this paper, based on the analysis of the motion pattern of the human body under the continuously varying walking speed, a gait phase recognition method based on the CNN-BiLSTM network model is proposed to classify the seven gait phases of both feet. Recently, the CNN-BiLSTM model has shown excellent performance in tasks such as text classification. This model combines the CNN model and BiLSTM model, which can not only extract local features in data, but also extract forward and backward temporal features of data. We compared CNN-BiLSTM network with long short-term memory (LSTM) network and gated recurrent unit (GRU) network. The experimental results show that the average accuracy of the CNN-BiLSTM network in seven subjects is 0.417% higher than that of the LSTM network and 0.596% higher than that of the GRU network. Therefore, the proposed network can better distinguish the gait phase of human lower limbs under the continuous change of walking speed.

2. Materials and Methods

2.1. Data Collection and Processing

Our team developed a gait acquisition system to capture and analyze human gait to aid in exoskeleton control, as shown in Figure 1. The system includes computer, micro control unit (MCU), IMU, plantar pressure sensor and camera. The computer is used to receive sensor and video data and play prompts to stimulate the subjects to control their walking speed. The MCU is used to generate some trigger signals to control the plantar pressure sensor and the camera to collect data. The IMU and the plantar pressure sensor are used to collect the leg movement information and the plantar pressure information, respectively. The image information of the subject when walking is collected by the camera to label the data.
In general, the Rancho Los Amigos (RLA) scale is used as the standard to divide gait into eight stages [27]. The RLA terminology classifies gait stages as follows: initial contact, load response, middle stance, terminal stance, front swing, initial swing, middle swing, and end swing. Since the design of the exoskeleton controller usually provides more assistance to the leg lift and swing stages, we divided the movement of one leg of the lower limb exoskeleton into three stages, foot lift, foot hang and foot support as shown in Figure 2. When the foot ends the hang phase and begins to contact the ground, we consider it as the support phase. This classification criterion will be used in this paper. According to this classification criterion, for both feet, there are seven phases in the normal walking gait. They are: left foot lift, right foot hang (LH); left foot lift, right foot support (LS); left foot hang, right foot lift (HL); left foot hang, right foot support (HS); left foot support, right foot lift (SL); left foot support, right foot hang (SH); left foot support, right foot support (SS).
The gait acquisition system used in the experiment is based on the passive lower limb weight-bearing exoskeleton developed by MeBotX Intelligent Technology (Suzhou) Co., Ltd. in Suzhou, China. Each leg of the exoskeleton robot has three joints: hip joint, knee joint and ankle joint. Each leg has seven degrees of freedom. Each joint can produce a flexion/extension motion to produce motion in the forward and backward direction. The role of the exoskeleton robot in this experiment is to fix the sensors and optical markers. Because all the degrees of freedom are passive and the motion range is sufficient, the robot will not affect the normal gait of the subjects during the experiment. Table 1 shows the specifications of the exoskeleton robot.
The gait data acquisition system includes two kinds of sensors. One is the IMU, which is fixed at the thigh and shank of the exoskeleton robot to collect the triaxial acceleration and triaxial angular velocity information of each limb. The gait data acquisition system can record an acceleration range of ±16 g at 2000 Hz through the IMU in the Noraxon Ultium biomechanical research system. The other sensor is the plantar pressure sensor, for which we chose pedar-x. The pedar-x is an in-shoe dynamic pressure distribution measuring system which utilizes capacitive sensors. The system has 100 measuring points on the sole of a foot, which can sum the values of the areas in front, middle and back of the foot to obtain the pressure data of these three areas. The sampling frequency of plantar pressure is 100 Hz. Due to the different foot sizes of the wearer, the device provides the user with different insole sizes for more accurate results.
We recorded gait data from people of different heights and weights. Six healthy male subjects and one healthy female subject, none of whom had ever had a gait disorder and all of whom were between 25 and 30 years of old, participated in the study. Table 2 gives the details of the subjects. All subjects gave informed written consent to participate. Participation was voluntary and could be discontinued at any time.
To ensure that the joint axis of the exoskeleton was aligned with the subject, the thigh and shank of the exoskeleton were adjusted before gait data were collected. The whole process was carried out on a GZ8643 motorized treadmill, which can walk at a maximum speed of 10 km/h. After the above adjustment, gait data was collected by the gait data acquisition system. In this experiment, each subject was asked to walk continuously at a speed of 4 km/h for 4 min, 6 km/h for 3 min, and 2 km/h for 3 min. A computer was used to record data, including limb movement data and plantar pressure data. Finally, the triaxial acceleration and triaxial angular velocity information of the thigh and shank were obtained at a sampling frequency of 2000 Hz, and the pressure information of the front, middle and heel of the foot was obtained at a sampling frequency of 100 Hz. In order to make the frequency of the two types of data consistent, the average value was taken every 20 inertial data.
In order to verify the effectiveness of the gait phase classification algorithm, an image acquisition system was used in this study to record the image information of the wearer walking in real time. The camera was triggered by PWM pulses for image capture at a frequency of 100 Hz. Figure 3 shows the time-varying curve of gait phase of subject No. 5 when walking at 6 km/h. As can be seen from Figure 3, due to the fast walking speed, there is no gait phase of both foot support, while other gait phases change periodically. This indicates that the gait acquisition system can capture gait phases of the subjects when they walk at the walking speed required by the experiment. In order to improve the efficiency of manual labeling of image data, optical markers were fixed on the hip joint, knee joint and ankle joint of the wearer, and a solid-color background plate was arranged. The positions of optical markers and IMU are shown in Figure 4.

2.2. CNN-BiLSTM Network Model for Gait Phase Classification

In this study, each group of gait data is represented as gait phase characteristic matrix I M × N , where M represents the time dimension and N represents the gait phase characteristic dimension, including the three-axis acceleration data of the thigh, the three-axis angular velocity data of the thigh, the three-axis acceleration data of the shank, the three-axis angular velocity data of the shank, and the pressure of the front, middle and heel of the foot. The network model includes input layer, CNN layer, BiLSTM layer and softmax layer. The overall structure of the model is shown in Figure 5.
The proposed compound network model simulates the specific process of human gait phase discrimination as follows: firstly, CNN is used to obtain the key local feature segments in the global gait data, and then BiLSTM network is used to obtain the gait feature context information from the key feature segments. The fully connected layer was introduced to reduce the dimension, and the softmax classifier was input to calculate the probability distribution of the sensor data belonging to the category label of gait phase, and the prediction results were directly output. Table 3 shows the formal definition of the output of each layer of the CNN-BiLSTM network model based on a set of gait data.

2.2.1. CNN Layer

For the gait phase input sequence, I = ( i 1 , i 2 , … , i M ) , i i ϵ R N is the gait data vector pre-trained by word2vec, where N is the dimension of gait data vector. The width of the convolution kernel is consistent with the embedding dimension of gait data, and the number of data taken in the window of each convolution operation is denoted as h ; that is, the convolution kernel i i ϵ R h ∗ N . The convolution result N i for each window sliding is shown in Equation (1). ReLU is the nonlinear activation function, ω is the number of data taken in each convolution operation, and b is the bias term.
N i = R e L U ( ω i i : i + h − 1 ) + b
The length of sequence I is M , the window slides M − h + 1 times, and the summary result of convolution is n = [ n 1 , n 2 , … n M − h + 1 ] . Then, according to the window size and step size of the pooling layer, maxpooling operation is performed on the results of the convolution layer to increase the receptive field of the upper convolution kernel, retain the main features of the gait data embedding vector sequence, and reduce the parameters and computation of the next layer to prevent overfitting.
A three-layer CNN structure with different window sizes, the same number of convolution kernels, and the padding parameter set to the same mode ensures that the dimensions of the input vector and output vector are the same, and the step size of the convolution kernel is one, and the gait data embedding vector matrix is convolved and pooled. Concatenate operation is performed on the output of the three-layer CNN according to the axial direction to enrich the contextual gait feature content of the CNN model convolved with the gait data window and better characterize the local features of the gait sequence.

2.2.2. BiLSTM Layer

The transmission direction of LSTM is one-way transmission from front to back. However, the internal correlation of gait phase is strong, and the current state is not only related to the state at the previous time, but also to the state at the later time. Therefore, BiLSTM network is constructed by LSTM layer with two directions to identify past and future gait phase information and realize gait phase context information modeling [28].
The structure of LSTM cyclic unit includes three “gates” and two states, which are input gate i t , forget gate f t , output gate o t , internal state c t and candidate state c t ′ , shown in Figure 6. Assuming that the external state at time t is h t and the external state at the previous time is h t − 1 , the calculation process of LSTM is: combining the external state h t − 1 at the previous time and the input gait phase feature vector n t at the current time, the three gate values and candidate state values in the LSTM cycle unit are calculated by Equations (2)–(5). The forgetting gate f t and the input gate i t are used to update the memory unit c t through Equation (6), and the output gate o t is used to transfer the gait phase information of the internal state to the external state h t through Equation (7).
Where, x ∈ { i , f , o , c } in W x , U x , and b x , W x is the weight matrix at the current time, U x is the weight matrix at the previous time, b x is the bias vector, σ is the Sigmoid function and t a n h is the hyperbolic tangent function.
i t = σ ( W i n t + U i h t − 1 + b i )
f t = σ ( W f n t + U f h t − 1 + b f )
o t = σ ( W o n t + U o h t − 1 + b o )
c t ′ = t a n h ( W c n t + U c h t − 1 + b c )
c t = f t ⊙ c t − 1 + i t ⊙ c t ′
h t = o t ⊙ t a n h ( c t )
BiLSTM extracts and saves past and future gait phase information by using Equations (8) and (9) through LSTM of forward layer and backward layer, respectively. The structure of single-layer BiLSTM network is shown in Figure 7. Assuming that the forward layer is in chronological order and the backward layer is in reverse chronological order, the hidden layer states at time t are defined as h t 1 and h t 2 , and the output vector l t of the bidirectional long and short term memory layer at time t is calculated according to the hidden layer states in the two directions, as shown in Equation (10).
h t 1 = f ( U 1 h t − 1 1 + W 1 n t + b 1 )
h t 2 = f ( U 2 h t − 1 2 + W 2 n t + b 2 )
l t = W t 1 h t 1 + W t 2 h t 2 + b 0
where, W x ( x ϵ { 1 , 2 } ) is the weight matrix of the current moment, U 1 and U 2 are the weight matrices of the previous time and the next time, respectively, f represents the activation function of the hidden layer, W t x ( x ϵ { 1 , 2 } ) is the state weight matrix of the hidden layer at the current time, and b x ( x ϵ { 0 , 1 , 2 } ) is the bias vector. The serialized gait phase feature matrix L D × H is obtained by two-layer BiLSTM.

2.3. Model Training Loss Function

As an indispensable part of deep learning model training, the mean square error (MSE) and mean absolute error (MAE) are often used as loss functions in regression model training. MAE is insensitive to outliers, and the gradient does not decrease with the loss value in the process of gradient updating, which is not conducive to model convergence. MSE is sensitive to outliers, and the gradient decreases with the decrease of the loss value in the process of gradient updating, which is conducive to the convergence of the model. Outliers are a few parts of data whose distribution rules are significantly different from the mainstream data, which often contain the changing trends, so they cannot be simply equated with noise. Because of the complex and diversified characteristics of gait phase, the outliers in gait phase information may be the points where the phase of gait suddenly changes, but the possibility of noise data cannot be excluded. Considering the sensitivity to outliers and convergence, MSE is chosen as the model training loss function in this paper, and its calculation formula is shown in Equation (11).
M S E ( i ) = 1 N ∑ i = 1 N ( y i − y ^ i ) 2
where N is the total number of gait phase data points, y i is the label truth value of the i -th gait phase data point, and y ^ i is the regression value of the i -th gait phase data point.

3. Results

Gait phases of subjects were classified by performing three experiments using machine learning methods: CNN-BiLSTM, LSTM and GRU. During the walking process, the gait acquisition system recorded the movement data of thigh and calf and the pressure data of plantar in real time. The input of the above model is shown in Table 3. Seven subjects were asked to wear the gait acquisition system and walk continuously at 4 km/h for 4 min, 6 km/h for 3 min and 2 km/h for 3 min. After processing, the data refresh frequency of each sensor is 100 Hz. Theoretically, the total number of gait data samples in each group is 60,000, but there will be a small deviation in the actual number of samples collected.
To evaluate the accuracy of gait stage classification, we defined a criterion, namely classification success rate (CSR), as shown in Equation (12).
N c represents the number of correct classifications and N t represents the total number of test data points. We use confusion matrices to illustrate classification performance and quantify error distributions. The definition is shown in Equation (13). Each element is defined as shown in Equation (14).
C S R = N c N t
C = ( c 11 c 12 c 21 c 22 c 13 c 14 c 23 c 24 c 31 c 32 c 41 c 42 c 33 c 34 c 43 c 44 )
c i j = m i j m i
where m i j is the number of test data points in phase pattern i classified as pattern j ; m i is the total number of test data points in phase pattern i . It is clear that the diagonal elements of the confusion matrix represent the classification success rate and the off-diagonal elements represent the error rate.
In order to verify the robustness and superiority of the CNN-BiLSTM used, the subject samples were randomly shuffled and divided into training-test sets. The training-test set percentages were 80% and 20% of the gait cycle samples of the subjects, respectively.
In the experiment, the model optimization algorithm of CNN-BiLSTM network was set as Adam, the learning rate was 0.005, and each batch had 32 samples. ReLU was used as the activation function in the model construction process, and the number of training rounds is 100. The number of BiLSTM units was set to 128 and the dropout ratio was set to 0.25. Based on the data of the gait acquisition system, the 30-dimensional features are extracted from the gait data. Each part of CNN and BiLSTM consists of two layers of neural networks. Because the output of the previous layer is the input of the next layer, Connection is used as the connection layer to avoid the repeated representation of the input and output layers. Based on the CNN-BiLSTM network, the average accuracy of the gait phase classification of seven subjects was 92.989%. Among them, the gait phase classification accuracy of the subject with the highest accuracy can reach 95.09%, and the confusion matrix is shown in Figure 8. The prediction accuracy of LH, LS, HL, HS, SL, SH and SS phases were 69%, 96%, 82%, 97%, 94%, 98% and 81%, respectively.
In order to verify the effectiveness of the proposed network, we compared its experimental results with the LSTM network and GRU network. The LSTM network is a special RNN network, mainly to solve the problem of gradient disappearance and gradient explosion in the process of long sequence training. The two-layer LSTM network with a softmax layer was selected for gait phase classification. The hidden layers include 128 memory blocks for each LSTM network to process the input gait phase feature matrix. Then, the corresponding gait phase is obtained by applying softmax layer. Based on LSTM network, the average accuracy of gait phase classification of seven subjects was 92.571%. Among them, the gait phase classification accuracy of the subject with the highest accuracy can reach 94.13%, and the confusion matrix is shown in Figure 9. The prediction accuracies of LH, LS, HL, HS, SL, SH and SS phase were 68%, 94%, 76%, 96%, 94%, 98% and 23%, respectively.
GRU network has fewer parameters than LSTM network, so it is easy to converge. The two-layer GRU network with a softmax layer was selected for gait phase classification. To facilitate comparison, the hidden layers also include 128 memory blocks for each GRU network to process the input gait phase feature matrix. Then, the corresponding gait phase is obtained by applying the softmax layer. Based on the GRU network, the average accuracy of gait phase classification of seven subjects was 92.393%. Among them, the gait phase classification accuracy of the subject with the highest accuracy can reach 94.31%, and the confusion matrix is shown in Figure 10. The prediction accuracies of LH, LS, HL, HS, SL, SH and SS phases were 70%, 95%, 82%, 96%, 92%, 98% and 17%, respectively.

4. Discussion

The main finding of our study is that the proposed CNN-BiLSTM network can reliably classify the seven phases of bipedal gait. In seven subjects, the average classification accuracy of each phase can reach 95.09%. We compared it with the classification results of the LSTM network and the GRU network. In the experimental results of subject No. 1, the average accuracy of the CNN-BiLSTM network was 0.16% higher than that of the LSTM network and 0.3% higher than that of the GRU network. In the experimental results of subject No. 2, the average accuracy of the CNN-BiLSTM network was 0.76% higher than that of the LSTM network and 0.58% higher than that of the GRU network. In the experimental results of subject No. 3, the average accuracy of the CNN-BiLSTM network was 0.26% higher than that of the LSTM network and 0.74% higher than that of the GRU network. In the experimental results of subject No. 4, the average accuracy of the CNN-BiLSTM network was 0.13% higher than that of the LSTM network and 0.44% higher than that of the GRU network. In the experimental results of subject No. 5, the average accuracy of the CNN-BiLSTM network was 0.96% higher than that of the LSTM network and 0.78% higher than that of the GRU network. In the experimental results of subject No. 6, the average accuracy of the CNN-BiLSTM network was 0.28% higher than that of the LSTM network and 0.33% higher than that of the GRU network. In the experimental results of subject No. 7, the average accuracy of the CNN-BiLSTM network was 0.37% higher than that of the LSTM network and 1% higher than that of the GRU network. The average classification accuracy of each phase of all subjects is shown in Table 4. As can be seen from Table 4, under the same experimental environment, compared with the LSTM network and GRU network, the proposed CNN-BiLSTM network model has significantly higher prediction accuracy and better robustness for gait phase classification of lower limb exoskeleton system.
Under different network models, the recognition accuracy of each gait phase of subjects with the highest accuracy is shown in Table 5. As can be seen from Table 5, the recognition accuracy of LH, HL and SS is low, which may be related to the short time of these three phases in the gait cycle and the small amount of data. At the same time, due to the small amount of data, these three phases have little influence on the average recognition accuracy. Taking subject No. 5 as an example, the sample sizes of each gait phase are shown in Figure 11. To verify this idea, data of subject No. 5 were uniformly sampled to reduce the frequency from 100 Hz to 50 Hz. Its classification accuracy under the CNN-BiLSTM network is shown in Figure 12. It can be seen from Figure 12 that the recognition accuracy of all gait phases decreases after the data frequency decreases. LS, SL; HS and SH are two states with symmetrical left and right feet, respectively, so the recognition accuracy is similar. These phases have a large amount of data. In the results of the CNN-BiLSTM network, the recognition accuracy is no less than 94%, and the highest is 98%. In most gait phases, the recognition accuracy of the CNN-BiLSTM network is higher than that of the LSTM network and GRU network.
There are some limitations to the current study. The experiment was limited to seven healthy young subjects and was limited to walking on a treadmill. In the future, experiments should include more participants, such as the elderly and people with walking disabilities, to verify the applicability of this approach in a wider context. Walking in different conditions, such as uphill or downhill or on uneven surfaces can also test the robustness of the method.

5. Conclusions

This paper presents a gait phase classification method for lower limb exoskeleton control. We designed a gait acquisition system and collected IMU data of legs and plantar pressure data for model training. We used the passive lower limb exoskeleton robot as the experimental platform, and labeled each group of data through the images taken by the camera. Finally, this paper proposes a network model based on CNN-BiLSTM. It combines the CNN model with the BiLSTM model to classify and recognize the multi-type gait data sets collected in this study. Firstly, CNN is used to obtain the key local feature segments in the global gait data, then the BiLSTM network is used to obtain the context information of gait features from the key feature segments, and softmax layer is used as the output layer to determine the gait phase classification at the current moment. The experimental results show that:
(1) The gait acquisition system designed in this paper can effectively complete the collection of four IMU signals and plantar pressure signal of both feet. Through the manual analysis of the motion image and the position of the optical marker, the seven gait phases of both feet can be accurately distinguished.
(2) The CNN-BiLSTM classification method proposed in this paper can extract features from gait data more adequately than a single network. Compared with the LSTM network and GRU network, the average accuracy of the proposed CNN-BiLSTM network model is increased by 0.417% and 0.596%, respectively. For each subject, the recognition accuracy of the proposed model was higher than that of the other two models. Therefore, the proposed CNN-BiLSTM network model has higher prediction accuracy and better generalization performance. It can adapt to the complex walking gait of exoskeleton wearers. In our future work, we will focus on assisting the wearer through a powered exoskeleton using the proposed gait stage classification method.

Author Contributions

Conceptualization, Y.X. and J.L.; methodology, Y.X.; validation, Y.X. and D.Y.; writing—original draft, Y.X.; writing—review and editing, Y.X. and J.L.; project administration, W.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

Data sharing not applicable.

Acknowledgments

The authors would like to thank all authors of previous papers for approving the use of their published research results in this paper.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Ma, Y.; Wu, X.; Yi, J.; Wang, C.; Chen, C. A Review on Human-Exoskeleton Coordination towards Lower Limb Robotic Exoskeleton Systems. Int. J. Robot. Autom. 2019, 34, 431–451. [Google Scholar] [CrossRef] [Green Version]
  2. Kazerooni, H.; Racine, J.L.; Huang, L.; Steger, R. On the Control of the Berkeley Lower Extremity Exoskeleton (BLEEX). In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, 18–22 April 2006; pp. 4353–4360. [Google Scholar] [CrossRef]
  3. Bogue, R. Exoskeletons and robotic prosthetics: A review of recent developments. Ind. Robot. Int. J. 2009, 36, 421–427. [Google Scholar] [CrossRef]
  4. Kawamoto, H.; Kandone, H.; Sakurai, T.; Ariyasu, R.; Ueno, Y.; Eguchi, K.; Sankai, Y. Development of an assist controller with robot suit HAL for hemiplegic patients using motion data on the unaffected side. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA, 26–30 August 2014; Volume 2014, pp. 3077–3080. [Google Scholar] [CrossRef]
  5. Quintero, H.A.; Farris, R.J.; Goldfarb, M. Control and implementation of a powered lower limb orthosis to aid walking in paraplegic individuals. In Proceedings of the International Conference on Rehabilitation Robotics, Zurich, Switzerland, 29 June–1 July 2011; Volume 2011, p. 5975481. [Google Scholar] [CrossRef] [Green Version]
  6. Talaty, M.; Esquenazi, A.; Briceno, J.E. Differentiating ability in users of the ReWalk(TM) powered exoskeleton: An analysis of walking kinematics. In Proceedings of the International Conference on Rehabilitation Robotics, Seattle, WA, USA, 20–24 June 2013; Volume 2013, p. 6650469. [Google Scholar] [CrossRef]
  7. Rifai, H.; Mohammed, S.; Djouani, K.; Amirat, Y. Toward Lower Limbs Functional Rehabilitation Through a Knee-Joint Exoskeleton. IEEE Trans. Control Syst. Technol. 2017, 25, 712–719. [Google Scholar] [CrossRef]
  8. Lim, D.-H.; Kim, W.-S.; Kim, H.-J.; Han, C.-S. Development of real-time gait phase detection system for a lower extremity exoskeleton robot. Int. J. Precis. Eng. Manuf. 2017, 18, 681–687. [Google Scholar] [CrossRef]
  9. Liu, Z.J.; Lin, W.; Geng, Y.L.; Yang, P. Intent Pattern Recognition of Lower-limb Motion Based on Mechanical Sensors. IEEE-CAA J. Autom. Sin. 2017, 4, 651–660. [Google Scholar] [CrossRef]
  10. Nguyen, N.D.; Bui, D.T.; Truong, P.H.; Jeong, G.M. Classification of Five Ambulatory Activities Regarding Stair and Incline Walking Using Smart Shoes. IEEE Sens. J. 2018, 18, 5422–5428. [Google Scholar] [CrossRef]
  11. Hsu, Y.L.; Yang, S.C.; Chang, H.C.; Lai, H.C. Human Daily and Sport Activity Recognition Using a Wearable inertial Sensor Network. IEEE Access 2018, 6, 31715–31728. [Google Scholar] [CrossRef]
  12. Wu, X.; Ma, Y.; Yong, X.; Wang, C.; He, Y.; Li, N. Locomotion Mode Identification and Gait Phase Estimation for Exoskeletons During Continuous Multilocomotion Tasks. IEEE Trans. Cogn. Dev. Syst. 2021, 13, 45–56. [Google Scholar] [CrossRef]
  13. Qian, Y.P.; Wang, Y.N.; Chen, C.H.; Xiong, J.F.; Leng, Y.Q.; Yu, H.Y.; Fu, C.L. Predictive Locomotion Mode Recognition and Accurate Gait Phase Estimation for Hip Exoskeleton on Various Terrains. IEEE Robot. Autom. Lett. 2022, 7, 6439–6446. [Google Scholar] [CrossRef]
  14. Wei, P.N.; Zhang, J.H.; Tian, F.F.; Hong, J. A comparison of neural networks algorithms for EEG and sEMG features based gait phases recognition. Biomed. Signal Proc. Control 2021, 68, 102587. [Google Scholar] [CrossRef]
  15. Qin, T.; Yang, Y.; Wen, B.; Chen, Z.X.; Bao, Z.; Dong, H.; Dou, K.; Yang, C.M. Research on human gait prediction and recognition algorithm of lower limb-assisted exoskeleton robot. Intell. Serv. Robot. 2021, 14, 445–457. [Google Scholar] [CrossRef]
  16. Yun, Y.; Kim, H.C.; Shin, S.Y.; Lee, J.; Deshpande, A.D.; Kim, C. Statistical method for prediction of gait kinematics with Gaussian process regression. J. Biomech. 2014, 47, 186–192. [Google Scholar] [CrossRef] [PubMed]
  17. Zeng, D.Z.; Qu, C.X.; Ma, T.; Qu, S.G.; Yin, P.; Zhao, N.; Xia, Y.M. Research on a gait detection system and recognition algorithm for lower limb exoskeleton robot. J. Braz. Soc. Mech. Sci. Eng. 2021, 43, 298. [Google Scholar] [CrossRef]
  18. Lee, T.H.; Kim, I.; Lee, S.H. Estimation of the Continuous Walking Angle of Knee and Ankle (Talocrural Joint, Subtalar Joint) of a Lower-Limb Exoskeleton Robot Using a Neural Network. Sensors 2021, 21, 2807. [Google Scholar] [CrossRef]
  19. Qiu, J.; Liu, H.X. Gait Recognition for Human-Exoskeleton System in Locomotion Based on Ensemble Empirical Mode Decomposition. Math. Probl. Eng. 2021, 2021, 5039285. [Google Scholar] [CrossRef]
  20. Lu, Y.Z.; Wang, H.; Hu, F.; Zhou, B.; Xi, H.L. Effective recognition of human lower limb jump locomotion phases based on multi-sensor information fusion and machine learning. Med. Biol. Eng. Comput. 2021, 59, 883–899. [Google Scholar] [CrossRef]
  21. Lu, Y.Z.; Wang, H.; Qi, Y.Y.; Xi, H.L. Evaluation of classification performance in human lower limb jump phases of signal correlation information and LSTM models. Biomed. Signal Proc. Control 2021, 64, 102279. [Google Scholar] [CrossRef]
  22. Huang, L.P.; Zheng, J.B.; Hu, H.C. Online Gait Phase Detection in Complex Environment Based on Distance and Multi-Sensors Information Fusion Using Inertial Measurement Units. Int. J. Soc. Robot. 2022, 14, 413–428. [Google Scholar] [CrossRef]
  23. Yunas, S.U.; Ozanyan, K.B. Gait Activity Classification Using Multi-Modality Sensor Fusion: A Deep Learning Approach. IEEE Sens. J. 2021, 21, 16870–16879. [Google Scholar] [CrossRef]
  24. Wang, X.J.; Dong, D.P.; Chi, X.K.; Wang, S.P.; Miao, Y.N.; An, M.L.; Gavrilov, A.I. sEMG-based consecutive estimation of human lower limb movement by using multi-branch neural network. Biomed. Signal Proc. Control 2021, 68, 102781. [Google Scholar] [CrossRef]
  25. Kang, I.; Molinaro, D.D.; Duggal, S.; Chen, Y.; Kunapuli, P.; Young, A.J. Real-Time Gait Phase Estimation for Robotic Hip Exoskeleton Control During Multimodal Locomotion. IEEE Robot. Autom. Lett. 2021, 6, 3491–3497. [Google Scholar] [CrossRef] [PubMed]
  26. Wu, X.; Yuan, Y.; Zhang, X.; Wang, C.; Xu, T.; Tao, D. Gait Phase Classification for a Lower Limb Exoskeleton System Based on a Graph Convolutional Network Model. IEEE Trans. Ind. Electron. 2022, 69, 4999–5008. [Google Scholar] [CrossRef]
  27. Gronley, J.K.; Perry, J. Gait Analysis Techniques—Rancho-Los-Amigos-Hospital Gait-Laboratory. Phys. Ther. 1984, 64, 1831–1838. [Google Scholar] [CrossRef]
  28. Schuster, M.; Paliwal, K.K. Bidirectional recurrent neural networks. IEEE Trans. Signal Proc. 1997, 45, 2673–2681. [Google Scholar] [CrossRef]
Symmetry 15 00163 g001 550
Figure 1. Block diagram of the gait acquisition system.
Figure 1. Block diagram of the gait acquisition system.
Symmetry 15 00163 g001
Symmetry 15 00163 g002 550
Figure 2. Relationship between the foot and ground of the lower limb exoskeleton robot: foot lift (L), foot hang (H), and foot support (S).
Figure 2. Relationship between the foot and ground of the lower limb exoskeleton robot: foot lift (L), foot hang (H), and foot support (S).
Symmetry 15 00163 g002
Symmetry 15 00163 g003 550
Figure 3. Time-varying curve of gait phase at 6 km/h.
Figure 3. Time-varying curve of gait phase at 6 km/h.
Symmetry 15 00163 g003
Symmetry 15 00163 g004 550
Figure 4. Positions of optical markers and inertial measurement units (IMUs).
Figure 4. Positions of optical markers and inertial measurement units (IMUs).
Symmetry 15 00163 g004
Symmetry 15 00163 g005 550
Figure 5. Overall Structure of the Network Model.
Figure 5. Overall Structure of the Network Model.
Symmetry 15 00163 g005
Symmetry 15 00163 g006 550
Figure 6. Structure diagram of long short-term memory (LSTM) cycle unit.
Figure 6. Structure diagram of long short-term memory (LSTM) cycle unit.
Symmetry 15 00163 g006
Symmetry 15 00163 g007 550
Figure 7. Bidirectional long short-term memory (BiLSTM) neural network.
Figure 7. Bidirectional long short-term memory (BiLSTM) neural network.
Symmetry 15 00163 g007
Symmetry 15 00163 g008 550
Figure 8. Classification accuracy under CNN-BiLSTM network model.
Figure 8. Classification accuracy under CNN-BiLSTM network model.
Symmetry 15 00163 g008
Symmetry 15 00163 g009 550
Figure 9. Classification accuracy under LSTM network model.
Figure 9. Classification accuracy under LSTM network model.
Symmetry 15 00163 g009
Symmetry 15 00163 g010 550
Figure 10. Classification accuracy under gated recurrent unit (GRU) network model.
Figure 10. Classification accuracy under gated recurrent unit (GRU) network model.
Symmetry 15 00163 g010
Symmetry 15 00163 g011 550
Figure 11. Sample size of each gait phase.
Figure 11. Sample size of each gait phase.
Symmetry 15 00163 g011
Symmetry 15 00163 g012 550
Figure 12. Classification accuracy at 50 Hz.
Figure 12. Classification accuracy at 50 Hz.
Symmetry 15 00163 g012
Table
Table 1. The specifications of the exoskeleton robot.
Table 1. The specifications of the exoskeleton robot.
ItemValue
Weight5.3 kg
Adjusted scope of the wears165–185 cm
Angle range of hip joint−90°–140°
Angle range of hip joint0°–135°
Angle range of hip joint−75°–75°
Table
Table 2. Subjects’ information.
Table 2. Subjects’ information.
NOHeight (cm)Weight (kg)Gender
116050female
217079male
318088male
417384.3male
517165male
618092male
717558male
Table
Table 3. Formal definitions of output Layers based on a set of gait data.
Table 3. Formal definitions of output Layers based on a set of gait data.
Long- and Short-Term Formal Definition DescriptionFormalized Definition
Input of the gait data is expressed as M × N matrix I I M × N = { i 1 , … , i M }
Output of the CNN layer is expressed as A × B matrix N N A × B = { n 1 , … , n A }
Output of the BiLSTM layer is expressed as V × H matrix L L D × H = { l 1 , … , l D }
Table
Table 4. Accuracy in different subjects.
Table 4. Accuracy in different subjects.
AccuracyCNN-BiLSTMLSTMGRU
191.24%91.08%90.94%
292.32%91.56%91.74%
393.53%93.27%92.79%
493.81%93.68%93.37%
595.09%94.13%94.31%
692.29%92.01%91.96%
792.64%92.27%91.64%
Table
Table 5. Accuracy in different gait phases.
Table 5. Accuracy in different gait phases.
AccuracyCNN-BiLSTMLSTMGRU
LH69%68%70%
LS96%94%95%
HL82%76%82%
HS97%96%96%
SL94%94%92%
SH98%98%98%
SS81%23%17%
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Xia, Y.; Li, J.; Yang, D.; Wei, W. Gait Phase Classification of Lower Limb Exoskeleton Based on a Compound Network Model. Symmetry 2023, 15, 163. https://doi.org/10.3390/sym15010163

AMA Style

Xia Y, Li J, Yang D, Wei W. Gait Phase Classification of Lower Limb Exoskeleton Based on a Compound Network Model. Symmetry. 2023; 15(1):163. https://doi.org/10.3390/sym15010163

Chicago/Turabian Style

Xia, Yuxuan, Jiaqian Li, Dong Yang, and Wei Wei. 2023. "Gait Phase Classification of Lower Limb Exoskeleton Based on a Compound Network Model" Symmetry 15, no. 1: 163. https://doi.org/10.3390/sym15010163

APA Style

Xia, Y., Li, J., Yang, D., & Wei, W. (2023). Gait Phase Classification of Lower Limb Exoskeleton Based on a Compound Network Model. Symmetry, 15(1), 163. https://doi.org/10.3390/sym15010163

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop