Adaptive online sequential extreme learning machine for dynamic modeling

Extreme learning machine (ELM) is an emerging machine learning algorithm for training single-hidden-layer feedforward networks (SLFNs). The salient features of ELM are that its hidden layer parameters can be generated randomly, and only the corresponding output weights are determined analytically through the least-square manner, so it is easier to be implemented with faster learning speed and better generalization performance. As the online version of ELM, online sequential ELM (OS-ELM) can deal with the sequentially coming data one by one or chunk by chunk with fixed or varying chunk size. However, OS-ELM cannot function well in dealing with dynamic modeling problems due to the data saturation problem. In order to tackle this issue, in this paper, we propose a novel OS-ELM, named adaptive OS-ELM (AOS-ELM), for enhancing the generalization performance and dynamic tracking capability of OS-ELM for modeling problems in nonstationary environments. The proposed AOS-ELM can efficiently reduce the negative effects of the data saturation problem, in which approximate linear dependence (ALD) and a modified hybrid forgetting mechanism (HFM) are adopted to filter the useless new data and alleviate the impacts of the outdated data, respectively. The performance of AOS-ELM is verified using selected benchmark datasets and a real-world application, i.e., device-free localization (DFL), by comparing it with classic ELM, OS-ELM, FOS-ELM, and DU-OS-ELM. Experimental results demonstrate that AOS-ELM can achieve better performance.

The authors would like to thank the anonymous reviewers for their insightful comments and suggestions.

Authors and Affiliations

1. School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, 100083, China

   Jie Zhang, Yanjiao Li & Wendong Xiao

Corresponding author

Correspondence to Wendong Xiao.

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Communicated by V. Loia.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A

Proof of Theorem 1

Assuming that from time q to time \(q+r-1\), there are input vector \({X_q}\) and output vector \({Y_q}\) of the r samples, where

The corresponding output weights \({\beta _{q + r - 1}}\) can be obtained by

where \({H_{q + r - 1}}\) is the hidden layer output matrix of the r samples from time q to time \(q+r\)-1:

Let \({P_{q + r - 1}} = K_{q + r - 1}^{ - 1}\), the solution of \({\beta _{q + r - 1}}\) is

where \({K_{q + r - 1}} = H_{q + r - 1}^T{H_{q + r - 1}}\).

At time \(q+r\), when a data pair \(({x_{q + r}},{y_{q + r}})\) comes, \({H_{q + r - 1}}\) becomes

The corresponding output weights \({\beta _{q + r}}\) can be obtained by

where \({Y_{q + r}} = [{y_q},{y_{q + 1}},\ldots ,{y_{q + r - 1}},{y_{q + r}}]\).

The solution of \({\beta _{q + r}}\) is

where

Then,

Accordingly, we have

Because \({P_{q + r - 1}} = {(H_{q + r - 1}^T{H_{q + r - 1}})^{ - 1}}\) and \({P_{q + r}} = {P_{q + r}} + h(q + r){h^T}(q + r)\), \({P_{q + r - 1}}\) and \({P_{q + r}}\) are positive definite matrices. Thus, we have

According to (40), we have the following relationship:

To sum up, extremely, OS-ELM will lose its correction capability. \(\square \)

Appendix B

Detailed derivation of ALD:

Let \({X_{{N_0}}} = [{x_1},{x_2},\ldots ,{x_{{N_0}}}]\), and the mean of each variable of the initial training data \({\aleph _0} = \{ ({x_i},{y_i})\} _{i = 1}^{{N_0}}\) is

where \({\mathbf{{1}}_{{N_0}}} = {[1,1,\ldots ,1]^T}\). The data scaled to zero mean and unit variance can be represented as

where \(\sum \nolimits _{{N_0}}^{} { = diag({\sigma _{{N_0}1}},{\sigma _{{N_0}2}},\ldots ,{\sigma _{{N_0}w}})} \), and \({\sigma _{{N_0}w}}\) stands for the standard deviation of the wth datum.

When the new datum \({x_{k + 1}}\) arrives, the corresponding mean and the standard deviation can be calculated by

Let \({x_{{N_0} + 1}}\) be scaled with

where \(\sum \nolimits _{{N_0} + 1}^{} { = diag({\sigma _{({N_0} + 1)1}},{\sigma _{({N_0} + 1)2}},\ldots ,{\sigma _{({N_0} + 1)w}})} \).

Expanding the ALD shown in (18), we have

where \({J_{{N_0}}} = {\tilde{X}_{{N_0}}} \cdot \tilde{X}_{{N_0}}^T\), \({j_{{N_0}}} = {\tilde{X}_{{N_0}}} \cdot \tilde{x}_{{N_0} + 1}^T\), \({\tilde{j}_{{N_0} + 1}} = {\tilde{x}_{{N_0} + 1}} \cdot \tilde{x}_{{N_0} + 1}^T\).

In order to minimize \({\delta _{k + 1}}\), we have

Substituting (48) into (47), we can obtain the recursive ALD:

Thus, we have

where \(\xi \) is the error.

Reprints and permissions