Application of Gradient Optimization Methods in Defining Neural Dynamics

Recurrent neural networks (RNNs) are an important class of algorithms for computing matrix (generalized) inverses. These algorithms are used to find the solutions of matrix equations or to minimize certain nonlinear matrix functions. RNNs are divided into two subgroups: Gradient Neural Networks (GNNs) and Zhang Neural Networks (ZNNs). The GNN design is explicit and mostly applicable to time-invariant problems, which means that the coefficients of the equations that are addressed are constant matrices. ZNN models can be implicit and are able to solve time-varying problems, where the coefficients of the equations depend on the variable $t\in \mathbb{R},t>0,$ representing time [1,2,3].

The Moore–Penrose inverse of $A\in {\mathrm{R}}^{p\times n}$ is the unique matrix $A^{†}=X\in {\mathrm{R}}^{n\times p}$ which is the solution to the well-known Penrose equations [4,5]: where ${\left(\right)}^{\mathrm{T}}$ denotes the transpose matrix. The rank of a matrix A, i.e., the maximum number of linearly independent columns in A, is denoted by $\mathrm{rank}\left(A\right)$.

Applications of linear algebra tools and generalized inverses can be found in important areas such as the modeling of electrical circuits [6], the estimation of DNA sequences [7] and the balancing of chemical equations [8,9], as well as in other important research domains related to robotics [10] and statistics [11]. A number of iterative methods for solving matrix equations based on gradient values have been proposed [12,13,14,15].

In the following sections, we will focus on GNN and ZNN dynamical systems based on the gradient of the objective function and their implementation. The main goal of this research is the analysis of convergence and the study of analytic solutions.

Models with GNN neural designs for computing the inverse or the Moore–Penrose inverse and linear matrix equations were proposed in [16,17,18,19]. Further, various dynamical systems aimed at approximating the pseudo-inverse of rank-deficient matrices were developed in [16]. Wei, in [20], proposed three RNN models for the approximation of the weighted Moore–Penrose inverse. Online matrix inversion in a complex matrix case was considered in [21]. A novel GNN design based on nonlinear activation functions (AFs) was proposed and analyzed in [22,23] for solving the constant Lyapunov matrix equation online. A fast convergent GNN aimed at solving a system of linear equations was proposed and numerically analyzed in [24]. Xiao, in [25], investigated the finite-time convergence of an appropriately accelerated ZNN for the online solution of the time-varying complex matrix equation $A\left(t\right)X\left(t\right)=B\left(t\right)$. A comparison with the corresponding GNN design was considered. Two improved nonlinear GNN dynamical systems for approximating the Moore–Penrose inverse of full-row or full-column rank matrices were proposed and considered in [26]. GNN-type models for solving matrix equations and computing related generalized inverses were developed in [1,3,13,16,18,20,27,28,29]. The acceleration of GNN dynamics to a finite-time convergence has been investigated recently. A finite-time convergent GNN for approximating online solutions of the general linear matrix equation $AX\left(t\right)B+CX\left(t\right)D=B$ was proposed in [30]. This goal was achieved using two activation functions (AFs) in the construction of the GNN. The influence of AFs on the convergence performance of a GNN design for solving the matrix equation $AXB+X=C$ was investigated in [31]. A fixed-time convergent GNN for solving the Sylvester equation was investigated in [32]. Moreover, noise-tolerant GNN models equipped with a suitable activation function (AF) able to solve convex optimization problems were developed in [33].

Our goal is to solve the equation $AXB=D$ and apply its particular cases in computing generalized inverses in real time by improving the GNN model developed in [34]. The developed dynamical system is denoted by GNN$(A,B,D)$. Or motivation is to improve the GNN model denoted by GNN$(A,B,D)$ and develop a novel gradient-based GGNN model, termed GGNN$(A,B,D)$, utilizing a novel type of dynamical system. The proposed GGNN model is based on the standard GNN dynamics along the gradient of the standard error matrix. The convergence analysis reveals the global asymptotic convergence of GGNN$(A,B,D)$ without restrictions, while the output belongs to the set of general solutions to the matrix equation $AXB=D$.

In addition, we propose gradient-based modifications of the hybrid models developed in [35] as proper combinations of GNN and ZNN models for solving the matrix equations $BX=D$ and $XC=D$ with constant coefficients. Analogous hybridizations for approximating the matrix inverse were developed in [36], while two modifications of the ZNN design for computing the Moore–Penrose inverse were proposed in [37]. Hybrid continuous-gradient–Zhang neural dynamics for solving linear time-variant equations were investigated in [38,39]. The developed hybrid GNN-ZNN models in this paper are aimed at solving the matrix equations $AX=B$ and $XC=D$, denoted by $\mathrm{HGZNN}(A,I,B)$ and $\mathrm{HGZNN}(I,C,D)$, respectively.

The implementation was performed in MATLAB Simulink, and numerical experiments were performed with simulations of the GNN, GGNN and HGZNN models.

The GNN used to solve the general linear matrix equation $AXB=D$ is defined over the error matrix $E\left(t\right)=D-AV\left(t\right)B$, where $t\in [0,+\infty )$ is time, and $V\left(t\right)$ is an unknown state-variable matrix that approximates the unknown matrix X in $AXB=D$. The goal function is $\epsilon \left(t\right)=\left|\right|D-{AV\left(t\right)B\left|\right|}_F^2/2$, where ${\parallel ·\parallel }_F=\sqrt{{\displaystyle \sum _{ij}}{a}_{ij}^2}$ denotes the Frobenius norm of a matrix. The gradient of $\epsilon \left(t\right)$ is equal to The GNN evolutionary design is defined by the dynamic system where $\gamma >0$ is a real parameter used to speed up the convergence, and $\dot{V}\left(t\right)$ denotes the time derivative of $V\left(t\right)$. Thus, the linear GNN aimed at solving $AXB=D$ is given by the following dynamics:The dynamical flow (2) is denoted as GNN$(A,B,D)$. The nonlinear GNN$(A,B,D)$ for solving $AXB=D$ is defined by The function array $\mathcal{F}\left(C\right)=\mathcal{F}\left(\left[c_{ij}\right]\right)$ is based on the appropriate odd and monotonically increasing activation function, which is applicable to the elements of a real matrix $C=\left(c_{ij}\right)\in {\mathbb{R}}^{m\times n}$, i.e., $\mathcal{F}\left(C\right)=\left[f\left(c_{ij}\right)\right],i=1,\dots ,m,j=1,\dots ,n,$.

Proposition 1 restates restrictions on the solvability of $AXB=D$ and its general solution.

The following results from [34] describe the conditions of convergence and the limit of the unknown matrix $V\left(t\right)$ from (3) as $t\to +\infty$.

The research in [40] investigated various ZNN models based on optimization methods. The goal of the current research is to develop a GNN model based on the gradient $E_G\left(t\right)$ of ${\parallel E\left(t\right)\parallel }_F^2$ instead of the original goal function $E\left(t\right)$.

The obtained results are summarized as follows:

The overall organization of this paper is as follows. The motivation and derivation of the GGNN and GZNN models are presented in Section 2. Section 3 is dedicated to the convergence analysis of GGNN dynamics. A numerical comparison of GNN and GGNN dynamics is given in Section 4. Neural dynamics based on the hybridization of GGNN and GZNN models for solving matrix equations are considered in Section 6. Numerical examples of hybrid models are analyzed in Section 6. Finally, the last section presents some concluding remarks and a vision of further research.

The standard GNN design (2) solves the GLME $AXB=D$ under constraint (4). Our goal is to resolve this restriction and propose dynamic evolutions based on error functions that tend to zero without restrictions.

Our goal is to define the GNN design for solving the GLME $AXB=D$ based on the error function According to known results from nonlinear unconstrained optimization [41], the equilibrium points of (7) satisfy

We continue the investigation from [40]. More precisely, we develop the GNN model based on the error function $E_G\left(t\right)$ instead of the error function $E\left(t\right)$. In this way, new neural dynamics are aimed at forcing the gradient $E_G$ to zero instead of the standard goal function $E\left(t\right)$. It is reasonable to call such an RNN model a gradient-based GNN (abbreviated GGNN).

Proposition 3 gives the conditions for the solvability of the matrix equations $E\left(t\right)=0$ and $E_G\left(t\right)=0$ and the general solutions to these systems.

In this way, the matrix equation $E\left(t\right)=0$ is solvable under condition (4), while the equation $E_G\left(t\right)=0$ is always consistent. In addition, the general solutions to equations $E\left(t\right)=0$ and $E_G\left(t\right)=0$ are identical [40].

The next step is to define the GGNN dynamics using the error matrix $E_G\left(t\right)$. Let us define the objective function ${\epsilon }_G=\left|\right|E_G{\left|\right|}_F^2/2$, whose gradient is equal to The dynamical system for the GGNN formula is obtained by applying the GNN evolution along the gradient of ${\epsilon }_G\left(V\left(t\right)\right)$ based on $E_G\left(t\right)$, as follows:

The nonlinear GGNN dynamics are defined as in which $\mathcal{F}\left(C\right)=\mathcal{F}\left(\left[c_{ij}\right]\right)$ denotes the elementwise application of an odd and monotonically increasing function $f(·)$, as  mentioned in the previous section for the GNN model (3). Model (10) is termed GGNN$(A,B,D)$. Three activation functions $f(·)$ are used in numerical experiments:

Figure 1 represents the Simulink implementation of GGNN$(A,B,D)$ dynamics (10).

On the other hand, the GZNN model, defined using the ZNN dynamics on the Zhangian matrix $E_G\left(t\right)$, is defined in [40] by the general evolutionary design

In this section, we will analyze the convergence properties of the GGNN model given by dynamics (10).

The numerical examples in this section are based on the Simulink implementation of the GGNN formula in Figure 1.

The parameter $\gamma$, initial state $V\left(0\right)$ and parameters $\rho$ and $\varrho$ of the nonlinear activation functions (12) and (13) are entered directly into the model, while matrices A, B and D are defined from the workspace. It is assumed that $\rho =\varrho =3$ in all examples. The ode15s differential equation solver is used in the configuration parameters. In all examples, $V^{*}$ denotes the theoretical solution.

The blocks powersig, smoothpowersig and transpmult include the codes described in [34,42].

All graphs shown in Figure 5 and Figure 6 confirm the applicability of the proposed GGNN design compared to the traditional GNN design, even if constraint (4) holds.

The numerical results arranged in Table 2 are divided into two parts by a horizontal line. The upper part corresponds to the test matrices of dimensions $\le 10$, while the lower part corresponds to the dimensions $m,n\ge 10$. Considering the first two columns, it is observable from the upper part that the GGNN generates smaller values ${\left|\right|E\left(t\right)\left|\right|}_F$ compared to the GGNN. The values of ${\left|\right|E\left(t\right)\left|\right|}_F$ in the lower part generated by the GNN and GGNN are equal. Considering the third and fourth columns, it is observable from the upper part that the GGNN generates smaller values $\left|\right|E_G\left(t\right){\left|\right|}_F$ compared to the GGNN. On the other hand, the values of $\left|\right|E_G\left(t\right){\left|\right|}_F$ in the lower part, generated by the GGNN, are smaller than the corresponding values generated by the GNN. The last two columns show that the GGNN requires less CPU time compared to the GNN. The general conclusion is that the GGNN model is more efficient in rank-deficient test matrices of larger order $m,n\ge 10$.

The gradient-based error matrix for solving the matrix equation $AX=B$ is defined by

The GZNN design (14) corresponding to the error matrix $E_{A,I,B}$, designated GZNN$(A,I,B)$, is of the form:

Now, the scalar-valued norm-based error function corresponding to $E_{G_{A,I,B}}\left(t\right)$ is given by The following dynamic state equation can be derived using the GGNN$(A,I,B)$ design formula based on (10):

Further, using a combination of ${\dot{E}}_{G_{A,I,B}}\left(t\right)=A^{\mathrm{T}}A\dot{V}\left(t\right)$ and the GNN dynamics (23), it follows that

The next step is to define the new hybrid model based on the summation of the right-hand sides in (22) and (24), as follows:

The model (25) is derived from the combination of the model GGNN$(A,I,B)$ and the model GZNN$(A,I,B)$. Hence, it is equally justified to use the term Hybrid GGNN (abbreviated HGGNN) and Hybrid GZNN (abbreviated HGZNN) model. But model (25) is implicit, so it is not a type of GGNN dynamics. On the other hand, it is designed for time-invariant matrices, which is not in accordance with the common nature of GZNN models, because usually, the GZNN is used in the time-varying case. A formal comparison of (25) and GZNN$(A,I,B)$ reveals that both these methods possess identical left-hand sides, and the right-hand side of (25) can be derived by multiplying the right-hand side of GZNN$(A,I,B)$ by the term ${\left(A^{\mathrm{T}}A\right)}^2+I$.

Formally, (25) is closer to GZNN dynamics, so we will denote the model (25) by HGZNN$(A,I,B)$, considering that this model is not the exact GZNN neural dynamics and is applicable to time-invariant case. This is the case of the constant coefficient matrices A, I and B. Figure 11 represents the Simulink implementation of HGZNN$(A,I,B)$ dynamics (25).

Now, we will take into account the process of solving the matrix equation $XC=D$. The error matrix for this equation is defined by

The GZNN design (14) corresponding to the error matrix $E_{I,C,D}$, denoted by GZNN$(I,C,D)$, is of the form:

On the other hand, the GGNN design formula (10) produces the following dynamic state equation:

The GGNN model (27) is denoted by GGNN$(I,C,D)$. It implies

A new hybrid model based on the summation of the right-hand sides in (26) and (28) can be proposed as follows:

The Model (29) will be denoted by HGZNN$(I,C,D)$. This is the case with the constant coefficient matrices I, C and D.

For the purposes of the proof of the following results, we will use $ECR\left(\mathcal{M}\right)$ to denote the exponential convergence rate of the model $\mathcal{M}$. With ${\lambda }_{min}\left(K\right)$ and ${\lambda }_{max}\left(K\right)$, we denote the smallest and largest eigenvalues of the matrix K, respectively. Continuing the previous work, we use three types of activation functions $\mathcal{F}$: linear, power-sigmoid and smooth power-sigmoid.

The following theorem determines the equilibrium state of HGZNN$(A,I,B)$ and defines its global exponential convergence.