Optimal Tracking Control of a Nonlinear Multiagent System Using Q-Learning via Event-Triggered Reinforcement Learning

Abstract

This article offers an optimal control tracking method using an event-triggered technique and the internal reinforcement Q-learning (IrQL) algorithm to address the tracking control issue of unknown nonlinear systems with multiple agents (MASs). Relying on the internal reinforcement reward (IRR) formula, a Q-learning function is calculated, and then the iteration IRQL method is developed. In contrast to mechanisms triggered by time, an event-triggered algorithm reduces the rate of transmission and computational load, since the controller may only be upgraded when the predetermined triggering circumstances are met. In addition, in order to implement the suggested system, a neutral reinforce-critic-actor (RCA) network structure is created that may assess the indices of performance and online learning of the event-triggering mechanism. This strategy is intended to be data-driven without having in-depth knowledge of system dynamics. We must develop the event-triggered weight tuning rule, which only modifies the parameters of the actor neutral network (ANN) in response to triggering cases. In addition, a Lyapunov-based convergence study of the reinforce-critic-actor neutral network (NN) is presented. Lastly, an example demonstrates the accessibility and efficiency of the suggested approach.

1. Introduction

Recently, distributed coordination control of MASs has received a great deal of attention as a result of its extensive applications in power systems [1,2], multi-vehicle [3] and multi-area power systems [4], and other fields. MASs have a variety problems, such as consensus control [5,6,7], synchronization control [8,9], anti-synchronization control [10], and tracking control [11]. Reinforcement learning (RL) [12] and adaptive dynamic programming (ADP) methods [13,14] have been employed by researchers as a means of solving the optimal control problems. Due to its excellent ability for global approximation, neural networks are excellent for dealing with nonlinearities and uncertainties [15]. ADP has great online learning and adaptive ability when it uses neural networks. Furthermore, researchers used RL/ADP algorithms to settle optimal coordination control matters, proposed a lot of directions, tracked control [16,17,18,19], graphical games [19], consensus control [20], containment control [21] and formation control [22]. The controller is designed in the above ways were relying on traditional time-triggered methods. Event-triggered in [23,24], it was suggested that the traditional implementation be changed to an event-triggered one. Because of the increasing number of agents, MASs are required to resolve many computing costs related to the exchange of information. Traditionally, the controller or actuator is constantly updated over a fixed period while the system is in operation. In order to minimize computation and preserve resources, aperiodic sampling is employed in the method of triggering events to improve the controller’s computation efficiency. There have been a number of developments in methods that are based on events for addressing discrete time systems [24]. The traditional implementation was suggested to be replaced by one that is triggered by events.

With an increase in the number of agents, MASs must solve a large number of computing costs related to information exchange. Traditionally, the controller or actuator is constantly updated frequently using a predetermined period of sampling during system operation. To lessen the computational and save resources, aperiodic sampling is used in the event-triggering scheme to improve the associated controller’s computational efficiency. Researchers have developed some event-based methods to address discrete time systems [25] as well as systems based on continuous time [26,27]. Several algorithms based on triggered events have been designed to solve discrete-time systems [25], as well as systems that operate in continuous time [26,27]. According to these results, the system dynamics are assumed to be accurate ahead of time. However, it is not always possible to understand dynamics properly in practice. According to [24], a controller that was triggered by events was proposed which was designed with inaccurate or unknown dynamics for the system.

The application of Q-learning to process control [28], chemical process control, industrial process automatic control, and other areas was an early application of reinforcement learning (RL). The Q-learning algorithm provides a modeless data-driven method for solving control problems. A key point to keep in mind is all potential actions in the present state. Q-learning is currently used primarily for routing optimization and reception processing in network communication within the context of network management. The Q-learning algorithm supports a modeless data-driven method for solving control problems. It is important to note that all potential actions in the present state [29] are evaluated in the Q-learning method, relying on the Q-function. At present, Q-learning is used primarily for routing optimization and reception processing in network communication in the domain of network management [30]. As a result of AlphaGo’s emergence, dynamic research has been conducted in the field of game theory, and tracking control research has been conducted on issues associated with nonlinear MAS tracking control based on Q-learning, such as in [31]. At present, there is some research for tracking control issues for nonlinear MASs based on Q-learning, such as in [32].

The MAS’s issue of optimal control was solved using the RL/ADP method, as mentioned above. The majority of the above results share two common features. First, the direct use of the immediate or immediate reward (IR) signal to define each agent’s performance index function results in limited learning opportunities. As a second step, a state’s value function is used to determine the Hamilton–Jacobi–Bellman (HJB) equation. The corresponding controller is designed using RL/ADP, which results in efficient learning of the MAS equation. It is beneficial to provide each agent with more information signals in a wide range of realistic applications in order to enhance their learning capabilities. In addition to merely considering performance in terms of status, performance can also be viewed from a broader perspective. The purpose of our research is to avoid the limitations described above.

Taking into consideration the aforementioned findings, this work investigates an ideal solution to the optimum control issue for MASs with unknown nonlinearity to enhance the process of learning as well as the effectiveness of control systems. Utilizing the graph theory, a coordination control problem is first identified. According to the gathered information of the IR, increased reinforcement reward (IRR) signals are provided for a longer-term reward period. Based on the IRR function, a Q-function is then developed to assess the efficacy of each agent’s control system. In addition, a tracking control technique is developed using iterative IrQL to derive the HJB equation for each agent. Then, based on the IrQL technique, triggering mechanisms are employed to establish a tracking control system. Finally, an optimum event-triggered controller based on a network topology of reinforce-actor-critic is created. The event triggering mechanism in a closed-loop approach guarantees that the network weights converge and the system remains stable. In light of the findings of this study, an additional contribution has been made to the literature:

(1) With respect to nonlinear MAS tracking control, the authors of [32] proposed an IrQL framework, which differs from [18,33,34], and the design of a new long-term IRR signal is completed. This product was designed on the basis of the data of neighbors to provide more information to the agent. The IRR function is used to define a Q-function, and an iterative IrQL method is proposed for obtaining control schemes that are optimally distributed.

(2) It is designed to trigger a new condition and cite in an asynchronous and distributed manner [24]. As a result, each agent triggers at its own time. Consequently, there is no need to update the controller on a regular basis. For the purpose of achieving online learning, a reinforce-actor-critic neural network based on triggered events is established to determine the optimal control scheme for triggered events. When compared with other papers [18,33,35,36], this paper adjusts the weights non-periodically, and the ANN is only adjusted when a trigger is encountered.

(3) In this paper, the objective is to develop the most effective tracking control method using a new triggering mechanism developed using the IrQL method. As far as event-triggered optimal control mechanisms are concerned, the Lyapunov approach is used to determine the rigorous stability assurance of closed-loop multi-agent networks. The designed RCA-NN framework [32] offers an effective means of executing the proposed method online without requiring any knowledge of the dynamics of the system. We made a comparison between the traditional activation method and the IrQL method. According to the simulation results, the designed algorithm is capable of detecting control problems with good tracking performance.

This article is organized as follows. The graph theory and problems of Section 2 provide an overview of some foundations. In Section 3, IrQL-based HJB equations are obtained. As described in Section 4, the most appropriate controller design should be triggered by an event to build the proposed algorithm. Section 5 develops the RCA-NN. The use of Lyapunov technology leads to convergence of weights in the neural networks. Through analogy examples and comparisons, its effectiveness and correctness of the method are demonstrated in Section 6. The last part includes our final thoughts.

2. Preliminary Findings

2.1. Theoretical Basis of Graphs

It would be possible to model the exchange of information using a directed graph between agents $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{A})$, in which $\mathcal{V}=\left\{{\upsilon }_1,{\upsilon }_2,\dots ,{\upsilon }_n\right\}$ represents N nonempty notes and $\mathcal{E}=\left\{({\upsilon }_i,{\upsilon }_j)\right|{\upsilon }_i,{\upsilon }_j\in \mathcal{V}\}\in \mathcal{V}\times \mathcal{V}$ represents an edge set, indicating agent i could derive the data from agent j. We define $\mathcal{A}=\left[a_{ij}\right]$, which is a matrix that is adjacency relevant and does not contain negative elements $a_{ij}$, where $a_{ij}>0$ is satisfied if $(i,j)\in \mathcal{E}$. Otherwise, $a_{ij}=0$. ${\mathcal{N}}_i=\left\{j\right|(i,j)\in \mathcal{E}\}$ is defined as the set of nodes that are neighbors with node i, and $a_{ij}>0$ is satisfied for each $j\in {\mathcal{N}}_i$. We denote the input matrix $\mathcal{D}=diag\left\{d_i\right\}$, where $d_i={\sum }_{j\in \mathcal{N}}{\mathcal{A}}_{ij}$. The Laplacian matrix is then defined as $L=\mathcal{D}-\mathcal{A}\in {\mathbb{R}}^{N\times N}.$

A leader’s relationship with its followers is the subject of this article. In order to describe follower-leader interactions, we propose an enhanced directed graph model, (i.e., $\widehat{\mathcal{G}}=(\widehat{\mathcal{V}},\widehat{\mathcal{E}})$, in which $\widehat{\mathcal{V}}=\{0,1,2,\dots ,N\}$ and $\widehat{\mathcal{E}}\in \widehat{\mathcal{V}}\times \widehat{\mathcal{V}}$). A leader’s communication with his or her followers is determined by $b_i$. If $b_i>0$, then there is an assumption that the leader and followers are in communication. Otherwise, $b_i=0$. $\mathcal{B}=diag\{b_1,\dots ,b_n\}\in {\mathbb{R}}^{N\times N}$ is defined as the matrix of related connections.

2.2. Problem Formulation

If a nonlinear MAS has one leader as well as N followers, then the dynamics for the ith follower would be as follows:

$$x_i(k+1)=A{x}_i\left(k\right)+B_i{u}_i\left(k\right)$$

(1)

In this case, $x_i\in {\mathbb{R}}^N$ represents the system state, $u_i\in {\mathbb{R}}^{p_i}$ represents the control input, and $A\in {\mathbb{R}}^{n\times n},B_i\in {\mathbb{R}}^{n\times n}$ represent unknown matrices for the plants and inputs.

The leader is written as follows:

$$x_0(k+1)=A{x}_0\left(k\right)$$

(2)

It is assumed that $x_0\in {\mathbb{R}}^n$ represents the leader state.

Assumption 1.

If there is a spanning tree with a leader, then $\widehat{\mathcal{G}}$ has a network of communication interactions, and $\widehat{\mathcal{G}}$ does not contain repeated edges.

Definition 1.

As a result of our design, we are able to develop a control scheme $u_i\left(k\right)$ that only requires agent information. Therefore, the followers can keep track of the leader. In the event that the funder’s conditions are met, we will be able to implement a perfect control scheme [32]:

$${\displaystyle \underset{k\to \infty }{lim}\Vert x_i\left(k\right)-x_0\left(k\right)\Vert =0,i=1,2,\dots ,n}$$

(3)

The MAS’s local consensus error is expressed as follows:

$$e_i\left(k\right)=\sum _{j\in {\mathcal{N}}_i}^{}{a}_{ij}(x_i\left(k\right)-x_j\left(k\right))+b_i(x_i\left(k\right)-x_0\left(k\right))$$

(4)

Then, an overview of the error vector is presented as follows:

$$e\left(k\right)=((L+B)⨂I_n)(x\left(k\right)-\widehat{x_0}\left(k\right))$$

(5)

$e\left(k\right)={({e_1}^T\left(k\right),{e_2}^T\left(k\right),\dots ,{e_n}^T\left(k\right))}^T\in {\mathbb{R}}^{nN}$, $x\left(k\right)={({x_1}^T\left(k\right),{x_2}^T\left(k\right),\dots ,{x_n}^T\left(k\right))}^T\in {\mathbb{R}}^{nN}$, $\widehat{x_0}\left(k\right)=I_n⨂x_0\in {\mathbb{R}}^{nN}$, as well as vector $I_n$ having n dimensions.

The tracking error is written as ${\zeta }_i\left(k\right)=x_i\left(k\right)-x_0\left(k\right)$, which has the vector form

$$\zeta \left(k\right)=x\left(k\right)-\widehat{x_0}\left(k\right)$$

(6)

In this equation, $\zeta \left(k\right)={({{\zeta }_1}^T\left(k\right),{{\zeta }_2}^T\left(k\right),\dots ,{{\zeta }_n}^T\left(k\right))}^T\in {\mathbb{R}}^{nN},$$\widehat{x_0}\left(k\right)={({x_0}^T\left(k\right),{x_0}^T\left(k\right),\dots ,{x_0}^T\left(k\right))}^T$.

Consequently, the localized neighbor error $e_i\left(k\right)$ is represented in the following manner, in agreement with Equations (1) and (4):

$$\begin{array}{cc}\hfill e_i(k+1)=& A{e}_i\left(k\right)+(d_i+b_i)B_i{u}_i\left(k\right)\hfill \\ & -\sum _{j\in {\mathcal{N}}_i}^{}{a}_{ij}{B}_j{u}_j\left(k\right)\hfill \\ \hfill =& {\mathcal{F}}_i(e_i\left(k\right),u_i\left(k\right))\hfill \end{array}$$

(7)

Given Equations (5) and (6), it is evident that $e\left(k\right)$ and $\zeta \left(k\right)$ are related as follows: ${\displaystyle \underset{k\to \infty }{lim}\Vert e\left(k\right)\Vert =0}$ as ${\displaystyle \underset{k\to \infty }{lim}\Vert \zeta \left(k\right)\Vert =0}$. Consequently, when the localized neighboring error is close to zero, the control problem is resolved.

3. Design of the IrQL Method

To resolve the issue of tracking control in systems with multiple agents, the authors of [32] developed the IrQL method. What is important is that in order to provide agents with a greater level of local information from other agents or environments, it is necessary to introduce IRR information, thereby improving control and learning efficiency. In addition, agents have been defined according to the Q-function, and the relevant HJB equation is acquired using the IrQL method.

As an example, consider the following IR function for the ith agent:

$$\begin{array}{cc}\hfill j_i(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))=& e_i{\left(k\right)}^T{R}_{ii}{e}_i\left(k\right)+u_i{\left(k\right)}^T{Q}_{ii}{u}_i\left(k\right)\hfill \\ & +\sum _{j\in N_i}^{}{u}_j{\left(k\right)}^T{Q}_{ij}{u}_j\left(k\right)\hfill \end{array}$$

(8)

In this case, we can represent the agent’s neighbors’ input with $u_{-i}=\left\{u_j\right|j\in {\mathcal{N}}_i\}$. The weight matrices $R_{ii}>0,Q_{ii}>0$, and $Q_{ij}>0$ are positive.

According to the IR function, as a function of IRR, the following is expressed:

$$R_i(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))=\sum _{s=k}^{\infty }{r}^{s-k}{j}_i(e_i\left(s\right),u_i\left(s\right),u_{-i}\left(s\right))$$

(9)

where the IRR function is defined as $r\in (0,1]$ and r is its discount factor.

The following performance indices must be minimized for every agent to find a solution to the issue of controlling tracking optimally:

$$J_i(e_i\left(0\right),u_i\left(0\right),u_{-i}\left(0\right))=\sum _{t=0}^{\infty }{\beta }^t{R}_i(e_i\left(t\right),u_i\left(t\right),u_{-i}\left(t\right))$$

(10)

In this case, its performance index discount factor is $\beta \in (0,1]$.

Remark 1.

The function of the designed IRR function incorporates accumulated prospective long-term reward data from the IR function. The performance factor is measured depending on IRR as opposed to IR, which is contrary to the majority of methods. The advantage is that we can enhance the control actions, and the learning process can be accelerated by using a great deal of data.

Remark 2.

Intrinsic motivation (IM) provides a possible method for enhancing the faculty of abstract actions or solving the difficulties associated with exploring the environment in its reinforcement learning direction. IRR acts as a driving agent that learns skills through intrinsic motivation [32].

Definition 2.

In order to resolve the MAS’s tracking control issue, we propose a distributed tracking control scheme. As the time step k approaches infinity, $e_i\left(k\right)⟶0$ minimizes the performance metrics $\left(10\right)$ simultaneously.

We can obtain a state value function as follows based on the control method of the agent as well as the neighbors $u_i\left(t\right)$ and $u_{-i}\left(t\right)$:

$$V_i\left(e_i\left(k\right)\right)=\sum _{t=k}^{\infty }{\beta }^{t-k}{R}_i(e_i\left(t\right),u_i\left(t\right),u_{-i}\left(t\right))$$

(11)

Equation (11) can also be expressed as the following formula:

$$V_i(e_i\left(k\right)=R_i(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))+\beta V_i\left(e_i(k+1)\right)$$

(12)

Based on the theory, the ideal state value function meets the following conditions:

$$\begin{array}{c}\hfill V_i^{\ast }\left(e_i\left(k\right)\right)=\underset{u_i\left(k\right)}{min}\left\{R_i(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))+\beta V_i^{\ast }\left(e_i(k+1)\right)\right\}\end{array}$$

(13)

In this case, in Bellman form, the function of IRR is expressed as

$$\begin{array}{cc}\hfill R_i(e_i& \left(k\right),u_i\left(k\right),u_{-i}\left(k\right))\hfill \\ \hfill =& j_i(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))\hfill \\ & +\varrho R_i(e_i(k+1),u_i(k+1),u_{-i}(k+1))\hfill \end{array}$$

(14)

On the basis of the condition of stationarity, (i.e., $\frac{\partial V_i^{\ast }\left(e_i\left(k\right)\right)}{\partial u_i\left(k\right)}$), the description of the optimal distributed control method is given below:

$$\begin{array}{cc}\hfill u_i^{\ast }\left(k\right)& =arg\underset{u_i\left(k\right)}{min}\left\{R_i(e_i\left(k\right),u_i\left(k\right),u_{N_i}\left(k\right))+\beta V_i^{\ast }\left(e_i(k+1)\right)\right\}\hfill \\ & =-\frac{1}{2}\beta (d_i+b_i)Q_{ii}^{-1}{h}_i^T\left(x_i\left(k\right)\right)▽V_i^{\ast }\left(e_i(k+1)\right)\hfill \end{array}$$

(15)

In this equation, $▽V_i^{\ast }\left(e_i(k+1)\right)=\frac{\partial V_i^{\ast }\left(e_i(k+1)\right)}{\partial e_i(k+1)}$.

Remark 3.

As is well known, the state value algorithm $V_i\left(e_i\left(k\right)\right)$ is highly concerned with the space of states. In accordance with the state action function, the Q-learning method is designed with RL. The Q-function can be used by each agent to estimate the properties of all possible decisions in the current situation, and we can determine what is the best behavior of the agent at each step by using the Q-function.

The Q-function is written as follows:

$$\begin{array}{cc}\hfill Q_i(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))=R_i(e_i\left(k\right),& u_i\left(k\right),u_{-i}\left(k\right))\hfill \\ & +\beta V_i\left(e_i(k+1)\right)\hfill \end{array}$$

(16)

In accordance with the optimal scheme, the optimal Q-function is given by

$$\begin{array}{cc}\hfill Q_i^{\ast }& (e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))\hfill \\ \hfill =& R_i(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))\hfill \\ & +\beta Q_i^{\ast }(e_i(k+1),u_i^{\ast }(k+1),u_{-i}^{\ast }(k+1))\hfill \end{array}$$

(17)

Based on Equations (16) and (17), we can express the optimal solution as follows:

$$u_i^{\ast }\left(k\right)=arg\underset{u_i\left(k\right)}{min}\left\{Q_i^{\ast }(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))\right\}$$

(18)

In comparison with the control method of Equation (15), its optimum Q-function provides the optimal solution for the control scheme here. As a result, we intend to calculate the solution to Equation (17).

4. Designs of the Event-Driven Controller

According to a previous work [18], a time-triggered controller was developed. Nevertheless, a new event-triggering mechanism is designed to minimize computing costs for this case.

$Q_i^{\ast },(e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right))$ is defined as the sequence of trigger times. At the triggering instant, the sampled disagreement error is expressed as ${\widehat{e}}_i^s$.

As a result of the threshold value and error, the triggering time varies. The control scheme can only be updated when $k=k{t}_s^i$ and cannot be updated under any other circumstances:

$$u_i\left(k\right)=u_i\left(k{t}_s^i\right),k\in [k{t}_s^i,k{t}_{s+1}^i)$$

(19)

To design a triggering condition, we propose a function that measures the gap arising from the existing error and the previously sampled error:

$${ϵ}_i^s\left(k\right)={\widehat{e}}_i^s-e_i\left(k\right),k\in [k{t}_s^i,k{t}_{s+1}^i)$$

(20)

We have set the triggering error equal to zero at $k=k{t}_s^i$.

The dynamic expression of localized mistakes based on an event-triggered controlling approach can be written as

$$e_i(k+1)={\mathcal{F}}_i(e_i\left(k\right),u_i\left(k{t}_s^i\right))$$

(21)

Thus, the equation for event-triggered events is obtained:

$$\begin{array}{cc}\hfill V_i^{\ast }& \left(e_i\left(k\right)\right)=\hfill \\ & \underset{u_i\left(k{t}_s^i\right)}{min}\left\{R_i(e_i\left(k\right),u_i\left(k{t}_s^i\right),u_{-i}\left(k{t}_s^i\right))+\beta V_i^{\ast }\left({\mathcal{F}}_i(e_i\left(k\right),u_i\left(k{t}_s^i\right))\right)\right\}\hfill \end{array}$$

(22)

$$\begin{array}{c}{Q}_i^{\ast }(e_i\left(k\right))\hfill \\ =R_i(e_i\left(k\right),u_i\left(k{t}_s^i\right),u_{-i}\left(k{t}_s^i\right))\hfill \\ +\beta Q_i^{\ast }\left({\mathcal{F}}_i(e_i\left(k\right),u_i\left(k{t}_s^i\right))\right)\hfill \end{array}$$

(23)

It is possible to express the optimal tracking control using an event-triggered approach in the following way:

$$u_i^{\ast }\left(k\right)=arg\underset{u_i\left(k{t}_s^i\right)}{min}\left\{Q_i^{\ast }\left(e_i\left(k\right)\right)\right\}$$

(24)

Assumption 2.

There is a constant $\mathcal{L}$ that explains the inequality below:

$$∥{\mathcal{F}}_i(e_i\left(k\right),u_i\left(k{t}_s^i\right))∥\le \mathcal{L}∥e_i\left(k\right)∥+\mathcal{L}∥{ϵ}_i^s\left(k\right)∥$$

(25)

Assumption 3.

There is a triggering condition which is as follows:

$${∥{ϵ}_i^s\left(k\right)∥}^2\le (1-2{\mathcal{L}}^2)/\left(2{\mathcal{L}}^2\right){∥e_i\left(k\right)∥}^2={\pi }_{i}T$$

(26)

where ${\pi }_{i}T$ represents the triggering threshold and $\mathcal{L}\in (0,\sqrt{2}/2)$ [24]. Once the multi-agent system dynamics have stabilized, followers are able to track their leaders.

5. Neural Network Implementation for the Event-Triggered Approach Using the IrQL Method

This section discusses the tree-NN structure, also known as RCA-NNs. Three virtual networks are included in the tree-NN structure.

5.1. Reinforce Neutral Network (RNN) Learning Model

The reinforced NN is employed to approximate the IRR signal as follows:

$$\widehat{R}\left(Z_{ri}\left(k\right)\right)={\phi }_{ri}({\omega }_{r2i}^T\left(k\right)·{\phi }_{ri}({\omega }_{r1i}^T\left(k\right)·Z_{ri}\left(k\right)))$$

(27)

where $Z_{ri}\left(k\right)$ represents the input vector, which has $e_i\left(k\right),$$u_i\left(k\right)$, while $u_{-i}\left(k\right)$. ${\omega }_{r1i}$ represents the matrix of weights for input-to-hidden layering. Meanwhile, ${\omega }_{r2i}$ represents the matrix of weights for hidden-to-output layering, and ${\phi }_{ri}(·)$ represents an activation function [24].

Due to the reinforced NN, the associated error function is as follows:

$$\begin{array}{cc}\hfill e_{ri}\left(k\right)=j_i(e_i(k-1),& u_i(k-1),u_{-i}(k-1))\hfill \\ & +\varrho {\widehat{R}}_i\left(Z_{ri}\left(k\right)\right)-{\widehat{R}}_i\left(Z_{ri}(k-1)\right)\hfill \end{array}$$

(28)

The loss function is written as

$$E_{ri}\left(k\right)=\frac{1}{2}{e}_{ri}^2\left(k\right)$$

(29)

For convenience’s sake, only the matrices ${\omega }_{r2i}$ are updated, and the matrices ${\omega }_{r1i}$ remain unchanged during the training process.

The RNN’s update law is expressed as

$${\omega }_{r2i}(k+1)={\omega }_{r2i}\left(k\right)-{\alpha }_{ri}·\left(\frac{\partial E_{ri}\left(k\right)}{\partial {\omega }_{r2i}\left(k\right)}\right)$$

(30)

In this equation, ${\alpha }_{ri}$ represents the rate at which the RNN learns.

The gradient descent rule (GDR) is used to obtain an updated law for the reinforced NN’s weight, which yields the following results:

$$\begin{array}{cc}\hfill & {\omega }_{ri}(k+1)\hfill \\ & ={\omega }_{ri}\left(k\right)-{\alpha }_{ri}·\left(\frac{\partial E_{ri}\left(k\right)}{\partial e_{ri}\left(k\right)}·\frac{\partial e_{ri}\left(k\right)}{\partial \widehat{R}\left(Z_{ri}\left(k\right)\right)}·\frac{\partial \widehat{R}\left(Z_{ri}\left(k\right)\right)}{\partial {\omega }_{r2i}\left(k\right)}\right)\hfill \\ & ={\omega }_{r2i}\left(k\right)-{\alpha }_{ri}\varrho e_{ri}\left(k\right)\left[1-{\phi }_{ri}^2\left({\omega }_{r2i}^T\left(k\right)·{\Delta }_{ri}\left(k\right)\right)\right]{\Delta }_{ri}\left(k\right)\hfill \end{array}$$

(31)

In this equation, ${\Delta }_{ri}\left(k\right)={\phi }_{ri}({\omega }_{r1i}^T\left(k\right)·Z_{ri}\left(k\right))$.

5.2. Critic Neutral Network (CNN) Learning Model

In the following section, when designing the critic NN, an attempt is made to achieve a close approximation of the Q-function:

$$\widehat{Q_i}\left(Z_{ci}\left(k\right)\right)={\omega }_{c2i}^T\left(k\right)·{\phi }_{ci}({\omega }_{c1i}^T\left(k\right)·Z_{ci}\left(k\right))$$

(32)

In this equation, $Z_{ci}\left(k\right)$ represents the relative vector of inputs that has ${\widehat{R}}_i\left(k\right)$, $e_i\left(k\right)$, and $u_i\left(k\right)$ as well as $u_{-i}\left(k\right)$, while ${\omega }_{c1i}^T\left(k\right)$ and ${\omega }_{c2i}^T\left(k\right)$ represent the input layer weight matrices and output layer weight matrices.

It is possible to express the function of the error for the CNN to be

$$e_{ci}\left(k\right)={\widehat{R}}_i\left(Z_{ri}(k-1)\right)+\beta {\widehat{Q}}_i\left(Z_{ci}\left(k\right)\right)-{\widehat{Q}}_i\left(Z_{ci}(k-1)\right)$$

(33)

Its function of loss is written to be

$$E_{ci}\left(k\right)=\frac{1}{2}{e}_{ci}^2\left(k\right)$$

(34)

In accordance with the operation of RNNs, only ${\omega }_{c2i}$ is updated, and ${\omega }_{c1i}$ remains unchanged.

With the help of the gradient descent rule (GDR), it can be used to express the weight update law:

$${\omega }_{c2i}(k+1)={\omega }_{c2i}\left(k\right)-{\alpha }_{ci}\left(\frac{\partial E_{ci}\left(k\right)}{\partial {\omega }_{c2i}\left(k\right)}\right)$$

(35)

where ${\alpha }_{ci}$ represents the critic NN’s learning rate. Furthermore, we can obtain its weight update schemes for the critic NN:

$$\begin{array}{cc}\hfill & {\omega }_{c2i}(k+1)\hfill \\ & ={\omega }_{c2i}\left(k\right)-{\alpha }_{ci}\left(\frac{\partial E_{ci}\left(k\right)}{\partial e_{ci}\left(k\right)}·\frac{\partial e_{ci}\left(k\right)}{\partial {\widehat{Q}}_i\left(Z_{ci}\left(k\right)\right)}·\frac{\partial {\widehat{Q}}_i\left(Z_{ci}\left(k\right)\right)}{\partial {\omega }_{c2i}\left(k\right)}\right)\hfill \\ & ={\omega }_{c2i}\left(k\right)-{\alpha }_{ci}\beta \left[{\widehat{R}}_i\left(Z_{ci}(k-1)\right)+\beta {\omega }_{c2i}^T\left(k\right)·{\Delta }_{ci}\left(k\right)\right.\hfill \\ \hfill & \left. -{\omega }_{c2i}^T(k-1)·{\Delta }_{ci}(k-1)\right]·{\Delta }_{ci}\left(k\right)\hfill \end{array}$$

(36)

In this equation, ${\Delta }_{ci}\left(k\right)={\phi }_{ci}\left({\omega }_{ci1}^T\left(k\right)Z_{ci}\left(k\right)\right)$.

5.3. Actor Neutral Network (ANN) Learning Model

Based on the actor NN, an approximate optimal scheme is defined as follows:

$${\widehat{u}}_i\left(k\right)={\omega }_{a2i}^T·{\phi }_{ai}({\omega }_{a1i}^T·Z_{ai}\left(k\right))$$

(37)

where the input data of the ANN is represented by $Z_{ai}\left(k\right)=e_i\left(k\right)$, ${\omega }_{a1i}$ represents the weight matrices of the input layer, and ${\omega }_{a2i}$ represents the weight matrices of the output layer.

Based on the prediction error of the actor NN, the following result is obtained:

$$e_{ai}\left(k\right)={\widehat{Q}}_i\left(Z_{ci}\left(k\right)\right)-{\mathcal{U}}_c$$

(38)

It is possible to express the function of loss of the ANN to be

$$E_{ai}\left(k\right)=\frac{1}{2}{e}_{ai}\left(k\right)$$

(39)

As with RNNs and CNNs, ${\omega }_{a1i}$ must remain unchanged throughout the learning process. The actor NN update laws are defined as follows:

$${\omega }_{a2i}(k+1)={\omega }_{a2i}\left(k\right)-{\alpha }_{ai}\left(\frac{\partial E_{ai}\left(k\right)}{\partial {\omega }_{a2i}\left(k\right)}\right)$$

(40)

where ${\alpha }_{ai}$ represents the ANN learning rate. We can design a weight-tuning scheme for an ANN as follows:

$$\begin{array}{cc}\hfill & {\omega }_{a2i}(k+1)\hfill \\ \hfill & ={\omega }_{a2i}\left(k\right)-{\alpha }_{ai}·\left(\frac{\partial E_{ai}\left(k\right)}{\partial e_{ai}\left(k\right)}·\frac{\partial e_{ai}\left(k\right)}{\partial {\widehat{Q}}_i\left(Z_{ci}\left(k\right)\right)}\right.\hfill \\ \hfill & \left. \times \frac{\partial {\widehat{Q}}_i\left(Z_{ci}\left(k\right)\right)}{\partial {\widehat{u}}_i\left(k\right)}·\frac{\partial {\widehat{u}}_i\left(k\right)}{\partial {\omega }_{a2i}\left(k\right)}\right)\hfill \\ \hfill & ={\omega }_{a2i}\left(k\right)-{\alpha }_{ai}{\Delta }_{ai}\left(k\right){\omega }_{a2i}^T\left(k\right)\hfill \\ \hfill & \times {▽}_{ci}^{{}^{\prime }}\left(k\right){\omega }_{c1i}^T\left(k\right){▽}_{{\widehat{u}}_i}\left(Z_{ci}\left(k\right)\right)\left[{\omega }_{a2i}^T{\Delta }_{ci}\left(k\right)\right]\hfill \end{array}$$

(41)

where ${\Delta }_{ai}\left(k\right)={\phi }_{ai}\left({\omega }_{a1i}^T\left(k\right)Z_{ai}\left(k\right)\right)$, ${▽}_{ci}^{{}^{\prime }}\left(k\right)=\frac{\partial {\phi }_{ci}\left({\omega }_{c1i}\left(k\right)Z_{ci}\left(k\right)\right)}{\partial {\phi }_{ci}\left({\omega }_{c1i}^T\left(k\right)Z_{ci}\left(k\right)\right)}$, ${▽}_{{\widehat{u}}_i}\left(Z_{ci}\left(k\right)\right)=\frac{\partial Z_{ci}\left(k\right)}{\partial {\widehat{u}}_i\left(k\right)}$.

Furthermore, we can obtain

$${\omega }_{a2i}(k+1)=\left\{\begin{array}{c}{\omega }_{a2i}\left(k\right)-{\alpha }_{ai}{\Delta }_{ai}\left(k\right){\omega }_{a2i}^T\left(k\right)\hfill \\ \times {▽}_{ci}^{{}^{\prime }}\left(k\right){\omega }_{c1i}^T\left(k\right){▽}_{{\widehat{u}}_i}\left(Z_{ci}\left(k\right)\right)\left[{\omega }_{a2i}^T{\Delta }_{ci}\left(k\right)\right],k=k{t}_s^i\hfill \\ {\omega }_{a2i}\left(k\right),k\in [k{t}_s^i,k{t}_{s+1}^i).\hfill \end{array}\right.$$

(42)

It is described in detail in Algorithm 1 how the controller is designed using RCA-NNs and event triggering. When the trigger conditions are met, the actor NN is updated.

For analysis of stability based on the Lyapunov method, we present an analysis of stability and convergence in the following section.

Assumption 4.

The following conditions are assumed to be true: $∥{\omega }_{r2i}\left(k\right)∥\le {\omega }_{rim},∥{\omega }_{c2i}\left(k\right)∥$$\le {\omega }_{cim},∥{\omega }_{a2i}\left(k\right)∥\le {\omega }_{aim}$. There are bounded activation functions, i.e., $∥{\Delta }_{ri}\left(k\right)∥\le {\Delta }_{rim},∥{\Delta }_{ci}\left(k\right)∥\le {\Delta }_{cim},∥{\Delta }_{ai}\left(k\right)∥\le {\Delta }_{aim}$. What’s more, the functions of activation ${\phi }_{ai}\left(k\right)$ is the function of Lipschitz that satisfies $∥{\phi }_{ai}\left(e_i\left(k{t}_s^i\right)\right)-{\phi }_{ai}\left(k\right)∥\le {\theta }_{ai}∥e_i\left(k{t}_s^i\right)-e_i\left(k\right)∥={\theta }_{ai}∥{ϵ}_i^s\left(k\right)∥\le {\theta }_{ai}{\pi }_{i}T$, where ${\theta }_{ai}$, ${\pi }_{i}T$ are positive constants. Approximation errors of NNs’ output can be defined to be: ${\delta }_{ci}\left(k\right)={\omega }_{c2i}\left(k\right){\Delta }_{ci}\left(k\right),{\delta }_{ai}\left(k\right)={\omega }_{a2i}\left(k\right){\Delta }_{ai}\left(k\right),{\vartheta }_{ri}\left(k\right)={\omega }_{r2i}\left(k\right){\Delta }_{ri}\left(k\right)$.

Theorem 1.

Assume that Assumptions 1 and 2 are true. CNN and ANN weights are renewed by (36) and (42). Upon satisfying the triggering term(26), the local inconsistency error is $e_i\left(k\right)$, critic evaluated error and actor evaluated error error are consistent and ultimately bounded. Furthermore the control method $u_i$ converges to the optimal value $u_i^{\ast }$.

Evidence: Set ${\tilde{\omega }}_{r2i}\left(k\right)={\omega }_{r2i}\left(k\right)-{\omega }_{r2i}^{\ast }$ as the weighting assessment error between the optimal weights for RNNs ${\omega }_{r2i}^{\ast }$. Its assessment ${\omega }_{r2i}\left(k\right)$, ${\tilde{\omega }}_{c2i}\left(k\right)={\omega }_{c2i}\left(k\right)-{\omega }_{c2i}^{\ast }$ is the error resulting from weighting evaluation involving the ideal CNN weights ${\omega }_{c2i}^{\ast }$; its assessed ${\omega }_{c2i}\left(k\right)$, as well as ${\tilde{\omega }}_{a2i}\left(k\right)={\omega }_{a2i}\left(k\right)-{\omega }_{a2i}^{\ast }$ is the weighting evaluated error involving the ideal ANN weightings ${\omega }_{a2i}^{\ast }$ and its estimation ${\omega }_{a2i}\left(k\right)$.

Algorithm 1 RCA neural networks based on the IrQL method with event triggering.

Set initial value:

1: Set initial values for ${\omega }_{r2i}\left(0\right),{\omega }_{a2i}\left(0\right),{\omega }_{c2i}\left(0\right)$ between $(0,1)$;

2: Set a low level of degree of precision for the calculation $\mathcal{E}$.

3: Initialize the score of $x_i\left(0\right),x_0\left(0\right)$ within $(0,1)$

The iterative process: Make $kisequalto0$. Error calculation at the localized level $e_i\left(k\right)$;

4: Keep on;

5: Based on actor NN, estimate ${\widehat{u}}_i\left(k\right)$ by $\left(37\right)$

6: Update the reinforce NN;

7: Via the inputting $[e_i\left(k\right),u_i\left(k\right),u_{-i}\left(k\right)]$ into the reinforce NN, and we can obtain the

estimated the function of IRR $R_i\left(Z_{ri}\left(k\right)\right)$ via $\left(27\right)$

8: Obtain $e_{ri}\left(k\right)$ by $\left(28\right)$;

9: Renew the matrices ${\omega }_{r2i}\left(k\right)$ by $\left(31\right)$;

10: Renew the critic NN:

11: Via the inputting $[{\widehat{R}}_i\left(Z_{ri}\left(k\right)\right),e_i\left(k\right),u_i\left(k\right),and{u}_{-i}\left(k\right)]$ into critic NN,

and we can obtain its estimated Q-function via $\left(32\right)$;

12: Obtain $e_{ci}\left(k\right)$ by $\left(33\right)$;

13: Renew the matrices ${\omega }_{c2i}\left(k\right)$ by $\left(36\right)$;

14: Renew the actor NN:

15: Input $\left[e_i\left(k\right)\right]$ to the actor NN, and we can obtain the estimated Q-function

${\widehat{u}}_i\left(k\right)$ via $\left(37\right)$

16: Calculation $e_{ai}\left(k\right)$ via $\left(38\right)$

17: In the event that the triggering conditions are met, renew the matrices

${\omega }_{a2i}\left(k\right)$ of the actor NN using $\left(41\right)$

18: Otherwise, do not update the weight matrices ${\omega }_{a2i}\left(k\right)$

19: Until $∥{\omega }_{c2i}(k+1)-{\omega }_{c2i}\left(k\right)∥\le \mathcal{E}$; otherwise, set $k=k+1$, then go to

procedure $\left(5\right)$

20: Keep on ${\omega }_{r2i}\left(k\right),{\omega }_{c2i}\left(k\right),{\omega }_{a2i}\left(k\right)$ as the optimal weights.

(1) We can obtain the following function at the time of triggering as follows:

$$L\left(k\right)=L_1\left(k\right)+L_2\left(k\right)+L_3\left(k\right)+L_4\left(k\right)+L_5\left(k\right)$$

(43)

In this equation,

$$\begin{array}{cc}\hfill & L_1\left(k\right)=\frac{1}{{\alpha }_{ri}}tr\left({\omega }_{r2i}^T\left(k\right){\omega }_{r2i}\left(k\right)\right),\hfill \\ \hfill & L_2\left(k\right)=\frac{1}{{\alpha }_{ci}}tr\left({\omega }_{c2i}^T\left(k\right){\omega }_{c2i}\left(k\right)\right),\hfill \\ \hfill & L_3\left(k\right)=\frac{1}{{\alpha }_{ai}}tr\left({\omega }_{a2i}^T\left(k\right){\omega }_{a2i}\left(k\right)\right),\hfill \\ \hfill & L_4\left(k\right)={\varrho }^k{\widehat{R}}_i\left(k\right),\hfill \\ \hfill & L_5\left(k\right)={\beta }^k{\widehat{Q}}_i\left(k\right).\hfill \end{array}$$

(44)

$\Delta L_1\left(k\right)$ is written to be

$$\Delta L_1\left(k\right)=\frac{1}{{\alpha }_{ri}}tr({\omega }_{r2i}^T(k+1){\omega }_{r2i}(k+1)-{\omega }_{r2i}^T\left(k\right){\omega }_{r2i}\left(k\right)).$$

(45)

In this equation, we have

$$\begin{array}{cc}\hfill {\tilde{\omega }}_{r2i}(k+1)& ={\omega }_{r2i}(k+1)-{\omega }_{r2i}^{\ast }\hfill \\ \hfill & ={\omega }_{r2i}\left(k\right)-{\alpha }_{ri}\varrho [j(k-1)+\varrho \widehat{R}\left(k\right)-\widehat{R}(k-1)]\hfill \\ & \times {\delta }_{ri}\left(k\right){\Delta }_{ri}\left(k\right)\hfill \end{array}$$

(46)

Furthermore, we have

$$\begin{array}{cc}\hfill \Delta L_1\left(k\right)=& -2{\varrho }^2{\delta }_{ri}\left(k\right)[{\varrho }^{-1}j\left(k\right)+\widehat{R}\left(k\right)-\widehat{R}(k-1)]\hfill \\ \hfill & +{\alpha }_{ri}{\varrho }^4{[{\varrho }^{-1}j\left(k\right)+\widehat{R}\left(k\right)-\widehat{R}(k-1)]}^2{\vartheta }_{ri}^2\left(k\right)\hfill \\ \hfill =& {\left\{{\delta }_{ri}\left(k\right)-{\varrho }^2[{\varrho }^{-1}j\left(k\right)+\widehat{R}\left(k\right)-\widehat{R}(k-1)]\right\}}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}\left(k\right){\Delta }_{ri}^2\left(k\right)){\varrho }^4{[{\varrho }^{-1}j\left(k\right)+\widehat{R}\left(k\right)-\widehat{R}(k-1)]}^2\hfill \\ \hfill & \times {\vartheta }_{ri}^2\left(k\right)-{\delta }_{ri}^2\left(k\right)\hfill \end{array}$$

(47)

$\Delta L_2\left(k\right)$ can be written as

$$\Delta L_2\left(k\right)=\frac{1}{{\alpha }_{ci}}tr({\omega }_{c2i}^T(k+1){\omega }_{c2i}(k+1)-{\omega }_{c2i}^T\left(k\right){\omega }_{c2i}\left(k\right)).$$

(48)

Within this equation, we have

$$\begin{array}{cc}\hfill {\tilde{\omega }}_{c2i}(k+1)& ={\omega }_{c2i}(k+1)-{\omega }_{c2i}^{\ast }\hfill \\ \hfill & ={\omega }_{c2i}\left(k\right)-{\alpha }_{ci}\beta {\Delta }_{ci}\left(k\right)[{\widehat{R}}_i(k-1)\hfill \\ & +\beta ({\omega }_{c2i}\left(k\right)+{\omega }_{c2i}^{\ast }){\Delta }_{ci}\left(k\right)-{\omega }_{c2i}^T(k-1){\Delta }_{ci}(k-1)]\hfill \end{array}$$

(49)

Furthermore, we have

$$\Delta L_2\left(k\right)=\frac{1}{{\alpha }_{ci}}\left\{D_1+D_2+D_3-{\omega }_{c2i}^T\left(k\right){\omega }_{c2i}\left(k\right)\right\}$$

(50)

where

$$\begin{array}{cc}\hfill D_1& ={\omega }_{c2i}^T\left(k\right){(I-{\alpha }_{ci}{\beta }^2{\Delta }_{ci}\left(k\right){\Delta }_{ci}^T\left(k\right))}^2{\omega }_{c2i}(k\hfill \\ \hfill & ={∥{\omega }_{c2i}\left(k\right)∥}^2-2{\alpha }_{ci}{\beta }^2{∥{\delta }_{ci}\left(k\right)∥}^2\hfill \\ \hfill & +{\alpha }_{ci}^2{\beta }^4{∥{\Delta }_{ci}\left(k\right)∥}^2{∥{\delta }_{ci}\left(k\right)∥}^2\hfill \\ \hfill D_2& =-2{\alpha }_{ci}{\beta }^2{\delta }_{ci}\left(k\right)[{\beta }^{-1}{\widehat{R}}_i(k-1)+{\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)\hfill \\ \hfill & -{\beta }^{-1}{\omega }_{c2i}(k-1){\Delta }_{ci}(k-1)]{∥{\Delta }_{ci}\left(k\right)∥}^2\hfill \\ \hfill D_3& ={\alpha }_{ci}^2{\beta }^4[{\beta }^{-1}{\widehat{R}}_i(k-1)+{\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)\hfill \\ \hfill & -{\beta }^{-1}{\omega }_{c2i}^T(k-1){\Delta }_{ci}{(k-1)]}^T\hfill \\ \hfill & \times [{\beta }^{-1}{\widehat{R}}_i(k-1)+{\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)-{\beta }^{-1}{\omega }_{c2i}^T(k-1){\Delta }_{ci}(k-1)]\hfill \end{array}$$

The following result is obtained by computation:

$$\begin{array}{cc}\hfill \Delta L_2\left(k\right)& =-{\beta }^2{∥{\delta }_{ci}\left(k\right)∥}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2{∥{\Delta }_{ci}\left(k\right)∥}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{\widehat{R}}_i(k-1)+{\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)\hfill \\ \hfill & -{\beta }^{-1}{\omega }_{c2i}^T(k-1){\Delta }_{ci}{(k-1)\left)\right||}^2\hfill \\ \hfill & +\left|\right|{\widehat{R}}_i(k-1)+\beta {\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)\hfill \\ \hfill & -{\omega }_{c2i}^T(k-1){\Delta }_{ci}(k-1){\left|\right|}^2\hfill \end{array}$$

(51)

In the case of the difference of the first order of $L_3\left(k\right)$, we can obtain

$$\Delta L_3\left(k\right)=\frac{1}{{\alpha }_{ai}}({\omega }_{a2i}^T(k+1){\omega }_{a2i}(k+1)-{\omega }_{a2i}^T\left(k\right){\omega }_{a2i}\left(k\right))$$

(52)

where,

$$\begin{array}{cc}\hfill {\tilde{\omega }}_{a2i}(k+1)& ={\omega }_{a2i}(k+1)-{\omega }_{a2i}^{\ast }\hfill \\ \hfill & ={\omega }_{a2i}\left(k\right)-{\alpha }_{ai}{\Delta }_{ai}\left(k\right){\omega }_{c2i}^T\left(k\right)C\left(k\right)\hfill \\ \hfill & \times \left[{\omega }_{c2i}^T\left(k\right){\Delta }_{ci}\left(k\right)\right]\hfill \end{array}$$

(53)

Therefore, we have

$$\begin{array}{c}\hfill \Delta L_3\left(k\right)=\frac{1}{{\alpha }_{ai}}(E1-{\omega }_{a2i}^T\left(k\right){\omega }_{a2i}\left(k\right))\end{array}$$

(54)

where

$$\begin{array}{cc}\hfill E1=& \left|\right|{\omega }_{a2i}\left(k\right){\left|\right|}^2-2{\alpha }_{ai}{\omega }_{c2i}^T\left(k\right)C\left(k\right){\delta }_{ai}\left(k\right)\left[{\omega }_{c2i}^T\left(k\right){\Delta }_{ci}\left(k\right)\right]\hfill \\ \hfill & +{\alpha }_{ai}\left|\right|{\omega }_{c2i}^T\left(k\right){\Delta }_{ci}{\left(k\right)\left|\right|}^2\left|\right|{\Delta }_{ai}{\left(k\right)\left|\right|}^2\left|\right|{\omega }_{c2i}^T\left(k\right)C\left(k\right){\left|\right|}^2\hfill \end{array}$$

(55)

In the case of $\Delta L_3\left(k\right)$, the simplified formula is given below:

$$\begin{array}{cc}\hfill \Delta L_3\left(k\right)=& -(1-{\alpha }_{ai}\left|\right|{\Delta }_{ai}{\left(k\right)\left|\right|}^2\left)\right||{\omega }_{c2i}^T\left(k\right){\Delta }_{ci}\left(k\right){\left|\right|}^2\hfill \\ \hfill & \times \left|\right|{\omega }_{c2i}^T{\left(k\right)C\left(k\right)\left|\right|}^2-\left|\right|{\delta }_{ai}\left(k\right){\left|\right|}^2\hfill \\ \hfill & +\left|\right|{\omega }_{c2i}^T\left(k\right)C\left(k\right){\Delta }_{ai}\left(k\right){\omega }_{c2i}^T\left(k\right)-{\delta }_{ai}\left(k\right){\left|\right|}^2\hfill \end{array}$$

(56)

By adding Equations (47), (51), and (56), we can obtain $L\left(k\right)$ as follows:

$$\begin{array}{cc}\hfill \Delta L\left(k\right)& =\Delta L_1\left(k\right)+\Delta L_2\left(k\right)+\Delta L_3\left(k\right)+\Delta L_4\left(k\right)+\Delta L_5\left(k\right)\hfill \\ \hfill & =-{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{\widehat{R}}_i(k-1)+{\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)\hfill \\ \hfill & -{\beta }^{-1}{\omega }_{c2i}^T(k-1){\Delta }_{ci}(k-1){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ai}|\left|{\Delta }_{ai}\left(k\right)\right|{|}^2)\hfill \\ \hfill & \times \left|\right|{\omega }_{c2i}^T\left(k\right){\Delta }_{ci}{\left(k\right)\left|\right|}^2\left|\right|{\omega }_{c2i}^T{\left(k\right)C\left(k\right)\left|\right|}^2+\left|\right|{\widehat{R}}_i(k-1)\hfill \\ \hfill & +\beta {\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)-{\omega }_{c2i}^T(k-1){\Delta }_{ci}(k-1){\left|\right|}^2\hfill \\ \hfill & +\left|\right|{\omega }_{c2i}^T\left(k\right)C\left(k\right){\Delta }_{ci}^T\left(k\right){\omega }_{c2i}^T\left(k\right)-{\delta }_{ai}\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}{\Delta }_{ri}^2\left(k\right)){\varrho }^4\hfill \\ \hfill & \times \left|\right|{\varrho }^{-1}j\left(k\right)+{\widehat{R}}_i\left(k\right)-{\varrho }^{-1}{\widehat{R}}_i(k-1){\left|\right|}^2\hfill \\ \hfill & {\vartheta }^2\left(k\right)+\left|\right|{\delta }_{ri}\left(k\right)-{\varrho }^2[{\varrho }^{-1}j\left(k\right)+{\widehat{R}}_i\left(k\right)\hfill \\ \hfill & -{\varrho }^{-1}{\widehat{R}}_i{(k-1)\left]\right||}^2\hfill \\ \hfill & -\left|\right|{\delta }_{ai}{\left(k\right)\left|\right|}^2-\left|\right|{\delta }_{ri}\left(k\right){\left|\right|}^2+{\beta }^{k+1}{Q}_i(k+1)-{\beta }^k{Q}_i\left(k\right)\hfill \\ \hfill & +{\varrho }^{k+1}{R}_i(k+1)-{\varrho }^k{R}_i\left(k\right)\hfill \end{array}$$

(57)

Therefore, we can obtain

$$\begin{array}{cc}\hfill \Delta L\left(k\right)& =-{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\times ||{\delta }_{ci}\left(k\right)\hfill \\ \hfill & +{\beta }^{-1}{V}_1{\left(k\right)\left|\right|}^2-(1-{\alpha }_{ai}\left|\right|{\Delta }_{ai}{\left(k\right)\left|\right|}^2\left)\right||X_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & \times \left|\right|W_1{\left(k\right)\left|\right|}^2+\left|\right|V_1{\left(k\right)\left|\right|}^2+\left|\right|W^1\left(k\right)X_1^T\left(k\right)-{\delta }_{ai}\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ai}\left(k\right)\left|\right|{\Delta }_{ri}{\left(k\right)\left|\right|}^2){\varrho }^4\left|\right|{\varrho }^{-1}{Y}_1\left(k\right){\left|\right|}^2{\upsilon }_{ri}^2\left(k\right)\hfill \\ \hfill & -\left|\right|{\delta }_{ai}{\left(k\right)\left|\right|}^2-\left|\right|{\delta }_{ri}\left(k\right){\left|\right|}^2+{\beta }^{k+1}{Q}_i(k+1)-{\beta }^k{Q}_i\left(k\right)\hfill \\ \hfill & +{\varrho }^{k+1}{R}_i(k+1)-{\varrho }^k{R}_i\left(k\right)\hfill \end{array}$$

(58)

where $V_1\left(k\right)={\widehat{R}}_i(k-1)+\beta {\left({\omega }_{c2i}^{\ast }\right)}^T{\Delta }_{ci}\left(k\right)-{\omega }_{c2i}^T(k-1){\Delta }_{ci}(k-1),W_1\left(k\right)={\omega }_{c2i}^T\left(k\right)C\left(k\right),$$X_1\left(k\right)={\omega }_{c2i}^T\left(k\right){\Delta }_{ci}\left(k\right),Y_1\left(k\right)=j\left(k\right)+\varrho {\widehat{R}}_i\left(k\right)-{\widehat{R}}_i(k-1)$, and we can obtain $\left|\right|V_1\left(k\right)\left|\right|\le V_{1m},\left|\right|W_1\left(k\right)\left|\right|\le W_{1m},\left|\right|X_1\left(k\right)\left|\right|\le X_{1m},\left|\right|Y_1\left(k\right)\left|\right|\le Y_{1m}$. Next, we can obtain

$$\begin{array}{cc}\hfill \Delta L\left(k\right)\le & -{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{V}_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ai}\left|\right|{\Delta }_{ai}{\left(k\right)\left|\right|}^2\left)\right||X_1{\left(k\right)\left|\right|}^2\left|\right|W_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & +2\left|\right|W_1\left(k\right)X_1^T{\left(k\right)\left|\right|}^2+\left|\right|{\delta }_{ai}\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}\left|\right|{\Delta }_{ri}{\left(k\right)\left|\right|}^2){\varrho }^2\left|\right|Y_1\left(k\right){\left|\right|}^2{\vartheta }_{ri}^2\left(k\right)\hfill \\ \hfill & +2\left|\right|{\delta }_{ri}{\left(k\right)\left|\right|}^2+2\left|\right|Y_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & -{\beta }^k{Q}_i\left(k\right)-{\varrho }^k{R}_i\left(k\right)\hfill \end{array}$$

(59)

Moreover, we can obtain

$$\begin{array}{cc}\hfill \Delta L\left(k\right)\le & -{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{V}_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ai}\left|\right|{\Delta }_{ai}{\left(k\right)\left|\right|}^2\left)\right||X_1{\left(k\right)\left|\right|}^2\left|\right|W_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & +V_{1m}^2+2W_{1m}^2{X}_{1m}^2+2\left|\right|{\left({\omega }_{a2i}^{\ast }\right)}^T{\Delta }_{ai}\left(k\right){\left|\right|}^2\hfill \\ \hfill & +2\left|\right|{\omega }_{a2i}^T{\Delta }_{ai}\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}\left|\right|{\Delta }_{ri}{\left(k\right)\left|\right|}^2){\varrho }^2\left|\right|Y_1\left(k\right){\left|\right|}^2{\vartheta }_{ri}^2\left(k\right)\hfill \\ \hfill & +2\left|\right|{\delta }_{ri}{\left(k\right)\left|\right|}^2+2\left|\right|Y_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & -{\beta }^k{Q}_i\left(k\right)-{\varrho }^k{R}_i\left(k\right)\hfill \\ \hfill & \le -{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{V}_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ai}\left|\right|{\Delta }_{ai}{\left(k\right)\left|\right|}^2\left)\right||X_1{\left(k\right)\left|\right|}^2\left|\right|W_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & +V_{1m}^2+2W_{1m}^2{X}_{1m}^2+4{\omega }_{aim}^2{\Delta }_{aim}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}\left|\right|{\Delta }_{ri}{\left(k\right)\left|\right|}^2){\varrho }^2\left|\right|Y_1\left(k\right){\left|\right|}^2{\vartheta }_{ri}^2\left(k\right)\hfill \\ \hfill & +2{\delta }_{rim}^2+2Y_{1m}^2\hfill \\ \hfill & -{\beta }^k{Q}_i\left(k\right)-{\varrho }^k{R}_i\left(k\right)\hfill \end{array}$$

(60)

If the conditions are met, then we can obtain

$$\begin{array}{cc}\hfill & {\alpha }_{ri}\le \frac{1}{\left|\right|{\Delta }_{ri}\left(k\right){\left|\right|}^2},{\alpha }_{ci}\le \frac{1}{{\beta }^2\left|\right|{\Delta }_{ci}\left(k\right){\left|\right|}^2},{\alpha }_{ai}\le \frac{1}{\left|\right|{\Delta }_{ai}\left(k\right){\left|\right|}^2}\hfill \\ \hfill & \left|\right|{\delta }_{ci}\left(k\right)\left|\right|>\sqrt{(V_{1m}^2+2W_{1m}^2{X}_{1m}^2+4{\omega }_{a2im}^2{\Delta }_{aim}^2+2{\delta }_{rim}^2+2Y_{1m}^2)/{\beta }^2}\hfill \end{array}$$

We can derive $\Delta L\left(k\right)\le 0$. The proof has been completed.

(2) In the absence of the triggering conditions, consider the following:

$$L\left(k\right)=L_1\left(k\right)+L_2\left(k\right)+L_4\left(k\right)$$

(61)

where

$$\begin{array}{cc}\hfill & L_1\left(k\right)=\frac{1}{{\alpha }_{ri}}tr\left({\omega }_{r2i}^T\left(k\right){\omega }_{r2i}\left(k\right)\right),\hfill \\ \hfill & L_2\left(k\right)=\frac{1}{{\alpha }_{ci}}tr\left({\omega }_{c2i}^T\left(k\right){\omega }_{c2i}\left(k\right)\right),\hfill \\ \hfill & L_4\left(k\right)=e_i^T\left(k\right)e_i\left(k\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill \Delta L\left(k\right)& =\Delta L_1\left(k\right)+\Delta L_2\left(k\right)+\Delta L_4\left(k\right)\hfill \\ \hfill & =-{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{V}_1{\left(k\right)\left|\right|}^2+\left|\right|V_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}\left|\right|{\Delta }_{ri}{\left(k\right)\left|\right|}^2){\varrho }^2\left|\right|Y_1\left(k\right){\left|\right|}^2{\vartheta }_{ri}^2\left(k\right)\hfill \\ \hfill & +2{\delta }_{rim}^2+2Y_{1m}^2\hfill \\ \hfill & +e_i^T(k+1)e_i(k+1)-e_i^T\left(k\right)e_i\left(k\right)\hfill \end{array}$$

(62)

$$\begin{array}{cc}\hfill \Delta L\left(k\right)& \le -{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{V}_1{\left(k\right)\left|\right|}^2+\left|\right|V_1\left(k\right){\left|\right|}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}\left|\right|{\Delta }_{ri}{\left(k\right)\left|\right|}^2){\varrho }^2\left|\right|Y_1\left(k\right){\left|\right|}^2{\vartheta }_{ri}^2\left(k\right)\hfill \\ \hfill & +2{\delta }_{rim}^2+2Y_{1m}^2\hfill \\ \hfill & +\left(\right(\iota \left|\right|e_i\left(k\right)+\iota \left|\right|{ϵ}_i^s{\left|\right|)}^2-\left|\right|e_i{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \le -{\beta }^2\left|\right|{\delta }_{ci}{\left(k\right)\left|\right|}^2-{\beta }^2(1-{\alpha }_{ci}{\beta }^2\left|\right|{\Delta }_{ci}{\left(k\right)\left|\right|}^2)\hfill \\ \hfill & \times \left|\right|{\delta }_{ci}\left(k\right)+{\beta }^{-1}{V}_1\left(k\right){\left|\right|}^2+V_{1m}^2\hfill \\ \hfill & -(1-{\alpha }_{ri}\left|\right|{\Delta }_{ri}{\left(k\right)\left|\right|}^2){\varrho }^2\left|\right|Y_1\left(k\right){\left|\right|}^2{\vartheta }_{ri}^2\left(k\right)\hfill \\ \hfill & +2{\delta }_{rim}^2+2Y_{1m}^2\hfill \\ \hfill & -(1-2{\iota }^2)\left|\right|e_i\left(k\right){\left|\right|}^2-2{\iota }^2\left|\right|{ϵ}_i^s{\left|\right|}^2\hfill \end{array}$$

(63)

In the event that it is satisfied that ${\alpha }_{ri}\le \frac{1}{\left|\right|{\Delta }_{ri}\left(k\right){\left|\right|}^2},{\alpha }_{ci}\le \frac{1}{{\beta }^2\left|\right|{\Delta }_{ci}\left(k\right){\left|\right|}^2},{\alpha }_{ai}\le \frac{1}{\left|\right|{\Delta }_{ai}\left(k\right){\left|\right|}^2}$, and $\left|\right|{\delta }_{ci}\left(k\right)\left|\right|>\sqrt{(V_{1m}^2+2{\delta }_{rim}^2+2Y_{1m}^2)/{\beta }^2}$, one has $\Delta L\left(k\right)\le 0$. Thus, we can derive $\Delta L\left(k\right)\le 0$, and the proof is completed.

6. Statistical Data Illustration

To demonstrate the viability of the proposed method, a simulation is presented in the following section.

Nonlinear MAS Consisting of One Leader and Six Followers

There were six followers and one leader in this tangled set of MASs which were considered. Figure 1 depicts the connection graph of the studied MASs. There was a leader of 0, and there were followers of 1, 2, 3, 4, 5, and 6. It is possible to obtain the corresponding adjacency matrix $a_{14}=a_{21}=a_{32}=a_{43}=a_{52}=a_{65}=1$. There is a weighted relationship involving the leaders and followers where $b_1=1,b_2=b_3=b_4=b_5=b_6=0$. It is possible for agent 1 to accept the information of the leader immediately. The system model parameters for MASs with one leader as well as six followers are as follows: $A=\left|\begin{array}{cc}0.995& 0.09980\\ -0.09982& 0.995\end{array}\right|, B_1={[0,0.2]}^T, B_2={[0,0.5]}^T, B_3={[0,0.4]}^T, B_4={[0,0.3]}^T, B_5={[0,0.6]}^T$, and $B_6={[0,0.7]}^T$.

The weight matrices are as follows: $Q_{11}=Q_{22}=Q_{33}=Q_{44}=Q_{55}=Q_{66}=1,R_{11}=R_{22}=R_{33}=R_{44}=R_{55}=R_{66}=I_{2\times 2}$, and $Q_{14}=Q_{21}=Q_{32}=Q_{43}=Q_{52}=Q_{65}=I_{2\times 2}$. The learning rates are ${\alpha }_{ri}=0.95,{\alpha }_{ai}=0.90$, and ${\alpha }_{ci}=0.07(iisequalto1,2,3,4,5,6)$, with a discount factor of $\varrho =0.57,\beta =0.9$.

For the agents, the activation function of the RNNs and ANNs is as follows: $Z_{r1}\left(k\right)={[e_1^T\left(k\right),u_1^T\left(k{t}_s^1\right),u_4^T\left(k{t}_s^4\right)]}^T,$$Z_{a1}\left(k\right)=e_1\left(k{t}_s^1\right),$$Z_{r2}\left(k\right)={[e_2^T\left(k\right),u_2^T\left(k{t}_s^2\right),u_1^T\left(k{t}_s^1\right)]}^T,$$Z_{a2}\left(k\right)=e_2\left(k{t}_s^2\right),$$Z_{r3}\left(k\right)={[e_3^T\left(k\right),u_3^T\left(k{t}_s^3\right),u_2^T\left(k{t}_s^2\right)]}^T,$$Z_{a3}\left(k\right)=e_3\left(k{t}_s^3\right),$$Z_{r4}\left(k\right)={[e_4^T\left(k\right),u_4^T\left(k{t}_s^4\right),u_3^T\left(k{t}_s^3\right)]}^T,$

$Z_{a4}\left(k\right)=e_4\left(k{t}_s^4\right),$$Z_{r5}\left(k\right)={[e_5^T\left(k\right),u_5^T\left(k{t}_s^5\right),u_2^T\left(k{t}_s^2\right)]}^T,$$Z_{a5}\left(k\right)=e_5\left(k{t}_s^5\right),$$Z_{r6}\left(k\right)={[e_6^T\left(k\right),u_6^T\left(k{t}_s^6\right),u_5^T\left(k{t}_s^5\right)]}^T,$$Z_{a6}\left(k\right)=e_6\left(k{t}_s^6\right).$ The initial values of the leader and followers are $x_0\left(0\right)={[0.6675,0.7940]}^T,x_1\left(0\right)={[0.5734,0.6000]}^T,x_2\left(0\right)={[0.5667,0.7348]}^T,x_3\left(0\right)={[0.8694,0.7140]}^T,x_4\left(0\right)={[1.0212,1.3842]}^T,x_5\left(0\right)={[0.8606,1.5565]}^T$, and $x_6\left(0\right)={[0.5274,1.3235]}^T$.

According to Figure 2, all followers of the leader were able to accurately follow the leader, and the whole MAS was able to achieve synchronization. Figure 3 illustrates the six agents’ cumulative amount of trigger instants. On average, the amount of trigger instants for the six agents was approximately 220. However, using the traditional RL method, the number was approximately 1000. As a result, the computational burden was reduced by $78.0\%$ in comparison with the conventional time-triggered method. According to Figure 4, the trigger mechanism of each agent is illustrated, which indicates that the actor network weight will be updated only when the trigger mechanism is satisfied. As can be seen in Figure 5, there is a correlation involving the error of triggering $\left|\right|{ϵ}_i^s\left(k\right){\left|\right|}^2$ as well as the minimum triggering requirements ${\pi }_{i}T$. Over time, it appears that the triggering error converged. Figure 6 and Figure 7 illustrate the evaluation of the local neighborhood errors using the proposed control method, and it is shown that they could be converged to 0 at k = 60. The local neighborhood errors of [32] are shown in Figure 8 and Figure 9. In comparison with Figure 8 and Figure 9, our proposed control method produced a better convergence effect. Figure 10 and Figure 11 show the estimation of the ANN weight parameters. With the proposed control method, the actor network weights can stabilize faster than with IrQL.

7. Conclusions

According to this study, an event-triggered optimum controlling problem for model-free MASs was examined using the IrQL method based on RL. A new IrQL method was introduced by adding additional IRR functions [32], As a result, more information could be obtained by the agent. As a consequence of defining the IRR formula, we defined the Q-function and derived the corresponding HJB equation. In an iterative approach to IrQL, this method was designed to calculate the optimal control strategy. Using the IrQL algorithm, an event-triggered controller utilizing the IrQL method was presented. It was designed to update the controller only at the time of triggering to reduce the burden on computing resources and the transmission network. An RCA-NN was used to implement the suggested approach, which eliminated the need for a model of the system. It is possible to determine the convergent weights of neural networks using the Lyapunov method. To assess the performance and control efficiency of the suggested algorithm, a simulation model was used. Further research will be conducted on the effect of the discount rates on system reliability.