Age of Information Minimization for Radio Frequency Energy-Harvesting Cognitive Radio Networks

3.1. Primary User Model

3.2. Secondary User Model

4.1. POMDP Formulation

• Actions: At the beginning of each time slot t, the SU needs to decide whether to sense the spectrum. If it decides not to sense the spectrum, then it captures energy from the PU transmissions and does not update, i.e., $x_t=(0,0)$. If it decides to sense the spectrum and finds that the PU is idle, it further decides whether to update based on the available energy, the AoI value, the channel state information from the SU to the CBS and from the PU to the SU, i.e., $x_t=(1,0)$ and $x_t=(1,1)$. Thus, the action for each time slot t is $x_t=({\varphi }_t,{\theta }_t)\in \mathcal{X}\triangleq \{(0,0),(1,0),(1,1):b_t\ge {\varphi }_t\delta +{\theta }_t{E}_{T,t}\}$, where ${\varphi }_t\in {\Gamma }_{\varphi }\triangleq \{0,1:b_t\ge {\varphi }_t\delta \}$ and ${\theta }_t\in {\Gamma }_{\theta }\triangleq \{0,1:b_t\ge \delta +{\theta }_t{E}_{T,t}\}$.
• Observations and beliefs: Let ${\widehat{q}}_t\in \{A,I\}$ denote the observation of the PU’s state. The belief ${\rho }_t\in [0,1]$ is a condition probability that the spectrum is vacated by the PU. The belief is updated according to the following cases.
  Case 1: The SU does not sense the spectrum; the new belief is given by

  $${\rho }_{t+1}={\Lambda }_0\left({\rho }_t\right)={\rho }_t{p}_{ii}+(1-{\rho }_t)p_{ai}.$$

  (11)
  Case 2: If the PU is sensed to be active, the SU harvests energy in the remaining time of the current time slot, i.e., the battery energy increases. This implies the true state of the PU is $q_t=A$. The belief is updated as

  $${\rho }_{t+1}=p_{ai}.$$

  (12)
  Case 3: If the PU is sensed to be active, the SU does not harvest energy; i.e., the battery energy does not change and is lower than $B_{\max }$. This implies the true state of the PU is $q_t=I$. The new belief is expressed as

  $${\rho }_{t+1}=p_{ii}.$$

  (13)
  Case 4: If the PU is sensed to be active, the battery energy is $B_{\max }$ at time slot t. The new belief is given by

  $${\rho }_{t+1}={\Lambda }_{1A}\left({\rho }_t\right)={\zeta }_t{p}_{ii}+(1-{\zeta }_t)p_{ai},$$

  (14)

  where

  $${\zeta }_t\triangleq \mathbb{P}(q_t=I|\widehat{q_t}=A)=\frac{{\rho }_t(1-p_f)}{{\rho }_t{p}_f+(1-{\rho }_t)(1-p_d)}.$$

  (15)
  Case 5: If the PU is sensed to be idle, the SU does not update. The belief is updated as

  $${\rho }_{t+1}={\Lambda }_{1I}\left({\rho }_t\right)=\overline{{\zeta }_t}{p}_{ii}+(1-\overline{{\zeta }_t})p_{ai},$$

  (16)

  where

  $$\overline{{\zeta }_t}\triangleq \mathbb{P}(q_t=I|\widehat{q_t}=I)=\frac{{\rho }_t(1-p_f)}{{\rho }_t(1-p_f)+(1-{\rho }_t)(1-p_d)}.$$

  (17)
  Case 6: If the PU is sensed to be idle, the SU updates successfully. This implies that the true state of the PU is $q_t=I$. Then, we have

  $${\rho }_{t+1}=p_{ii}.$$

  (18)
  Case 7: If the PU is sensed to be idle, the SU update fails. This implies that the true state of the PU is $q_t=A$. Then, we have

  $${\rho }_{t+1}=p_{ai}.$$

  (19)
  Although (11)–(19) cover seven cases from case one to case seven, the new beliefs in both case two and case seven are denoted as $p_{ai}$, and the new beliefs in both case three and case six are denoted as $p_{ii}$. Hence, the SU can only transit to five beliefs. That is, the number of possible beliefs is finite over T time slots. Thus, for the length of T time slots, the belief space $\mathsf{\Phi }$ is a finite set.
• States: Denote the discrete battery energy level of the SU at the beginning of time slot t by $b_t^{{}^{\prime }}\in \mathcal{B}\triangleq \{0,1,2,...,b_{\max }\}$, where $b_{\max }$ is the maximum battery energy level that can be stored inside the battery of the SU. Then, each energy quantum of the SU’s battery contains $\frac{B_{\max }}{b_{\max }}$ Joules. In this case, we use $b_t^{{}^{\prime }}=⌊\frac{b_t{b}_{\max }}{B_{\max }}⌋$ to convert the continuous battery energy of the SU to the discrete battery energy level, by which a lower bound to the AoI of the original continuous system is obtained. Similarly, divide continuous channel power gain into finite number of intervals according to fading probability density function (PDF). Thus, the discrete channel power gain levels from the SU to the CBS and from the PU to the SU are expressed as $g_t^{{}^{\prime }}\in \mathcal{G}\triangleq (0,1,2,...,g_{\max })$ and $h_t^{{}^{\prime }}\in \mathcal{H}\triangleq (0,1,2,...,h_{\max })$, respectively. Here, $g_{\max }$ and $h_{\max }$ denote the corresponding maximum channel power gain levels. At each time slot t, the completely observable states include channel state from the PU to the SU, channel state from the SU to the CBS, the AoI state, and battery state, denoted by $s_t\triangleq (h_t^{{}^{\prime }},g_t^{{}^{\prime }},a_t,b_t^{{}^{\prime }})$. Note that the state space, i.e., $\mathcal{S}\triangleq \mathcal{H}\times \mathcal{G}\times \mathcal{A}\times \mathcal{B}$, is finite. Due to imperfect sensing, an update may be unsuccessful when the sensing result is $\widehat{q_t}=I$ and the update decision is ${\theta }_t=1$. Thus,

  $$a_{t+1}=\left\{\begin{array}{cc}1,\hfill & \mathrm{when}{x}_t=(1,1)\mathrm{and}\widehat{q_t}=q_t,\hfill \\ a_t+1,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

  (20)

  for $t=1,2,....,T$. Alternatively, we can express $a_{t+1}=(1-{\theta }_t)a_t+1$ when the sensing result $\widehat{q_t}=I$ is correct. Additionally, the PU’s spectrum state is only partially observable, which is described by the belief ${\rho }_t$. Thus, for each time slot t, the complete system state is denoted by $(s_t,{\rho }_t)$. Since $\mathcal{S}$ and $\mathsf{\Phi }$ are finite, the SU experiences a finite number of possible system states $(s_t,{\varrho }_t)\in \mathcal{S}\times \mathsf{\Phi }$.
• Transition probabilities: For time slot t, given the current state $s_t=(h_t^{{}^{\prime }},g_t^{{}^{\prime }},a_t,b_t^{{}^{\prime }})$ and the action $x_t=({\varphi }_t,{\theta }_t)$, the transition probability to the next state $s_{t+1}=(h_{t+1}^{{}^{\prime }},g_{t+1}^{{}^{\prime }},a_{t+1},$$b_{t+1}^{{}^{\prime }})$ is denoted by $p_{x_t}\left(s_{t+1}\right|s_t)$. Since the captured energy and the channel power gains are independently and identically distributed (i.i.d), the transition probabilities for taking actions other than $x_t=(1,1)$ are given as follows.

  $$\begin{array}{c}\hfill p_{x_t}\left(s_{t+1}\right|s_t)=\mathbb{P}\left(a_{t+1}\right|a_t,x_t)\mathbb{P}\left(b_{t+1}^{{}^{\prime }}\right|b_t^{{}^{\prime }},g_t^{{}^{\prime }},h_t^{{}^{\prime }},x_t)\mathbb{P}\left(g_{t+1}^{{}^{\prime }}\right)\mathbb{P}\left(h_{t+1}^{{}^{\prime }}\right),\end{array}$$

  (21)

  where

  $$\mathbb{P}\left(a_{t+1}\right|a_t,x_t)=\left\{\begin{array}{cc}1,\hfill & \mathrm{when}{a}_{t+1}=(1-{\theta }_t)a_t+1,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

  (22)

  $$\begin{array}{c}\mathbb{P}\left(b_{t+1}^{{}^{\prime }}\right|b_t^{{}^{\prime }},g_t^{{}^{\prime }},h_t^{{}^{\prime }},x_t)=\left\{\begin{array}{cc}1,\hfill & \mathrm{when}{\varphi }_t=0\mathrm{and}{b}_{t+1}=\min \{b_t+E_{H,t}^1,B_{\max }\},\hfill \\ 1,\hfill & \mathrm{when}{\varphi }_t=0\mathrm{and}{b}_{t+1}=b_t,\hfill \\ 1,\hfill & \mathrm{when}{\varphi }_t=1,{\theta }_t=0,\mathrm{and}{b}_{t+1}=min\{b_t-\delta +E_{H,t}^2,B_{\max }\},\hfill \\ 1,\hfill & \mathrm{when}{\varphi }_t=1,{\theta }_t=0,\mathrm{and}{b}_{t+1}=b_t-\delta ,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

  (23)
  For the action $x_t=(1,1)$, the transition probability is expressed as follows.

  $$\begin{array}{c}\hfill p_{x_t}\left(s_{t+1}\right|s_t,\widehat{q_t},q_t)=\mathbb{P}\left(a_{t+1}\right|a_t,x_t\widehat{q_t},q_t)\times \mathbb{P}\left(b_{t+1}^{{}^{\prime }}\right|b_t^{{}^{\prime }},g_t^{{}^{\prime }},h_t^{{}^{\prime }},x_t)\times \mathbb{P}\left(g_{t+1}^{{}^{\prime }}\right)\mathbb{P}\left(h_{t+1}^{{}^{\prime }}\right),\end{array}$$

  (24)

  where

  $$\mathbb{P}\left(a_{t+1}\right|a_t,x_t)=\left\{\begin{array}{cc}\overline{\zeta },\hfill & \mathrm{when}{a}_{t+1}=1\mathrm{and}{q}_t=\widehat{q_t},\hfill \\ 1-\overline{\zeta },\hfill & \mathrm{when}{a}_{t+1}=a_t+1\mathrm{and}{q}_t\le \widehat{q_t},\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

  (25)

  and

  $$\mathbb{P}\left(b_{t+1}^{{}^{\prime }}\right|b_t^{{}^{\prime }},g_t^{{}^{\prime }},h_t^{{}^{\prime }},x_t)=\left\{\begin{array}{cc}1,\hfill & \mathrm{when}{\varphi }_t=1,{\theta }_t=1,b_{t+1}=b_t-\delta -E_{T,t},\mathrm{and}\widehat{q_t}=q_t,\hfill \\ 1,\hfill & \mathrm{when}{\varphi }_t=1,{\theta }_t=1,b_{t+1}=0,\widehat{q_t}=I,\mathrm{and}{q}_t=A,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

  (26)
• Cost: Let the immediate cost at state $s_t$ denoted by $C\left(s_t\right)$, which is the accumulated AoI at time slot t. Then, we have

  $$C\left(s_t\right)=a_t,t=0,1,...,T-1.$$

  (27)
• Policy: The policy is expressed as $\pi =\{{\vartheta }_0,{\vartheta }_1,...,{\vartheta }_{T-1}\}$, where ${\vartheta }_t$ is the deterministic decision rule that maps a system state $(s_t,{\rho }_t)\in \mathcal{S}\times \mathsf{\Phi }$ into an action $x_t\in \mathcal{X}$, i.e., $x_t={\vartheta }_t(s_t,{\rho }_t)$. In this paper, let $\Pi$ denote the set of all deterministic decision policies.

4.2. POMDP Solution