1. Introduction
With the rapid development of technologies such as the Internet of Things (IoT), augmented reality (AR), and virtual reality (VR), traditional communication methods can no longer meet the growing demands for interaction. Users now expect more immersive and intuitive ways to access and communicate information beyond simple texts and images. This shift imposes higher requirements on next-generation wireless communication systems like B5G (Beyond Fifth Generation) and 6G (Sixth Generation) [
1]. In addition to offering higher data transmission rates and more reliable mobile connections, next-generation wireless systems will enable emerging applications such as smart cities, VR, smart homes, and intelligent transportation. Integrated sensing and communications (ISAC) is widely regarded as one of the practical solutions to achieve this goal [
2]. ISAC integrates wireless communication and radar sensing, sharing the same spectrum resources and hardware platforms [
3], thus alleviating the issue of spectrum scarcity. Therefore, exploring and applying ISAC technology holds significant social, economic, and technical value.
Non-orthogonal multiple access (NOMA), as one of the critical technologies in 5G, allows users to share time–frequency resources through power domain multiplexing [
4]. The transmitter superimposes the signals of different users on the same frequency band and distinguishes them by allocating power and coding. This non-orthogonal signal superposition effectively improves spectrum efficiency, meeting multi-user access requirements while reducing system complexity [
5]. The receiver uses successive interference cancellation (SIC) to process multiple users’ signals sequentially, eliminating interference and decoding them accordingly. Many studies are currently focusing on applying NOMA in integrated sensing and communication (ISAC). In [
6], the NOAM-ISAC framework was first established, and NOMA-assisted joint radar and multicast–unicast communication (Rad-MU-Com) was proposed. A hybrid multicast–unicast message transmission between radar and communication users was enabled using a multiple-input multiple-output (MIMO) dual-function radar-communication (DFRC) base station. In [
7], the NOMA-based ISAC framework optimized beamforming design, enhancing communication throughput and sensing power. In [
8], the NOMA-based joint communication, sensing, and multi-layer computing (JCSMC) framework optimized resource allocation for efficient computation offloading.
These studies emphasize the improvement of the integrated sensing and communication (ISAC) system performance through NOMA; however, the system performance under NOMA communication largely depends on the interference intensity between users and the difficulty of SIC decoding, which relies on user clustering [
9]. The authors of [
10] reduce user interference through an efficient user clustering algorithm considering different transmission power scenarios and provide a robust power allocation (PA) solution under imperfect CSI assumptions. The authors of [
11] propose a low-complexity learning-based user clustering method using an improved mean-shift clustering algorithm, which effectively utilizes the degrees of freedom in the system to form NOMA clusters and significantly enhances performance. The authors of [
12] investigate user clustering and power control in MISO-NOMA networks. The study proposes a two-step user clustering and power control algorithm, with the proposed K-means-based iterative power control scheme significantly outperforming other reference methods regarding power consumption and energy efficiency. The authors of [
13] address the issues of high node death rates and energy consumption in energy-efficient clustered routing communication for wireless sensor networks, proposing an energy-efficient clustered routing algorithm based on an energy iterative model and swarm optimization. The authors of [
14] employ Gaussian Mixture Models (GMMs) for the unsupervised clustering of received signals, optimizing decision boundaries to improve bit error rate (BER) performance.
However, the clustering algorithms in the literature above are rarely applied in NOMA-ISAC systems. The K-means algorithm requires each cluster to have a regular shape, and its high sensitivity to initial values, like that of the GMM algorithm, makes it challenging to significantly improve the performance of NOMA-ISAC systems. Moreover, most existing user clustering algorithms [
15] group users with close Euclidean distances, which contradicts the NOMA clustering approach that aims to maximize differences in user channel gains. The authors of [
16] introduce a dynamic user clustering method based on CDA and COA algorithms, using a joint optimization approach to maximize total rate and sensing capability. Although this algorithm has some applications in ISAC, it is not closely related to sensing, and very few user clustering algorithms are applied in NOMA-ISAC systems. To date, no scholars have employed clustering algorithms to optimize the performance of integrated sensing and communication systems in multi-user distribution scenarios. This paper proposes a reliable user clustering algorithm suitable for NOMA-ISAC systems.
To further enhance the performance of the integrated sensing and communication (ISAC) system and address the non-convex problem of beamforming design, this paper utilizes a dual-function base station that can simultaneously achieve communication and sensing functions. The improved Gaussian Mixture Model (GMM) clustering algorithm forms strip-shaped clusters that adapt to NOMA-ISAC, simultaneously simplifying the objective function. Additionally, due to the difficulty in obtaining stable solutions for the multi-rate problem involved in the objective function and the high coupling of multiple objective variables, to address the integrated sensing and communication beamforming design problem, fractional programming is used to transform the maximization of the logarithmic and fractional composite spectral efficiency problem into a series of convex problems. After obtaining the simplified objective function using the cluster splitting algorithm, a suboptimal solution to the original objective function can be obtained by fixing the auxiliary variables [
17] and taking the partial derivatives of the reconstructed problem.
The main contributions of this paper can be summarized as follows:
An improved GMM user clustering algorithm is proposed for multi-user NOMA-assisted ISAC systems as follows:
Sensitivity to initial centroid positions is reduced by computing the Sum of Squared Errors (SSE).
Directional weight factors and penalty functions are introduced to ensure the algorithm forms strip-like clusters suited to the NOMA-ISAC system.
The posterior probability in the GMM algorithm is modified, and an adaptive covariance matrix update mechanism is used to optimize the coordination between communication and sensing.
The improved GMM user clustering algorithm reduces inter-user interference in traditional multiple-access methods and increases bandwidth reuse gain.
Fractional programming handles the joint beamforming design for ISAC systems. Converting the maximization of spectral efficiency involving logarithmic and fractional terms into a series of convex problems achieves more effective system power gain and channel gain.
2. System Model
The integrated sensing and communication system model assisted by NOMA is illustrated in
Figure 1, featuring a dual-functional base station that simultaneously performs communication and sensing. The base station is equipped with
antennas, utilizing a Uniform Linear Array (ULA) to serve
single-antenna users and
sensing targets, ensuring minimal interference between antenna units and maintaining good phase alignment. The spacing between each antenna is set to half the wavelength. Users are unevenly distributed around the base station, while the sensing targets are considered line-of-sight links relative to the base station. All users are eventually divided into
clusters, with the
cluster denoted as
.
is the number of users within the
cluster and satisfies
. After clustering, users within each cluster utilize NOMA to share time–frequency resources. Users are distinguished through power multiplexing. Inter-cluster communication employs orthogonal multiple access (OMA). Each cluster is orthogonal. Interference between clusters is ignored to simplify system analysis and design.
2.1. Communications Model
During downlink communication with NOMA, the signal received by the
user
of the
cluster can be expressed as the following:
where
,
represents the superimposed signal transmitted by the base station,
denotes the beamforming vector of the user
,
denotes the information data stream of the user
,
denotes the channel gain from the base station to the user
,
denotes the remaining users other than the user
in the
cluster and
denotes additive Gaussian white noise with a variance of
.
In the system model of this paper, the link from the base station to the user is considered a line-of-sight (LOS) link. Therefore, the free-space path loss model is used as the path loss model for the user, which is given explicitly by the following:
where
represents the distance from the user
to the base station. Let the position of the base station be denoted by
and the position of the user
be denoted by
, where
and
denote the horizontal and vertical coordinates of the base station and user
, respectively. Then,
can be expressed as the following:
Without loss of generality, let the distances from the users in the
cluster to the base station be sorted as
. The channel gain for users within the cluster can thus be expressed as
. Based on the above channel gain condition, the signal-to-interference-plus-noise ratio (SINR) for user
in cluster m is given as follows:
where
is the interfering signal that the SIC decodes and fails to eliminate. The reachable rate of user
can be expressed as the following:
where
is the system bandwidth. The sum rate of all users is expressed as the following:
In NOMA communication, complex user channel conditions are one of the critical factors affecting the sum rate. By increasing the channel gain difference between users sharing the same channel, the difficulty of SIC decoding and user interference can be reduced. This, in turn, improves the spectral efficiency for users.
2.2. Sensing Model
In an ISAC system, the communication waveform can be used for user communication and sensing detection. To achieve good sensing performance, the goal of the sensing system is to maximize the sensing power under the condition of prior target information [
16]. The sensing power is expressed as follows:
where
is the direction of the sensing target,
is the steering vector in the direction of the target, and
is the covariance matrix of the transmitted signal. In the waveform design for sensing, this corresponds to designing the covariance matrix of the transmitted signal [
18].
Constant modulus constraints are typically introduced to improve transmitter efficiency and enable the radar’s nonlinear amplifiers to operate at maximum efficiency, enhancing radar transmission performance. However, in practical operation, there may be a conflict between radar detection and estimation performance. Therefore, the Peak Average Power Ratio (PAPR) is typically introduced to address more general energy constraint issues. The mathematical definition of PAPR is as follows:
where
. When
, this corresponds to the constant envelope constraint. When
, the peak-to-average ratio is converted into the continuous modulus constraint. The authors of [
19] have verified that relaxing the constant envelope constraint can yield better performance for the transmitted waveform. Moreover, the peak-to-average constraint is more flexible than the constant modulus constraint, providing more degrees of freedom for the sensing waveform.
3. Problem Description
In the ISAC system model, considering the trade-off between communication and sensing performance, a trade-off factor
is introduced to balance the system’s performance of communication and sensing. Under this condition, the joint waveform design problem for the ISAC system can be described as follows:
where
is the trade-off factor for balancing communication and sensing performances. When
, the system considers only sensing performance. When
, the system considers only communication performance. It is necessary to adjust the trade-off factor to balance communication and sensing continuously. The sensing target
.
is the Quality of Service (QoS) constraint, ensuring the minimum communication rate for each user,
ensures that each user belongs to only one cluster,
is the total transmission power constraint,
is the total transmit power of the antennas and
is the PAPR constraint, addressing the peak-to-average ratio problem in the sensing waveform. Because of the waveform design problem,
is a non-convex problem and is generally difficult to solve directly. This paper decouples the original objective function through quadratic transformations and applies an improved Gaussian clustering algorithm to cluster the users. This reduces the difficulty of SIC decoding and simplifies the objective function. Fractional programming is then used to handle the beamforming design problem at the base station, decoupling the objective function and reducing its computational complexity. Finally, alternating iterative optimization is used to find the optimal variables and obtain a suboptimal solution to the objective function.
5. Simulation Experiments
To validate the effectiveness of the proposed algorithms and methods, we conducted a detailed numerical simulation of the clustering performance of the improved GMM algorithm and its impact on the system performance within the NOMA-ISAC framework. The communication area is assumed to be 100 × 100 m
2, with the base station located at the center. The communication users are randomly distributed around the base station, while the sensing targets are located at the edge of the region at angles of 45° and −45°. The other parameters are shown in
Table 1.
To demonstrate the clustering performance of the improved GMM algorithm, we compare it with the original GMM clustering algorithm and the K-means clustering algorithm. In this comparison, parameter a is set to 0.4 and b to 1.2. Based on the initialization of the improved GMM algorithm in
Section 4.1.3, the optimal number of clusters is determined. For the K-means algorithm, the initial K value is set to 5. Due to the sensitivity of the K-means algorithm to the initial positions of the cluster centers, as shown in
Figure 3, the user distribution in Cluster 2 is scattered and not concentrated. This is because the initial cluster centers in the K-means algorithm are randomly chosen, resulting in suboptimal clustering. In
Figure 4, it is clear that the GMM algorithm can accommodate user clusters of various shapes and sizes. However, in the clustering results of both algorithms, the number of users in each cluster is not evenly distributed, and there are significant weight differences between clusters. Therefore, both algorithms perform poorly for user clustering in NOMA communication.
Figure 5 shows the clustering effect of the improved GMM algorithm. It can be seen that the algorithm clusters users into strip-like categories centered on the base station. Each cluster has a weight close to 0.25, and the number of users in each cluster is more evenly distributed than the K-means and traditional GMM algorithms. Considering the base station’s location, this algorithm is more suitable for user classification in NOMA communication. Additionally,
Figure 5 shows that Cluster 4 has fewer users than the other clusters. This is because when
is relatively large, the algorithm emphasizes intra-cluster diversity, which may cause the mixture coefficient
of each cluster to become uneven. During iteration
, introducing directional weights makes one cluster’s mixture coefficient more significant than the others, resulting in a smaller proportion for another cluster.
The objective function of this paper maximizes the weighted sum of communication rate and sensing power within the NOMA-ISAC system. The goal is to maximize this function value without compromising user communication quality. Therefore, the choice of the trade-off factor in the objective function is crucial, as it must balance both user communication rate and sensing performance.
Figure 6 shows the relationship between the number of iterations and the objective function value under different values. It can be observed that since the sensing power has a higher value compared to the communication rate, the objective function value increases more significantly as
decreases. However, regardless of the value
, the algorithm consistently converges to a stable function value after about four iterations.
To further illustrate the impact of the proposed improved GMM algorithm on the performance of the ISAC system under NOMA communication, we compared the spectral efficiency of the system using the K-means algorithm, traditional GMM algorithm, and OMA-ISAC with the spectral efficiency of the improved GMM algorithm. The improved GMM clustering algorithm introduces a penalty function and direction weights that enhance the diversity of users within clusters and provide an advantage in signal processing at the receiver end during SIC compared to the base station’s strip-like cluster categories.
Figure 7 shows that the improved GMM algorithm provides a superior increase in spectral efficiency for users compared to the other two algorithms.
In NOMA communication, received signals are affected by interference from other users and noise, so the receiver needs to perform SIC (successive interference cancellation) to eliminate interference. The transmitter allocates power based on the channel differences between users, making it easier to decode signals from users with more considerable channel differences.
Figure 5 has verified that the improved GMM clustering algorithm is more effective in identifying strip-like clusters with more significant user differences. To demonstrate the performance improvement of this algorithm for the ISAC system, we compare the ISAC system using this algorithm with the NOMA-ISAC system, OMA-ISAC system, and the ideal NOMA-ISAC system. As shown in
Figure 8, in the ideal NOMA-ISAC system, communication and sensing operate independently and do not affect each other. As shown in Equation (9), the ideal spectral efficiency and sensing power are obtained by excluding sensing and communication from the problem. Since NOMA allows multiple users to communicate simultaneously on the same frequency band, it can significantly improve spectrum utilization, thereby supporting more user connections and enhancing the spectral efficiency of the ISAC system. In contrast, the OMA-ISAC system requires orthogonal resources, resulting in reduced system capacity and spectral efficiency compared to the NOMA-ISAC system, with a significant gap compared to the ideal NOMA-ISAC system. The introduction of the improved GMM algorithm into the NOMA-ISAC system aims to enhance its channel quality, allowing different users within the same cluster to be effectively distinguished. This leads to a 4.3% increase in spectral efficiency compared to the OMA-ISAC system, further improving the overall system performance.
Figure 9a and
Figure 9b show the beam intensity of the base station’s transmitted beam as a function of angle in Cartesian and polar coordinates, respectively. In the Cartesian coordinate system, the beam energy of the NOMA-ISAC system is relatively concentrated in the target sensing directions of 45° and −45°, indicating that under the QoS constraints of numerous communication users, the system can still maintain good sensing performance. However, due to interference between multiple users, the base station’s beam covers multiple clusters, preventing the absolute concentration of beam intensity in the target sensing direction. The polar coordinates also show the beam intensity in the target sensing direction. It can be observed that in the OMA communication-based ISAC system, there is a significant energy leakage in the target direction. This occurs because as the number of communication users increases, the system becomes overloaded, resulting in insufficient spatial degrees of freedom, increased interference between users, and a substantial reduction in beam intensity in the target direction. The improved GMM clustering algorithm effectively reduces the system burden, increases the channel gain difference among users within clusters, and further enhances the SIC decoding efficiency. Compared to traditional communication–sensing integrated systems, beam intensity improves by approximately 5.4%, and in NOMA communication, users can share spectrum resources through power multiplexing, thus enhancing spectral efficiency.
6. Conclusions
To enhance the performance of the ISAC system, this paper employs non-orthogonal multiple access (NOMA) and proposes an improved GMM user clustering algorithm. The goal is to maximize the weighted sum of the communication–sensing objective function. This is achieved by using fractional programming to decouple and convert the original problem into a convex problem for optimization. The simulation results show that the NOMA-assisted ISAC system can reduce user interference during system overload. The improved GMM user clustering algorithm retains the algorithm’s ability to capture the distribution of users within clusters and enhances the system’s ability to optimize the sensing target signal. It yields more uniform strip-shaped clusters, improving bandwidth reuse and reducing serial interference, resulting in a 4.3% and 5.4% increase in spectral efficiency and beam strength, respectively, compared to traditional ISAC systems. Additionally, fractional programming effectively enhances power gain and channel gain, further improving the performance of the ISAC system.
This study only considers the case where the base station beam covers multiple-user clusters. Future research will design base station beams tailored to each user cluster’s characteristics.