AMCFCN: attentive multi-view contrastive fusion clustering net

Formulation of the problem

In this section, we introduce the goal of our study. The objective of AMCFCN is to go from a dataset consisting of N data samples, which comprises Q views $\left\{{\mathrm{M}}^{\left(1\right)},{\mathrm{M}}^{\left(2\right)}\cdots ,{\mathrm{M}}^{\left(\mathrm{Q}\right)}\right\}$, where ${\mathrm{M}}^{\left(\mathrm{i}\right)}\in {\mathbb{R}}^{\mathrm{N}\times \mathrm{d}\left({\mathrm{M}}^{\left(\mathrm{i}\right)}\right)}$ represents data samples of the dimension $\mathrm{d}\left({\mathrm{M}}^{\left(\mathrm{i}\right)}\right)$ from the i-th views. The aim is to derive a set of view-common representations denoted as G and subsequently cluster the N data samples into c clusters using this view-common representation. The proposed method AMCFCN consists of Q view-specific encoder networks $\mathrm{E}\left(\cdot \right)$, Q hybrid attentive module $\mathrm{A}\left(\cdot \right)$, a fusion module, and a clustering head $\mathrm{C}\left(\cdot \right)$ as illustrated in Fig. 1.

The AMCFCN contains Q view-specific encoder networks $\mathrm{E}\left(\cdot \right)$, which extracts view-specific feature maps $\left\{{{\mathrm{X}}^{\left(\mathrm{i}\right)}}_{\mathrm{i}=1}^{\mathrm{Q}}\right\}$ from multi-view data $\left\{{{\mathrm{M}}^{\left(\mathrm{i}\right)}}_{\mathrm{i}=1}^{\mathrm{Q}}\right\}$; ${\mathrm{A}}_{\mathrm{i}}\left(\cdot \right)$ is utilized to obtain view-specific robust representations $\left\{{{\mathrm{Y}}^{\left(\mathrm{i}\right)}}_{\mathrm{i}=1}^{\mathrm{Q}}\right\}$ from these view-specific feature maps $\left\{{{\mathrm{X}}^{\left(\mathrm{i}\right)}}_{\mathrm{i}=1}^{\mathrm{Q}}\right\}$; $\mathrm{f}\left(\cdot \right)$ is responsible for generating the view-common representation by amalgamating the view-specific representations into G. In COATS, we optimize the view-common representation G by taking G and $\left\{{{\mathrm{Y}}^{\left(\mathrm{i}\right)}}_{\mathrm{i}=1}^{\mathrm{Q}}\right\}$ as inputs. Finally, $\mathrm{C}\left(\cdot \right)$ applies off-the-shelf clustering methods on G to evaluate the effectiveness of the view-common representation G. Below we describe the functions of each module in detail.

View-specific encode network

$\mathrm{E}\left(\cdot \right):$ The survey (Zhu et al., 2023) provides a summary of feature extraction methods applicable to multimodal domains, encompassing text, vision, and audio. The survey (Zhu et al., 2023) recommends the utilization of a Convolutional Neural Network (CNN) for feature extraction. CNN is preferred due to its capacity to alleviate the laborious process of manual feature extraction and its effectiveness in extracting features from images. The network structure diagram of CNN is depicted in Fig. 2. $\mathrm{E}\left(\cdot \right)$ transforms the input image data ${{\mathrm{M}}^{\left(\mathrm{i}\right)}}_{\mathrm{i}=1}^{\mathrm{Q}}$, which is converted into the feature map type $\left\{{{\mathrm{X}}^{\left(\mathrm{i}\right)}}_{\mathrm{i}=1}^{\mathrm{Q}}\right\}\in {\mathbb{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$, as follows: (1)${\displaystyle {\mathrm{X}}^{\left(\mathrm{i}\right)}={\mathrm{E}}^{\left(\mathrm{i}\right)}\left({\mathrm{M}}^{\mathrm{i}};{\mathrm{W}}_{\mathrm{E}}^{\left(\mathrm{i}\right)}\right),\text{where}\mathrm{i}=1,2,\dots ,\mathrm{Q}}$ here, ${\mathrm{W}}_{\mathrm{E}}^{\left(\mathrm{i}\right)}$ denotes the weight of the i-th view encoder ${\mathrm{E}}^{\left(\mathrm{i}\right)}\left(\cdot \right)$.

Hybrid attentive module

In this subsection, we introduce the hybrid attentive module of detail. The hybrid attentive module is shown in Fig. 3 and contains the channel attention module, the spatial attention module, and positional attention module, and the dimensionality reduction module T() mentioned in the previous section. We first introduce the roles of these modules, before detailing their methods. We go over these three components in the following.

The inner modules of the hybrid attentive module play distinct roles. The channel attention module of the hybrid attentive module enhances feature refinement by generating channel attention maps, aiding in understanding the “content” being learned. Simultaneously, the spatial attention module of the hybrid attentive module determines the “location” of features through the generation of spatial attention maps along the spatial dimension. To mitigate long-distance dependency issues, we adopt the approach of Hou, Zhou & Feng (2021), embedding location features into the channel within the location module. The positional attention module of the hybrid attentive module addresses the long-range dependency problem, ensuring the retention of positional information. Lastly, the dimensionality reduction module converts the 3D feature map HA ∈ ℝ^C×H×W into matrix form $\mathrm{Y}\in {\mathbb{R}}^{\mathrm{N}\times \mathrm{d}\left(\mathrm{Y}\right)}$.

The hybrid attentive module enhances downstream tasks by focusing on feature maps with comprehensive channel, spatial, and positional information, thereby obtaining view-specific robust representations. To prevent convergence to a trivial solution, the hybrid attentive module employs a skip connection and decoupling channel, spatial, and positional information. The overall attention process is summarized in Eq. (5), where ⊗ is the element-wise multiplication, X ∈ ℝ^C×H×W is the view-specific feature map, P ∈ ℝ^C×H×W is the channel attention information, P′ ∈ ℝ^C×H×W is the spatial attention information, P^′′ ∈ ℝ^C×H×W is the positional attention information, and HA ∈ ℝ^C×H×W is the refined output. (5)${\displaystyle \mathrm{HA}=\mathrm{X}⨂\left(\mathrm{P}+{\mathrm{P}}^{{}^{\prime }}+{\mathrm{P}}^{{}^{\prime \prime }}\right)}$

Table 1 contains pertinent notations for this section. Figure 3 illustrates the entire process, while comprehensive descriptions of each attention module follow.

Channel attention module

We describe the flow of feature maps in the channel attention module, along with the expression of relevant formulae. Assuming we are initially given a view-specific feature map X ∈ ℝ^C×H×W. Following Woo et al. (2018), the channel attention infers a one-dimensional channel attention map z ∈ ℝ^C×1×1 which represents the importance of each channel in the feature map, as shown in Eq. (6). (6)${\displaystyle z=\delta \left({\text{Conv}}^{1\times 1}\left({\mathrm{GAP}}^{\mathrm{C}}\left(\mathrm{X}\right)\right)+{\text{Conv}}^{1\times 1}\left({\mathrm{GMP}}^{\mathrm{C}}\left(\mathrm{X}\right)\right)\right)}$ X ∈ ℝ^C×H×W possesses a total of C channels, while z ∈ ℝ^C×1×1 signifies a weight for each channel. Therefore, z ∈ ℝ^C×1×1 is a channel weighting information employed to allocate weights to each channel within a network. This empowers the network to prioritize channels with higher weights, thereby augmenting its focus on essential information. Therefore, Let the one-dimensional channel attention graph z ∈ ℝ^C×1×1 be corrected for the feature map X ∈ ℝ^C×H×W by correcting it with the feature map z_c ∈ ℝ^C×H×W, i.e., letting the two be multiplied element-by-element to obtain a channel attention graph-weighted feature map z_c ∈ ℝ^C×H×W, as follows in Equation (7). The channel attention module accentuates the significance of particular channels in the overall computation process. (7)${\displaystyle z_c=\left(z\otimes \mathrm{X}\right).}$

However, previous work (He et al., 2016) shows that as the network becomes deeper (i.e., the number of layers increases), the network may learn mundane solutions. For this reason, we introduce skip connections. As a result, the channel attention module of the hybrid attentive module produces the feature graph information P ∈ ℝ^C×H×W, as shown in Eq. (8). (8)${\displaystyle \mathrm{P}=\mathrm{X}+z_c}$

Spatial attention module

We detail the flow of feature maps in the spatial attention module as well as the expression of formulas. We initiate the process by applying both the GAP and GMP operations to the weighted feature maps P ∈ ℝ^C×H×W derived from the channel attention maps, performed along the channel dimensions. This yields the feature description operators, known as GAP^s, another called GMP^s. Subsequently, we merge GAP^s and GMP^s, and apply a 7 × 7 convolution operation to reduce dimensionality. Next, we determine the direction of the two-dimensional spatial attention map z′ ∈ ℝ^1×H×W by applying the sigmoid activation function, which plays a crucial role in highlighting relevant features, as shown in Eq. (9). (9)${\displaystyle z^{{}^{\prime }}=\delta \left({\text{Conv}}^{7\times 7}\left({\mathrm{GAP}}^{\mathrm{s}}\left(\mathrm{P}\right);{\mathrm{GMP}}^{\mathrm{s}}\left(\mathrm{P}\right)\right)\right)}$ z′ ∈ ℝ^1×H×W is a 2D weight map that records the magnitudes of location weights within the process. This facilitates the network in attending to regions with prominent features. The large weights of the 2D maps represent stronger features. Following Woo et al. (2018), we rectify the 2D spatial attention map, z′ ∈ ℝ^1×H×W, by correcting it with P ∈ ℝ^C×H×W through the channel attention module. In simple terms, these two entities undergo element-wise multiplication, leading to the creation of a channel attention map-weighted feature map denoted as z_s ∈ ℝ^C×H×W, as shown in Eq. (10). (10)${\displaystyle z_s=z^{{}^{\prime }}\otimes \mathrm{P}}$

We add the above-generated attention information to the original view-specific feature map X. This is to prevent the attention information from focusing on “biased” information, so the feature map information P′ ∈ ℝ^C×H×W generated by the channel attention module of the hybrid attentive module is as shown in Eq. (11). (11)${\displaystyle {\mathrm{P}}^{{}^{\prime }}=\mathrm{X}+z_s}$

Positional attention module

The following is a detailed description of the workflow and formula expression of the positional attention module. We put P′ ∈ ℝ^C×H×W, pooling along two different directions of the feature map with two pooling kernels (H, 1) and (1, W) yields two high direction embedded information feature maps ${\mathrm{I}}_{\mathrm{c}}^{\mathrm{h}}\in {\mathbb{R}}^{\mathrm{C}\times \mathrm{H}\times 1}$, and wide direction embedded information feature maps embedded information feature map ${\mathrm{I}}_{\mathrm{c}}^{\mathrm{w}}\in {\mathbb{R}}^{\mathrm{C}\times 1\times \mathrm{W}}$, respectively. Therefore, the output of the c-th channel at height h is expressed as in Eq. (12). (12)${\displaystyle {\mathrm{I}}_{\mathrm{c}}^{\mathrm{h}}\left(\mathrm{h}\right)=\frac{1}{\mathrm{w}}\sum _{\mathrm{i}=0}^{\mathrm{W}}{\mathrm{P}}^{{}^{\prime }}\left(\mathrm{h},\mathrm{i}\right).}$

Similarly, the output of the c-th channel with width w is expressed as in Eq. (13). (13)${\displaystyle {\mathrm{I}}_{\mathrm{c}}^{\mathrm{w}}\left(\mathrm{w}\right)=\frac{1}{\mathrm{H}}\sum _{\mathrm{j}=0}^{\mathrm{H}}{\mathrm{P}}^{{}^{\prime }}\left(\mathrm{w},\mathrm{j}\right).}$

To establish cross-channel links, we concatenate the two embedded information feature maps obtained along the spatial dimension to obtain comprehensive global information. Subsequently, a ReLU activation follows a 1 × 1 convolutional transformation, resulting in f_z ∈ ℝ^C×1×(W+H), as illustrated in Eq. (14). (14)${\displaystyle f_{\mathrm{z}}=\sigma \left({\text{Conv}}^{1\times 1}\left({\mathrm{I}}_{\mathrm{c}}^{\mathrm{h}}\left(\mathrm{h}\right);{\mathrm{I}}_{\mathrm{c}}^{\mathrm{w}}\left(\mathrm{w}\right)\right)\right)}$

This is done to subsequently perform a split operation along the spatial dimension to obtain two separated feature maps f^h ∈ ℝ^C×H×1, andf^w ∈ ℝ^C×1×W, as shown in Eq. (15). (15)${\displaystyle f^{\mathrm{h}},f^{\mathrm{w}}=\text{split}\left({\mathrm{f}}_{\mathrm{z}}\right).}$

Then, the 1 × 1 convolution variant is sigmoid activated to obtain the positional information feature maps located on the width and the height z_h ∈ ℝ^C×H×1 and z_w ∈ ℝ^C×1×W. Its formula is obtained as: (16)${\displaystyle z_{\mathrm{h}}=\delta \left({\text{Conv}}_{\mathrm{h}}^{1\times 1}\left({\mathrm{f}}^{\mathrm{h}}\right)\right),}$ (17)${\displaystyle z_{\mathrm{w}}=\delta \left({\text{Conv}}_{\mathrm{w}}^{1\times 1}\left({\mathrm{f}}^{\mathrm{w}}\right)\right).}$

Finally, let the positional weighting information z_h and z_w be corrected after dot multiplication with the feature map P′ ∈ ℝ^C×H×W. That is, multiply them element-wise to obtain positional weighting information with the positional attention information of the embedded channel $z_{\mathrm{p}}\left(\mathrm{i},\mathrm{j}\right)\in {\mathbb{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$, as shown in Eq. (18). (18)${\displaystyle z_{\mathrm{p}}\left(\mathrm{i},\mathrm{j}\right)=z_{\mathrm{h}}\left(\mathrm{j}\right)\otimes z_{\mathrm{w}}\left(\mathrm{i}\right)\otimes {\mathrm{P}}^{{}^{\prime }}}$

To prevent the positional attention information from focusing on “biased” information, we add to the positional attention information the original view-specific feature map X ∈ ℝ^C×H×W. Thus, the feature map information P^′′ ∈ ℝ^C×H×W, produced by the positional attention module of the hybrid attentive module, as shown in Eq. (19). (19)${\displaystyle {\mathrm{P}}^{{}^{\prime \prime }}=\mathrm{X}+z_{\mathrm{p}}\left(\mathrm{i},\mathrm{j}\right)}$

All in all, our designed hybrid attentive module efficiently allocates effective feature weights, and we summarize its effectiveness in two aspects. (1) The produced weighting information, z ∈ ℝ^C×1×1, z′ ∈ ℝ^1×H×W, z_h ∈ ℝ^C×H×1, z_w ∈ ℝ^C×1×W, capture comprehensive attention to feature maps from multiple perspectives: channel dimension (c-th), spatial dimensions (h and w-th), and high and wide positional dimensions (c, h-th and c, w-th). This differs from previous attention mechanisms that focused on one or two dimensions. We exhaustively verify in the ablation experiments section that features map weighting that considers full aspects will be better than one that considers only one or two dimensions. (2) Simultaneously, a crucial design aspect of the hybrid attention mechanism is the addition of the original feature map X after generating each attention map. For instance, assuming z ∈ ℝ^3×1×1 with weight distribution 0, 0.9, 0.1, where the channel with weight 0 is significant, we add the original feature map to correct biased information. This design ensures correction even when attention weights deviate towards incorrect information. Consequently, we attribute the success of the hybrid attentive module to these two design elements.

Contrastive attentive strategy

In this chapter, we present the contrastive attentive strategy (COATS), as illustrated in Fig. 4. COATS processes the input view-specific robust representations and the view-common representation by passing them through the mapping head $\mathrm{PRO}\left(\cdot \right)$. Subsequently, it computes the cosine similarity between the two representations and calculates the contrastive loss based on this cosine similarity. It’s worth noting that COATS does not directly align view-specific representations with each other, which helps in preserving the completeness of the views. The mapping head $\mathrm{PRO}\left(\cdot \right)$ follows the same settings as previously defined in our work (Chen et al., 2020b) for constructing the projection header.

We conclude that the contrastive attentive strategy has two advantages: (1) it ensures that the intrinsic structure of the original view is not destroyed, and (2) it weights the view-specific representations to obtain a robust view-specific representation and thus a more comprehensive view-common representation. Following Chen et al. (2020b), we compute the similarity between view-specific robust representations (after projection head $\mathrm{PRO}\left(\cdot \right)$) and the view-common representation of the view (after projection head $\mathrm{PRO}\left(\cdot \right)$) as cosine similarity, as shown in Eq. (20). (20)${\displaystyle {\mathrm{d}}_{\mathrm{i},\mathrm{j}}=\frac{{\left({\mathrm{G}}_{\mathrm{i}}\right)}^{⊺}\left({\mathrm{Y}}_{\mathrm{j}}^{\left(\mathrm{q}\right)}\right)}{\parallel {\mathrm{G}}_{\mathrm{i}}\parallel \cdot \parallel {\mathrm{Y}}_{\mathrm{j}}^{\left(\mathrm{q}\right)}\parallel }}$

In this context, the notation d_i,j(i ≠ j) denotes the calculation of similarity for a pair of negative examples, while d_i,i signifies the computation of similarity for a pair of positive examples. Intuitively, d_i,j means that at this point the view-common representation G_i computes the similarity with the view-specific robust representations of the negative examples ${\mathrm{Y}}_{\mathrm{j}}^{\left(\mathrm{q}\right)}$, and vice versa. Following Chen et al. (2020b), we design a contrastive learning loss function for AMCFCN, as shown in the following: (21)${\displaystyle {\mathrm{L}}_{\mathrm{sim}}^{\text{AMCFCN}}=-\sum _{\mathrm{i}=1}^{\mathrm{N}}log\frac{exp\left(\frac{{\mathrm{d}}_{\mathrm{i},\mathrm{i}}}{\tau }\right)}{\sum _{\mathrm{j}}^{2\mathrm{N}}{\mathbb{l}}_{\left[\mathrm{i}\ne \mathrm{j}\right]}exp\left(\frac{{\mathrm{d}}_{\mathrm{i},\mathrm{j}}}{\tau }\right)},}$ (22)${\displaystyle {\mathrm{L}}_{\mathrm{c}}=\sum _{\mathrm{q}=1}^{\mathrm{Q}}{\mathrm{L}}_{\mathrm{sim}}^{\text{AMCFCN}}.}$

The exp() denotes an exponential function with a natural number e as the base. The symbol N represents the number of samples, while (i, i) and (i, j) represent a pair of positive and a pair of negative examples, respectively; ${\mathbb{l}}_{\left[\mathrm{i}\ne \mathrm{j}\right]}\in \left\{0,1\right\}$ denotes an indicator function that takes the value of one if i = j, and τ denotes a temperature constant. In our experiments, we adopt the optimal setting of τ = 0.1 for the temperature parameter suggested by Ke et al. (2021).

The objective function

The total loss function is used to measure the degree of difference between the predicted and true values of the model. Before this, the whole network process lacked a certain steering direction, and we chose the clustering steering instead of the task, and we empirically used the off-the-shelf deep divergence-based DDC (Kampffmeyer et al., 2019) online clustering task. It assigns the consistency representation G to a compact space, i.e., a soft label can be obtained from this go, i.e., B = C(G). To reduce the computational cost, AMCFCN only uses DDC to constrain the view-common representation G.

The loss function of DDC is described below. DDC uses Cauchy–Schwarz scatter (CS-scatter) to enhance the separability and compactness of the clustering. We predefine some theoretical foundations, where B is an n × k cluster assignment matrix and b_i is the i-th column of matrix B, and ϱ is Gaussian kernel bandwidth, defaulting 0.15. 𝕂 is computed as in Eq. (23). m_i,j is computed as in Eq. (24), where e_j is corner j of the standard simplex on ℝ^k. (23)${\displaystyle {\mathbb{K}}_{\mathrm{i},\mathrm{j}}=exp\left(-\parallel {\mathrm{G}}_{\mathrm{i}}-{\mathrm{G}}_{\mathrm{j}}{\parallel }^2/{\left(2\varrho \right)}^2\right)}$ (24)${\displaystyle {\mathrm{m}}_{\mathrm{i},\mathrm{j}}=exp\left(-\parallel {\mathrm{b}}_{\mathrm{i}}-{\mathrm{e}}_{\mathrm{j}}{\parallel }^2\right)}$ triu(BB)^T denotes B^T the upper strictly triangular element, and 𝕂 denotes kernel similarity matrix computed by 𝕂_i,j. Then the loss function of DDC consists of three terms as shown below: (25)${\displaystyle {\mathrm{L}}_{\mathrm{ddc}}=\sum _{\mathrm{i}=1}^{\mathrm{k}-1}\sum _{\mathrm{j}>\mathrm{i}}^{\mathrm{k}}\frac{{\mathrm{b}}_{\mathrm{i}}^{\mathrm{T}}\mathbb{K}{\mathrm{b}}_{\mathrm{j}}}{\sqrt{{\mathrm{b}}_{\mathrm{i}}^{\mathrm{T}}\mathbb{K}{\mathrm{b}}_{\mathrm{j}}{\mathrm{b}}_{\mathrm{j}}^{\mathrm{T}}\mathbb{K}{\mathrm{b}}_{\mathrm{i}}}}+\sum _{\mathrm{i}=1}^{\mathrm{k}-1}\sum _{\mathrm{j}>\mathrm{i}}^{\mathrm{k}}\frac{{\mathrm{m}}_{\mathrm{i}}^{\mathrm{T}}\mathbb{K}{\mathrm{m}}_{\mathrm{j}}}{\sqrt{{\mathrm{m}}_{\mathrm{i}}^{\mathrm{T}}\mathbb{K}{\mathrm{m}}_{\mathrm{j}}{\mathrm{m}}_{\mathrm{j}}^{\mathrm{T}}\mathbb{K}{\mathrm{m}}_{\mathrm{i}}}}+\text{triu}{\left(\mathrm{BB}\right)}^{\mathrm{T}}}$

In summary, we combined the objective function of AMCFCN as follows: (26)${\displaystyle {\mathrm{L}}_{\mathrm{sum}}={\mathrm{L}}_{\mathrm{ddc}}+\vartheta {\mathrm{L}}_{\mathrm{c}},}$ here, ϑ is a trade-off parameter intended to restrict the contrastive loss term from becoming excessively large. ϑ is meticulously determined in the experimental section.

Dataset

We evaluate the model AMCFCN on four public multi-view datasets and compare it to seven other mainstream models. The four datasets are.

1. Edge-MNIST (E-MNIST) (Liu & Tuzel, 2016): which is from the staff of the US Census Bureau. This is a large and widely used benchmark dataset. A set of public benchmark datasets consists of 60,000 handwritten digital images (10 categories) of 28 × 28 pixels. In this dataset, there are two views containing the original figure and the edge detection versions, respectively.

2. Edge-Fashion MNIST (E-FMNIST) (Xiao, Rasul & Vollgraf, 2017): This is provided by Zalando’s research department (German fashion tech company). This dataset, characterized by its diverse range of clothing items in terms of shapes, colors, and patterns, presents a higher level of complexity compared to the standard MNIST dataset. We create a second view by running the same edge detector used to create Edge-MNIST.

3. COIL-20 (Nene, Nayar & Murase, 0000): This is Columbia University’s description of grayscale images containing 20 categories from separate perspectives. We create a three-view dataset by randomly grouping the coil-20 dataset into three sets. The details of creating the three views are covered in the next paragraph.

4. COIL-100 (Geusebroek, Burghouts & Smeulders, 2005): This is from Columbia University; separate RGB images containing 100 categories were taken from different angles, and we created a three-view dataset by randomly grouping the coil-100 dataset into three sets. The coil-100 constructs the three views in the same way as the coil-20.

We randomize the objects of a category using a random arrangement technique to generate a random index for image generation, the purpose of utilizing random is for the diversity of the selected view images. Multiple views are constructed from the generated indexes. For example, in Coil-20, a category comprises 72 images, and when three views are constructed, the index can form a 3 × 24 matrix. The selected images are organized into three views and stored as an array that represents three different views of the images. In short, items for coil-20 are generated by randomly selecting images related to the project and organizing them into three different views.

The details are shown in Table 2:

Implementation details

The experimental system runs on Ubuntu 18.04.6, utilizing hardware comprising an NVIDIA RTX 3090 GPU with 16GB of memory and an AMD EPYC 7702 64-core Processor. The software includes CUDA 11.4 and Python 3.9.7, while the deep learning framework PyTorch (Paszke et al., 2019) is employed for model development and validation on a publicly available benchmark dataset.

The model is validated on a publicly available benchmark dataset. For the MNIST class dataset, key hyperparameters are configured as follows: the training batch epoch is set to 100; model optimization employs the ADAM optimizer with default parameters; the batch size is fixed at 100; and reference to specific encoder CNN structures and configurations in Abavisani & Patel (2018). The training dataset’s batch size is 128; the initial learning rate is 0.001; and the hidden representation dimensionality is 288. On the other hand, for the COIL class dataset, we adopt AlexNet (Krizhevsky, Sutskever & Hinton, 2017) as the view-specific encoder, with a batch size of 24, an initial learning rate of 0.0001, and a hidden representation size of 1,024.

Model performance

In this subsection, we conduct a series of experiments to evaluate the performance of the model, and we explain our experimental design and results. We introduce model evaluation metrics, model stability experiments, model visualization experiments, and model comparison experiments.

Stability experiments

In the field of unsupervised learning, stability plays a crucial role in its performance and reliability. The stabilization of the clustering is indirectly illustrated by the evolution of ACC and NMI. The stability evaluation experiments for AMCFCN are conducted by monitoring the rate of change in these two metrics over one run. AMCFCN and the next best model (CONAN) are executed on two datasets, there are E-MNIST and COIL-20, from which the rate of change of ACC and NMI metrics is obtained, as follows in Fig. 5.

As shown in Fig. 5, the rate of change of the two metrics of CONAN is very large, which indirectly indicates that the performance of CONAN is not stable enough. Compared to CONAN, AMCFCN exhibits elevated levels in both metrics with minimal rates of change. It is noteworthy that the contrastive attentive strategy yields a robust view-common representation, contributing to the stability demonstrated by AMCFCN.

Visualization experiments

To show the effectiveness of the contrastive attentive strategy, we visualize the view-common representation. We demonstrate the clustering process on the E-MNIST and COIL-20 datasets, as depicted in Figs. 6 and 7. The following T-SNE (Vander Maaten & Hinton, 2008), is a dimensionality reduction technique that visualizes high-dimensional data by preserving pairwise similarities in a lower-dimensional space. AMCFCN employs T-SNE to project the view-common representation onto the 2D plane for both the E-MNIST and E-FMNIST datasets. The visualization reveals that the structure specific to clustering becomes more compact and separable as the training period is extended. The two experimental studies demonstrate that the view-specific robust representation obtained through the hybrid attentive module enhances the view-common representation through view-specific representation fusion, which in turn improves the stability of the multi-view clustering.

Comparison experiments

AMCFCN evaluates its performance against seven current mainstream multi-view clustering models to assess its effectiveness. The comparison models include:

1. Deep Canonical Correlation Analysis (DCCA) (Gao et al., 2020);

2. Deep Canonically Correlated Auto-encoders (DCCAE) (Wang et al., 2015);

3. Deep Multi-modal Subspace Clustering (DMSC) (Yang et al., 2021);

4. Deep Adversarial Multi-view Clustering (DAMC) (Li et al., 2019);

5. End-to-end Adversarial attention network for Multi-modal Clustering (EAMC) (Zhou & Shen, 2020);

6. Autoencoder in autoencoder networks (Ae²-Nets) (Zhang, Liu & Fu, 2019);

7. Contrastive Fusion Networks for Multi-view Clustering (CONAN) (Ke et al., 2021);

To mitigate the influence of randomness in experimental outcomes, AMCFCN conducts five repetitions of training on each dataset and utilizes the average values for evaluation. We conduct a comparative experiment involving seven classical deep models, evaluating AMCFCN alongside the baseline model across four datasets. The resulting clustering performance, expressed in percentages (%), is presented in Table 3. As evident from the comparative results in the table, AMCFCN outperforms the suboptimal model on the Coil-20 and E-MNIST datasets across two metrics, achieving improvements of 10% and 4.9%, another 4.6% and 3.1%, respectively in ACC and NMI.

Table 3:

Comparison results of different data sets in different models (%).

Data set	E-MNIST		Coil-20		E-FMNIST		Coil-100
Evaluation indicators	ACC	NMI	ACC	NMI	ACC	NMI	ACC	NMI
DCCA (Gao et al., 2020)	47.6	44.3			42.6	41.9
DCCAE (Wang et al., 2015)	50.8	47.5			45.8	43.6
DMSC (Yang et al., 2021)	65.3	61.4	65.1	72.0	52.4	51.8	54.1	53.8
DAMC (Li et al., 2019)	64.6	59.4	67.2	72.9	49.5	48.7	52.6	70.1
Ae2 -Nets (Zhang, Liu & Fu, 2019)	53.7	46.3	71.7	81.2	41.0	39.2	54.1	75.3
EAMC (Zhou & Shen, 2020)	66.8	62.8	69.0	75.3	54.1	62.2	51.8	68.0
CONAN (Ke et al., 2021)	90.5	86.1	72.5	80.3	59.2	55.2	55.6	77.4
AMCFCN (OURS)	95.1	89.2	82.2	85.2	63.2	55.8	66.4	84.8

DOI: 10.7717/peerjcs.1906/table-3

Notes:

Large-scale or high-dimensional datasets require substantial computational resources. DCCA and DCCAE are applicable exclusively to two-view datasets. Bolded values represent the highest value of an indicator for this experiment. ‘p’ denotes the inclusion of the current module, while ’-’ signifies its exclusion.

To make it easier to observe AMCFCN, we visualize the seven models and AMCFCN in Fig. 8. AMCFCN is represented in violet. Higher bars in the figure represent better performance, and it can be seen from the figure that the bar levels of AMCFCN are basically higher than those of the seven methods. Thus, AMCFCN outperforms the mainstream methods on most datasets, but its NMI metrics on the E-FMNIST dataset are slightly lower than those achieved by EAMC.

Ablation experiments

In this subsection, we conduct a total of two ablation experiments. The first ablation experiments involve the hybrid attentive module proposed in the article, with the ablation focus on the three types of attention within the module. The other ablation experiments assess the contrastive attentive strategy’s impact when contrastive learning and attention mechanisms are absent in AMCFCN. In the experimental data table below, ‘√’ denotes the inclusion of the current module, while ‘-’ signifies its exclusion.

The hybrid attentive module for AMCFCN Ablation experiments conducts ablation experiments on the EMNIST dataset, as detailed in Table 4. The results demonstrate that the hybrid attentive module internally achieves the best performance when all three attention modules are employed, with metrics of 95.1% and 89.2%, respectively. In contrast, the hybrid attentive module performed worst when relying only on the spatial and positional attention modules, with metrics of 82.1% and 75.5%, respectively. These ablation experiments demonstrate the robust noise suppression capability of the hybrid attentive module, especially when using three modules, which allows the hybrid attentive module to focus more comprehensively on the feature information of the feature map. This enhancement allows AMCFCN to improve its information capture, thus allowing the clustering to obtain a good result.

This ablation experiment aims to assess the contrastive attentive strategy. The experiment involves using the contrastive attentive strategy to obtain a view-common representation, followed by incomplete strategies in subsequent experiments. These incomplete strategies include one utilizing only the contrastive learning strategy, another employing the attention mechanism strategy, and the last one employing no specific strategy. To validate the superior performance of the proposed fusion strategy of the contrastive attentive strategy, ablation experiments conduct on four datasets, as detailed in Table 5. When employing the contrastive attentive strategy, AMCFCN consistently outperforms the incomplete strategies across both metrics on the various datasets. The improvement is particularly noteworthy on the COIL-100 dataset, where the use of the contrastive attentive strategy leads to a 27.5% increase in ACC and a 17.4% increase in NMI compared to the no-strategy approach.

We compare the differences in model learning performance under each of the four steps, as shown in Fig. 9. Specifically, Step-A is the model with the hybrid attentive module, Step-B is the model with contrastive learning, and Step-C is the model with none of the why strategy. In Fig. 9, where higher bars denote better performance, it is evident that AMCFCN achieves the most favorable results when utilizing COATS. The utilization of the hybrid attentive strategy enhances AMCFCN’s performance compared to models employing incomplete strategies (Step-A, Step-B) or no strategies (Step-C) across all four datasets.

Hyperparametric analysis of the objective function

We conduct a hyperparametric analysis experimental study for ϑ of Eq. (26). ϑ represents the hyperparameter for the contrastive loss within the objective function, and we analyze the study on the E-MNIST dataset, utilizing the optimal value determined for ϑ as the parameter value for all experiments in this context. Then, AMCFCN is analyzed and studied on the EMNIST dataset, employing values ranging from 10⁻³ to 10. The specific experimental results are displayed in Table 6.

It is observed that when ϑ is set to 10, the ACC and NMI are 86.7% and 80.1%, respectively, resulting in the best total loss value of AMCFCN reaching 184.4. Conversely, when ϑ is set to 0.001, the ACC and NMI are 87.4% and 85.4%, respectively, with a total loss of only 0.64. Values of ϑ that are excessively large or small can cause one of the loss components to dominate network training, contrary to our intended hyperparameter setting. With ϑ set to 0.01, the ACC and NMI are 95.1% and 85.4%, respectively, while the total loss is 0.82. Therefore, we consider ϑ = 0.01 an appropriate value, ensuring that the contrastive loss and clustering loss of AMCFCN are of comparable magnitude, allowing both loss components to guide network training effectively.

Convergence analysis

In this subsection, we focus on the convergence of the methods. Some of the popular methods based on autoencoders, such as COMPLETER (Lin et al., 2021) and EMC-Nets (Ke et al., 2022), require 200 and 400 training epochs, respectively, to achieve good convergence. We discuss the convergence of AMCFCN by visualizing the change of loss values with training epochs, and the convergence analysis of AMCFCN loss values on four datasets is shown in Fig. 10.

The visualization analysis in Fig. 10 reveals that the loss value of AMCFCN consistently decreases with each epoch, ultimately converging to a stable target value by the 30th training epoch. The rapid and effective convergence of AMCFCN is attributed to the COATS. This strategy preserves the complete view-specific representations, eliminating the necessity to continually learn features from corrupted view-specific representations over time. In summary, the accelerated convergence of AMCFCN can be attributed to a contrastive attentive strategy. The strategy retains both view completeness and improves the robustness of the view-common representation, thus expediting the convergence process. AMCFCN requires about 30 training epochs to achieve convergence on the four datasets, which is much faster than the autoencoder-based methods to converge.