Semantic-visual shared knowledge graph for zero-shot learning

Zero-shot learning based on attributes

Attribute-based methods account for the largest proportion of zero-shot learning (ZSL) research since ZSL was first proposed in DAP (Lampert, Nickisch & Harmeling, 2009) in 2009 with a proposed dataset called AwA which contains pre-defined attributes. It principally annotates the attributes of images (such as whether there is a tail, hair color, etc.), then learn the semantic attribute characteristics of visual objects, and finally judge whether the attribute combinations are satisfied by the visual objects.

Following the design principles of AwA dataset, a series of benchmark datasets with pre-defined attributes are established today, including AwA2, CUB, SUN, FLO and aPY. Based on these benchmark datasets, ZSL has a blowout development with several milestones.

In 2013, with the development of semantic embedding technology, the first milestone of ZSL was the ALE model proposed by Akata et al. (2013) which can encode the attributes as semantic vectors, encode images as visual vectors, and then learn a function to calculate the similarity between semantic vectors and visual vectors, so as to match the corresponding mappings between attributes and images.

In 2017, with the development of deep learning technology in the field of visual computing, the second milestone of ZSL is the SAE model proposed by Kodirov, Xiang & Gong (2017). By using the Auto-Encoder network, it can encode more fine-grained attribute features and visual features, so as to better achieve “semantic-to-visual” matching. The overall performance of SAE is significantly improved compared with ALE.

In 2018, with the remarkable performance of Generative Adversarial Networks (GAN) (Goodfellow et al., 2020) in image processing, ZSL achieved the third milestone with the representative model GAZSL proposed by Zhu et al. (2018). It uses GAN to synthesize fake visual features from semantic features, and then match the fake visual features with the real visual features to predict the unseen objects.

Recently, some researchers have started to investigate how to mix various types of semantics (attributes, word2vec, text description, knowledge graph) to address further “domain shift” issues. For instance, Wang et al. (2021) learns discriminative classifiers using a variety of semantic viewpoints. In Xie et al. (2021) and Naeem et al. (2021), open knowledge is regarded as auxiliary or augmented semantics added with pre-defined features.

However, the above methods use pre-defined properties as their primary semantic source. As a result, these methods are essentially restricted to benchmark datasets with pre-defined features. The applicability of these methods in real-world environments devoid of pre-defined features is restricted to a small number of cases.

Zero-shot learning based on knowledge graph

Knowledge graph (KG) actually is a third part knowledge base that can provide semantic information for semantic-to-visual transformation in zero-shot learning. Knowledge graph (KG) actually constitutes a third-party knowledge base that can provide semantic information for semantic-to-visual transformation in zero-shot learning. Thus, KG is intuitively considered a substitution for pre-defined attributes. Benefiting from the development of GNN, GCNZ (Wang, Ye & Gupta, 2018a) creates a formal knowledge graph-based on WordNet to substitute pre-defined attributes and learn semantic embedding from structure information and word embedding. It is, indeed, the first work that tries to apply knowledge graphs and GNN as the backbone for the real-world dataset zero-shot learning task. Based on Wang, Ye & Gupta (2018a) and Kampffmeyer et al. (2019a) proposes a so-called Dense Graph Propagation (DGP) method to gather the semantic information through the relations of the knowledge graph. CL-DKG (Wang & Jiang, 2021a) applies contrastive learning on dual knowledge graph to learn the projection between semantic space and visual space without any pre-defined attributes. It exploits multiple knowledge relationships among classes simultaneously to learn robust and discriminative classifiers for unseen classes.

However, compared to the pre-defined features in the six benchmark datasets, the knowledge network used in GNN-based approaches (Wang, Ye & Gupta, 2018a; Kampffmeyer et al., 2019a; Liu et al., 2020; Wang et al., 2021) provides far less fine-grained semantics. In contrast to them, we propose a solution that enables fine-grained semantics to be extracted from raw images and shared with the existing knowledge graph in order to enhance recognition performance in the ZSL challenge.

Visual knowledge applied for zero-shot learning

To enhance the fine-grained semantic information for existing KG, the latest research tries to apply “visual knowledge” in zero-shot learning. Zhu, Wang & Saligrama (2019) propose a novel low-dimensional embedding for visual objects called “visually semantic” to narrow the semantic gap between high-dimensional visual space and semantic space in zero-shot learning. Xu et al. (2020) proposed a zero-shot learning framework that jointly learns discriminative visual semantics only using class-level attributes. Liu et al. (2021) propose a goal-oriented gaze estimation module (GEM) to improve the discriminative attribute localization based on the class-level attributes for ZSL. They introduce so-called “human gaze locations” to obtain new regions of visual semantics. Xie et al. (2020) extracts the relationships among visual regions to enhance the semantic information for both seen and unseen classes and then can lunch region-based relational reasoning for ZSL. Song & Zhang (2022) consists of two graphs constructed by semantic and visual representations respectively to enhance semantic representations.

However, the above methods continue to see visual semantics as auxiliary or augmented semantics. In contrast to them, we want to combine semantic and visual representations into a single common knowledge graph, so-called visual-semantic shared knowledge graph.

Problem definition

Knowledge graph-based Zero-Shot learning uses the knowledge-aided method to learn an image classification model from seen classes to predict unseen classes. First, given a knowledge graph $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$ , Then, we have X ∈ ℝ^N×F be the word-embedding set for all the nodes in $\mathcal{V}$, and A ∈ ℝ^N×N be the adjacency matrix transferred from $\mathcal{E}$. Here, F is the dimensions of the embedding. After, a graph representation model g_Θ(X, A, I_seen) is learned to transfer X to the graph node representation H ∈ ℝ^N×F′ supervised by I_seen. Here, I_seen ∈ ℝ^N×F′′ is the seen image feature set extracted by a CNN model. At last, an image classification model ${\mathcal{L}}_{\Theta }\left(H,I_{unseen}\right)$ is learned to classify I_unseen to the particular class according to the graph node representation H. Here, I_unseen ∈ ℝ^N×F′′ is the unseen image feature set extracted by a CNN model.

Semantic-visual shared knowledge graph

Semantic-visual shared knowledge graph (SVKG) is a multi-modal graph that contains both semantic embedding and visual embedding in the same graph. Let X represents the embedding set of the graph nodes with X_s denotes Word-Embedding nodes and X_v represents CNN-Embedding nodes. A represents the edge set while A_s denotes the edges among Word-Embedding nodes and A_v denotes the edges among CNN-Embedding nodes. SVKG is defined in Algorithm 1.

 
________________________________________________________________________________ 
Algorithm 1 Semantic-visual shared knowledge graph 
________________________________________________________________________________ 
Input: G = (V,E), seen images 
Output: SV KG 
 1:  Xs ←− Put X into Glove 
 2:  As ←− Put X into WordNet hyponym/hypernym 
 3:  Xv ←− Put seen images into EfficientDet 
 4:  As ←− Put Xv connecte with semantic object node 
 5:  X′ ←− Xs ∪ Xv 
 6:  A′ ←− As ∪ Av 
 7:  SV KG ←−{ X′, A′ } 
 8:  return  SV KG 
 _______________________________________________________________________________    

In SVKG, the semantic node set X_s is constructed from WordNet (https://wordnet.princeton.edu/) Noun words and embedded into word features by Glove (https://nlp.stanford.edu/projects/glove/). The edge matrix A_s is established by WordNet hyponym/hypernym links among these words. The visual node set X_v is obtained by using EfficientDet (Tan, Pang & Le, 2020) to detect fine-grained parts visual features (such as head, back, belly, breast, leg, etc.) from ImageNet-1k. Then, these visual nodes are connected to their semantic object node by the edge matrix A_v (such as “hasHead”, “hasBelly”, etc.). An example of SVKG about bird Finch is presented in Fig. 3.

Graph augmentation for SVKG

In the zero-shot learning process, semantic feature space X_s needs to be transferred into semantic-visual shared space H. However, the visual feature space (fine-grained visual parts) X_v might cause interference during X_s⟶H transfer process, leading to a serious over-fitting problem. Empirically, many data augmentation methods have shown to improve the generalization and robustness of the learning model (Zhao et al., 2021; Rong et al., 2020; Srivastava et al., 2014; Chen et al., 2020a). Thus, in this work, we introduce an extra graph node with zero-initialized features to augment SVKG. This zero-initialized node X_aug then links to all seen nodes by the edge matrix A_aug. After that, the node is indirectly connected to all visual feature nodes X_v through the corresponding seen nodes. It can moderate the strength of feature propagation from visual feature node X_v to alleviate the over-fitting problem. Augmentation graph SVKG_aug is defined in Eq. (1). (1)${\displaystyle {SVKG}_{aug}=\left(X^{\prime },A^{\prime }\right)X^{\prime }=X\cup X_{aug},A^{\prime }=A\cup A_{aug}}$

where X_aug is the augmentation node with zero-initialized feature. A_aug is the adjacency matrix to link X_aug to all seen nodes. And X′, A′ represents feature matrix and adjacency matrix of SVKG, respectively. An example of SVKG_aug is presented in Fig. 4.

Multi-modals graph representation learning

As explained in the ‘Problem Definition’ section, knowledge graph SVKG_aug needs to be represented in a particular feature space to support the downstream task (zero-shot image classification). In SVKG_aug, there are two feature modals. One is semantic feature modal {X_s generated from Glove-Embedding. The other is visual feature modal X_v extracted by EfficientDet. Thus, multi-modals graph representation learning is aimed to transfer both X_s and X_v into a semantic-visual shared feature modal H.

More specifically, we propose a dual ways propagation graph convolution network to transfer X_s and X_v into H, as defined in Eq. (2). ${\displaystyle X⟶H⟶H^{\prime }}$ (2)${\displaystyle H=g_{\Theta }\left(\sigma \left(D_s^{-1}\left(A_s\cup A_{aug}\right)X\right)\right)}$ ${\displaystyle H^{\prime }=g_{\Theta }\left(\sigma \left(D_v^{-1}\left(A_v\cup A_{aug}\right)H\right)\right)}$ where g_Θ is a Graph Convolutional Network (GCN) employed from Kipf & Welling (2017). $\sigma \left(\cdot \right)$ represents a nonlinear activation function. D is the diagonal degree matrix of A matrix, where D_(i,i) = ∑_jA_(i,j). H′ is the semantic-visual shared graph representation feature set that can be used for downstream task.

Cross-modals zero-shot learning

So far, we already obtained the final representation H′ for the knowledge graph SVKG_aug. Each $H_i^{\prime }$ represents the centroid feature of the corresponding class (this class could be seen class or unseen class). Here, H′ is knowledge graph feature modal. However, in the zero-shot image classification, the targets are the real images which is an image feature modal. Thus, the problem becomes a cross-modal transfer problem (Zhuang et al., 2021). Let I represents the real image features. I_seen and I_unseen denote seen classes features and unseen classes features relatively. I_i,seen represents the centroids feature of ith seen class. Then, we have following the loss function, Eq. (3), to force knowledge graph features H′ close to real image features I. (3)${\displaystyle H^{\prime }⟶differences⟵I\mathcal{L}=\frac{1}{N}\sum _{i=1}^N{\left(H_i^{\prime }-I_{i,seen}\right)}^2.}$

The base idea is simple. In the training process, the graph features H′ could be updated supervised by real image features I. We calculate the similarities for all ($H_i^{\prime }-I_{i,seen}$) pairs and use average similarity differences as the loss function to train the zero-shot classification model. Note that, in the experiment, a pre-trained Resnet-50 model is used to extract real image features I.

In the predicting process, we first use the same Resnet-50 model to extract unseen image feature I_unseen (Algorithm 2 Input). Then, I_unseen is input into the trained classification model to calculate the similarities with all graph features H′. To obtain the predicting result, Result_set records all the similarities between H′ and I_unseen, and the label_i of corresponding $H_i^{\prime }$ (Algorithm 2 2-6). After, the top-k highest similarities with labels are selected as the zero-shot classification result (Algorithm 2 7-9). Algorithm 2 presents the whole zero-shot predicting process.

 
________________________________________________________________________________________ 
Algorithm 2 Zero-shot Predicting Process 
_______________________________________________________________________________________ 
Input: H′, Iunseen 
Output: Top-k result 
 1: Resultset ←−{∅, ∅} 
 2: for H′i in H′ do 
 3:   Simi ←− H′i ⋅ Iunseen 
 4:   lableli ←− findLabel(H′i) 
 5:   Resultset ←−{Simi, lableli} 
 6: end for 
 7: Resultset ←− SortBySim(Resultset) 
 8: Resultset ←− Top-k(Resultset) 
 9: return  Resultset 
 ____________________________________________________________________________________    

It can be observed, the unseen image feature I_unseen is not used during the whole training process. In the predicting process, the input I_unseen is the first time the model touches the unseen images.

Datasets and evaluation metrics

The experiments are conducted on ImageNet (Deng et al., 2009), which is a real-world dataset and also the most large-scale benchmark for zero-shot image classification. The divisions for zero-shot tasks are 1K seen classes from ImageNet-2012-1K while 1.5K, 7.8K and 20K unseen classes for 2-hops, 3-hops and all from ImageNet. These divisions are grouped together in our experiments named “General” group. Here, n-hops means the most nth jumps to connect to other classes of ImageNet through the relations of a WordNet graph. For example, as showed in Fig. 2, given a seen class “Indigo Bunting”, 2-hops can be unseen classes “Ortolan” and “Finch”, but 3-hops will has extra unseen classes including “Chaffinch”, “Redpoll” and “Grosbeak”. Note that there is no overlap between the seen and unseen classes in the three divisions.

In order to show the effectiveness of visual feature X_v in SVKG, a “Detailed” group which contains four subsets (‘bird’, ‘snake’, ‘primate’ and ‘dog’ categories) are divided from ImageNet-1K and corresponding parts visual features X_v for all these detailed categories are extracted from their image samples by EfficientDet. In a short, the experiments involve two groups and seven divisions divided by ImageNet. The detailed information for each corresponding division is shown in Table 2.

There, the label rate demonstrates the difficulty of the corresponding division. This is because with the reduction of supervision in semi-supervised manner, there are higher requirements for the generalization and robustness of the model. The performance of corresponding division will also degrade as the supervised label rate decreases. Therefore, the accuracy in widely different divisions can reflect the robustness of the model. As can be seen from Table 2, the most difficult task is the all division. The label rate of the four subsets increases in turn. Among them, the bird division simulates the division of “All”, which most intuitively illustrates the impact of fine-grained semantics. In contrast, the dog division is completely different from the “All” division where the majority of classes are seen.

The proposed method is evaluated according to Generalized Zero-Shot Learning (GZSL) setting, which is the most challenging evaluation metric for zero-shot learning. In GZSL, all classes, no matter seen or unseen classes, are all considered as the candidate classes in the testing, while the test samples are only from unseen classes. We adopt the same train/test splits and the Top-k Hit Ratio (Hit@k) metric, in accordance with Wang, Ye & Gupta (2018b), Kampffmeyer et al. (2019b) and Wang & Jiang (2021b) for all divisions.

Implementation details

In the model implementations, Resnet-50 (He et al., 2016) is used as feature extractor to extract 2048-dimensions visual feature that has been pre-trained on the ImageNet 2012 dataset. GloVe (Pennington, Socher & Manning, 2014) is used to extract 300-dimensions semantic embedding for initial graph representation on WordNet nodes. EfficientDet (Tan, Pang & Le, 2020) is used to extract 300-dimensions visual features for initial graph representation on fine-grained part nodes. The proposed method is trained for 1000 epochs using ADAM Kingma & Ba (2015) optimizer with learning rate 0.001 and weight decay 0.0005. For each convolutional layer, we employ Dropout operation and leaky ReLUs with a dropout rate of 0.5 and a negative slope of 0.2 respectively. Besides, the model is implemented by PyTorch, training on 4 ×GTX-2080Ti GPUs.

Comparisons with state-of-the-art

In this section, all the comparisons conducted in both General and Detailed groups follow the GZSL setting. For General group, DeViSE (Frome et al., 2013), ConSE (Norouzi et al., 2014), ConSE2 (Wang, Ye & Gupta, 2018b), GCNZ (Wang, Ye & Gupta, 2018b), DGP (Kampffmeyer et al., 2019b) and CL-DKG (Wang & Jiang, 2021b) are selected as counterparts. The results of these counterparts are directly copied from their original publications. For Detailed group, we apply the sources code of each counterpart to conduct the comparison in four sub-categories datasets. Only DGP, GCNZ and SGCN (Kampffmeyer et al., 2019b) are compared since other methods have not provided the source code.

General group comparisons: From Table 3, it can be observed that our method achieves a new state-of-the-art on Top-1 accuracy in all divisions of ImageNet. Especially, our method obtains +2.8%, +0.5% and +0.2% increasing compared with DGP in 2-hops, 3-hops and all divisions relatively. It is a significant progress since Top-1 accuracy in ImageNet zero-shot learning is the most challenging and representative task. The result also demonstrates that fine-grained semantic and visual features guide model generates more discriminative graph representations, which are helpful for recognizing fine-grained unseen classes.

But from Top-2 to Top-20 accuracy evaluations, the latest method CL-DKG surpasses our method. The reason might be the fine-grained features strengthen discrimination ability but weaken, to some extent, the generalization ability of the model. However, it can be seen from Table 3, the results of our method are very close to CL-DKG and DGP that are still comparable with the state-of-the-art.

Detailed group comparisons: The most significant contribution of the proposed method is that fine-grained visual features are shared with semantic features in the same knowledge graph. Here, Detail comparisons are aimed to show the effects of such shared visual features through some representative categories of ImageNet. It can be seen from From Table 4, our method significantly surpasses other methods on Top-1 to Top-5 accuracy evaluations in bird, snake, primate and dog categories. Specially, on Top-1 accuracy evaluations, our method even achieves 2-3 times increasing compared with the state-of-the-art. It demonstrates the shared visual features boost the description ability of the knowledge graph that helps to find more unseen classes. And shared visual features also play a greater role than normal semantic features where the model can double the performance with only a small amount of shared visual features added. On Top-10 and Top-20 accuracy evaluations, DGP and our method have the similar performances.

Ablation study

As defined in Eqs. (1) and (2), the proposed SVKG mainly consists of X_s, X_v and X_aug. Thus, four combinations, “X_s” only, “X_s+ X_v”, “X_s+ X_aug” and “All”, for ablation study are established. Indeed, “X_s” denotes the traditional knowledge graph while “All” represents the proposed knowledge graph SVKG_aug. The experiments are conducted also on the Detailed group (Table 2) to evaluate Top-1 to Top-20 accuracy. The experimental result is presented in Table 5. It is clearly observed that the Top-1 and Top-2 accuracy has significant improvements with the addition of X_s, X_v and X_aug. Specially, in dog category, the Top-1 accuracy has a very steep growth from 6.2% (X_s) to 17.7% (All). The reason is that dog is a well-known category and widely used everywhere. This leads the upstream feature extractor, EfficientDet, to work well and extract more accurate part visual features for SVKG that significantly improves the Top-1 accuracy. A similar phenomenon is also observed in bird and primate categories. The study thus also reveals that the better performance of the upstream feature extractor, the better accuracy the proposed SVKG might achieve. In a short, the study demonstrates the effectiveness of each proposed module in real-world zero-shot learning tasks.

In the cases from Top-5 to Top-20, the best performance is “ X_s+ X_aug”. The influence of graph augmentation X_aug is weakened after two or three hops, and distant nodes are easy to overfit with the presence of part visual nodes. Therefore, in the long-distance prediction, the overall performance of the model decreases.

Discussions

For further discussion, we conducted an unseen class search compared with DGP on Detailed group. Some representative results are presented in Fig. 5. Obviously, our method obtains better accuracy on unseen class searching than DGP. Interestingly, it can be seen that our method predicts normally more specific class labels than DGP does, such as “plover - sea” (ours - DGP), “Australian blacksnake - lyre snake”, “grivet - old word monkey” and “toy spaniel - toy”. This means more fine-grained information is learned in our model that can be used to predict more specific unseen classes.

Meanwhile, t-SNE was used to visualize knowledge graph features for initial graph, GCNZ graph, DGP graph and SVKG graph on dog category. The orange-colored nodes represent seen classes and the blue-colored nodes represent unseen classes. Figure 6A is the initial graph, where the features are represented by word embedding. It can be found that the graph feature distribution is disorderly and the unseen classes are close to each other, which is not conducive for unseen class predicting. In GCNZ (Fig. 6B) and DGP (Fig. 6C), after cross-modal learning (semantic-to-visual modal), the graph features gradually become clear and show a hierarchical structure. Figure 6D shows the feature distribution of our graph SVKG. Intuitively, SVKG is more structured and order than the graph features of GCNZ and DGP. It can be observed that blue-colored nodes distribute among orange-colored nodes reveals the hidden relationships between seen and unseen classes. This is because the proposed visual features X_v lead the graph representations of SVKG close to real-world dog taxonomy. Unseen class nodes thus are distributed to the most related seen class nodes by the meaning of taxonomy. It obviously alleviates coincidence and proximity during unseen class predicting that improves the performance of zero-shot learning.

Moreover, we also compared feature distribution of SVKG with GCNZ graph feature distribution and real image feature distribution, as shown in Fig. 7. It can be observed that the feature distribution of SVKG is significantly closer to the real image feature distribution than GCNZ. This indicates the feature of SVKG is more close to the real visual feature that is easier to be matched with unseen classes. This also explains why the proposed method achieves the best accuracy on Top-1 and Top-2 unseen class predicting.