Keywords

1 Introduction

Among the different image interpretation methods exploiting some kind of knowledge transfer in their design, Zero Shot Classification (ZSC) can be considered as a domain adaptation problem where the new target domain is defined using an intermediate level of representation made of human understandable semantic attributes. The source domain is defined by an annotated image database expected to capture the relation between data and attribute based representation of classes.

Most of the recent approaches addressing ZSC [2–8] rely on the computation of a similarity function in the semantic space. They learn the semantic embedding, either from data or from class description, and compare the embedded data using standard distance.

Recently, [1] proposed to add a metric learning (ML) step to adapt empirically the similarity distance in the embedding space, leading to a multi-objective criterion optimizing both the metric and the embedding. The metric is learned using an empirical optimized criterion on random but equally sampled pairs of similar (positive) and dissimilar (negative) data.

In this paper, following observations from the active learning community (see e.g., [9]), we show that a careful choice of the negative pairs combined with the multi-objective criterion proposed in [1] leads to above state of the art results.

2 Improved ZSL by Efficient Hard Negative Mining

In a recent work, Bucher et al. [1] introduced a metric learning step in their zero-shot classification pipeline. Their model is trained from pairs of data, where positive (resp. negative) pairs are obtained by taking the training images associated with their own provided attribute vector (resp. by randomly assigning attribute vector of another image) and are assigned to the class label ‘1’ (resp. ‘\(-1\)’). The set of positive pairs are denoted \(\mathbf {D_{+}}\) in the following, and \(\mathbf {D_{-}}\) for the negatives. This paper investigates improved ways to select negative pairs.

2.1 Bucher et al. Metric Learning Framework for Zero-Shot Classification

This section summarizes Bucher et al.  [1] paper, to which the reader should refer for more details. Zero-shot classification problem is cast into an optimal framework of the form:

$$\begin{aligned} \mathbf {Y}^* = {\mathop {\text {arg min}}\limits _{\mathbf {Y}\in \mathcal {Y}}}\, S(\mathbf {X},\mathbf {Y}), \end{aligned}$$

where \(\mathbf {X}\) is an image, \(\mathbf {Y}\) a vector of attributes and \(\mathbf {S}\) a parametric similarity measure. A metric matrix denoted as \(\mathbf {W}_A\) transforms the attribute embedding space into a space where the Euclidean distance can be used. The similarity between images and attributes is computed as:

$$\begin{aligned} S(\mathbf {X},\mathbf {Y}) = \left\| (\hat{\mathbf {A}}_X(\mathbf {X}) - \mathbf {Y})^T \mathbf {W}_A \right\| _2 \end{aligned}$$
(1)

where \(\hat{\mathbf {A}}_X\) embeds the \(\mathbf {X}\) modality into the space of \(\mathbf {Y}\) using a linear transformation combined with a ReLU type transfer function:

$$\begin{aligned} \hat{\mathbf {A}}_X(\mathbf {X}) = \max (0,\mathbf {X}^T\mathbf {W}_X + \mathbf {b}_X). \end{aligned}$$
(2)

The role of learning is to estimate jointly the two matrices \(\mathbf {W}_X\) and \(\mathbf {W}_A\). The empirical learning criterion used is the sum of 3 terms:

  1. (i)

    a term for the metric \(\mathbf {W}_A\):

    $$\begin{aligned} l_H(\mathbf {X}_i,\mathbf {Y}_i,Z_i,\tau ) = \max \left( 0,1-Z_i(\tau - S(\mathbf {X}_i,\mathbf {Y}_i)^2)\right) . \end{aligned}$$
    (3)

    where \(Z_i\) states that the two modalities are consistent (\(Z_i=1\)) or not (\(Z_i=-1\)). \(\tau \) is the threshold separating similar and dissimilar examples.

  2. (ii)

    a quadratic loss for the linear attribute prediction \(\mathbf {W}_X\) (only applied to positive pairs):

    $$\begin{aligned} l_A(\mathbf {X}_i,\mathbf {Y}_i,Z_i) = \max (0,Z_i).\left\| \mathbf {Y}_i - \hat{\mathbf {A}}_X(\mathbf {X}_i) \right\| _2^2. \end{aligned}$$
    (4)
  3. (iii)

    a quadratic penalization to prevent overfitting:

    $$\begin{aligned} R(\mathbf {W}_A,\mathbf {W}_X, \mathbf {b}_X) = \left\| \mathbf {W}_X \right\| _F^2 + \left\| \mathbf {b}_X \right\| _2^2 + \left\| \mathbf {W}_A \right\| _F^2 \end{aligned}$$
    (5)

    The overall objective function can then now be written as the sum of the previously defined terms:

    $$\begin{aligned} \begin{aligned} \mathcal {L}(\mathbf {W}_A,\mathbf {W}_X, \mathbf {b}_X, \tau ) = \sum _i l_H(\mathbf {X_i},\mathbf {Y_i},Z_i,\tau ) + \lambda \sum _i l_A(\mathbf {X_i},\mathbf {Y_i},Z_i)\\ + \mu R(\mathbf {W}_A,\mathbf {W}_X, \mathbf {b}_X) \end{aligned} \end{aligned}$$
    (6)

    In this context, ZSC is achieved by finding the most consistent attribute description from a set of exclusive attribute class descriptors \(\{\mathbf {Y}^*_k\}_{k=1}^C\) given the image where k, is the index of a class:

    $$\begin{aligned} k^{*} = {\mathop {\hbox {arg min}}\limits _{k\in \{1\ldots C\}}}\, S(\mathbf {X},\mathbf {Y}^*_k) \end{aligned}$$
    (7)

2.2 Hard Negative Mining

In a metric learning problem, while the set of positive pairs \(\mathbf {D_{+}}\) is fixed and given by the training set with one pair per positive image, the set of negative pairs \(\mathbf {D_{-}}\) can be chosen more freely; indeed, as there are many more ways of being different than being equal, the number of negative and positive pairs may not be identical. Moreover, we will see that increasing the size of \(\mathbf {D_{-}}\) compared to that of \(\mathbf {D_{+}}\) by some factor n leads to better overall results.

In the following, we explore three different strategies to sample the distribution of negative pairs using several learning epochs: we first present a variant of the method of [1] and then describe two iterative greedy schemes.

Random. In [1], negative pairs are obtained by associating a training image with an attribute vector chosen randomly among those of other seen classes, with one negative pair for each positive one. As a variant, we propose to generate randomly n negative pairs (instead of one) for each positive pair, chosen as in [1], i.e., by randomly sampling the set of attribute vectors from the other classes. We include in the objective function a penalization to compensate for the unbalance between positive and negative pairs (see Sect. 2.3).

Uncertainty. This strategy is inspired by hard mining for object detection [9–12] and consists in selecting the most informative negative pairs and iteratively updating the scoring function given by Eq. (1). We denote by \(S_{t}(\mathbf {X},\mathbf {Y})\) this score at time t. During training, each time step t corresponds to a learning epoch. At the first epoch, \(S_{1}(\mathbf {X},\mathbf {Y})\) is learned using the random negative pairs of [1]. At each time t each pair of training image \(\mathbf {X}_{i}\) and candidate annotation \(\mathbf {Y}\) coming from different (but seen) classes is ranked according to the uncertainty score:

$$\begin{aligned} u_t(\mathbf {Y}|{\mathbf {X}_{i}}) = \exp (-(S_{t}(\mathbf {X}_{i},\mathbf {Y})-S_{t}(\mathbf {X}_{i},\mathbf {Y}^*))) \end{aligned}$$
(8)

where \(\mathbf {Y}^*\) is the true vector of attributes of \(\mathbf {X}_{i}\). The vector of attributes which are most similar to the actual one while coming from different classes are the most relevant for improving the model. We define a probability of generating the pair based on this similarity score and sample this distribution.

Uncertainty/Correlation. We propose to improve the previous approach by taking into account the intra-class correlation. The underlying principle governing the selection is that the most correlated vectors of attribute, in a given class, are the most useful ones to consider. The correlation can be measured by:

$$\begin{aligned} q(\mathbf {Y}) = \exp \left( \frac{-1}{|\mathcal {Y}_{k}|}\sum _{\mathbf {Y}'\in \mathcal {Y}_k} \left\| \mathbf {Y}-\mathbf {Y}'\right\| _2 \right) \end{aligned}$$
(9)

where k is the true class index of \(\mathbf {Y}\) and \(\mathcal {Y}_k\) is the set of attribute vector representations.

A trade-off between uncertainty and correlation is obtained globally by using the following scoring function:

$$\begin{aligned} p_t(\mathbf {Y} | \mathbf {X}_{i}) = u_t(\mathbf {Y} | \mathbf {X}_{i}) * q(\mathbf {Y}) \end{aligned}$$
(10)

where each image attribute vector \(\mathbf {Y}\) at epoch t has a score of \(p_t\) to be associated with \(\mathbf {X}_{i}\). The current set of negative pairs \(\mathbf {D}_{-}\) at epoch t is obtained by iteratively increasing the set with new data sampled according to Eq. (10).

2.3 Adaptation of the Objective Function

The original learning criterion (6), such as defined in [1], assumes that negative and positive pairs are evenly distributed. This is not the case in the proposed approach: the criterion must be adapted to compensate for the imbalance between positive and negative pairs, by weighting the positive and negative pairs according to their frequencies:

$$\begin{aligned} \begin{aligned} {\mathcal {L}}(\mathbf {W}_{A},\mathbf {W}_{X}, \mathbf {b}_{X}, \tau ) = \frac{1}{|{\mathbf{D}_{+}}|}\left( \sum _{i\in \mathcal {{\mathbf{D}_{+}}}} l_{H}(\mathbf {X_{i}},\mathbf {Y_{i}},Z_{i},\tau ) + \lambda l_{A}(\mathbf {X_i},\mathbf {Y_{i}},Z_{i})\right) \\ + \frac{1}{|{\mathbf{D}_{-}}|}\left( \sum _{j\in \mathcal {{\mathbf{D}_{-}}}} l_{H}(\mathbf {X_{j}},\mathbf {Y_{j}},Z_{j},\tau )\right) + \mu R(\mathbf {W}_{A},\mathbf {W}_{X}, \mathbf {b}_{X}) \end{aligned} \end{aligned}$$
(11)

This criterion is updated at each new epoch when learning the model.

3 Experiments

Datasets. In this section we evaluate the proposed hard mining strategy on different challenging zero-shot learning tasks, by doing experiments on the 4 following public datasets: aPascal&aYahoo (aP&Y) [13], Animals with Attributes (AwA) [14], CUB-200-2011 (CUB) [15] and SUN attribute (SUN) [16] datasets. They have been designed to evaluate ZSC methods and contain a large number of categories (indoor and outdoor scenes, objects, person, animals, etc.) described using various semantic attributes (shape, material, color, part name etc.). To make comparisons with previous works possible, we used the same training/testing splits as [13] (aP&Y), [14] (AwA), [3] CUB and [17] (SUN).

Image Features. For each dataset, we used the VGG-VeryDeep-19 [18] CNN models, pre-trained on imageNet (without fine tuning) and extract the fully connected layer (e.g., FC7 4096-d) for representing the images.

Hyper-parameters. To estimate the three hyper-parameters (\(\lambda \), the dimensionality of the metric space (m) and \(\mu \)) we apply a grid search validation procedure by randomly keeping 20 % of the training classes. \(\mathbf {W}_A\) and \(\mathbf {W}_X\) are randomly initialized with normal distribution and optimized with stochastic gradient descent.

3.1 Zero-Shot Classification

The experiments follow the standard ZSC protocol: during training, a set of images from known classes is available for learning the model parameters. At test time, images from unseen classes have to be assigned to one of the possible classes. Classes are described by a vector of attributes. Performance is measured by mean accuracy and std over the classes.

Table 1. Zero-shot classification accuracy (mean ± std) on 5 runs. We report results with VGG-verydeep-19 [18] features. \(\text {unc./cor.}= Uncertainty/Correlation\) method. The unc./cor. method can’t be apply to the AwA dataset since all images of the same class have the same attributes, contrarily to the aP&Y, CUB and SUN datasets.
Full size table

Tables 1 and 2 show the performance given by our hard-mining approach, which outperforms previous methods on 3 of the 4 datasets by more than 3 % on average (+9 % on AwA). The smart selection of negative pairs plays a role on the decision boundaries especially where classes have close attribute descriptions. We did not compare our results with [3] or [8] as they use different image features. The 2 alternatives explored in the paper (Uncertainty vs Uncertainty/Correlation) give similar performance, but, as shown in the next section Uncertainty/Correlation is faster.

3.2 Performance as a Function of the Ratio of Positive/Negative Pair

Table 2 give accuracy performances on aP&Y dataset for the three methods in function of the number of negative examples for each positive pair. Bucher et al. [1] configuration corresponds to random method with one negative example per positive one. Our new negative pair selection method have a strong impact on the performance with a noticeable mean improvement of 3 %. Augmenting the ratio of negative pairs over positive ones has a positive influence on the accuracy.

Table 2. Zero-shot classification accuracy (mean ± std) on aP&Y dataset as a function of the ratio of positive/negative pairs.
Full size table
Fig. 1.
figure 1

Evolution of the performance as a function of the number of epochs, on the aP&Y dataset, with a neg/pos ratio of 1 and 100.

Full size image

3.3 Convergence

We also made experiments to evaluate the impact of the hard mining selection on the convergence during training. Figure 1 shows that Uncertainty/Correlation converges around 4 times faster than the Uncertainty and Random methods. This confirms the fact that more informative (negative) pairs are selected with this strategy. The negative/positive ratio has a (small) positive impact on the convergence.

4 Conclusions

This paper extended the original work of Bucher et al. [1] by proposing a novel hard negative mining approach used during training. The proposed selection strategy gives close or above state-of-the-art performance on four standard benchmarks and has a positive impact on convergence.