Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning

Object recognition in the real-world requires handling long-tailed or even open-ended data. An ideal visual system needs to recognize the populated head visual concepts reliably and meanwhile efficiently learn about emerging new tail categories with a few training instances. Class-balanced many-shot learning and few-shot learning tackle one side of this problem, by either learning strong classifiers for head or learning to learn few-shot classifiers for the tail. In this paper, we investigate the problem of generalized few-shot learning (GFSL)—a model during the deployment is required to learn about tail categories with few shots and simultaneously classify the head classes. We propose the ClAssifier SynThesis LEarning (Castle), a learning framework that learns how to synthesize calibrated few-shot classifiers in addition to the multi-class classifiers of head classes with a shared neural dictionary, shedding light upon the inductive GFSL. Furthermore, we propose an adaptive version of Castle  (a Castle) that adapts the head classifiers conditioned on the incoming tail training examples, yielding a framework that allows effective backward knowledge transfer. As a consequence, a Castle can handle GFSL with classes from heterogeneous domains effectively. Castle and a Castle demonstrate superior performances than existing GFSL algorithms and strong baselines on MiniImageNet as well as TieredImageNet datasets. More interestingly, they outperform previous state-of-the-art methods when evaluated with standard few-shot learning criteria.

1. We use the super-script \({{\mathcal {S}}}\) and \({{\mathcal {U}}}\) to denote a set or an instance sampled from \({{\mathcal {S}}}\) and \({{\mathcal {U}}}\), respectively.

2. \(|\mathcal {S}|\) and \(|\mathcal {U}|\) denote the total number of classes from the seen and unseen class sets, respectively.

3. More choices of Eq. 6 are investigated in “Appendix 9”.

4. Our implementation will be publicly available on https://github.com/Sha-Lab/aCASTLE.

Thanks to Fei Sha for valuable discussions. This research (61773198, 61751306, 61632004, 62006112), NSFC-NRF Joint Research Project under Grant 61861146001, NSF Awards IIS-1513966/ 1632803/1833137, CCF-1139148, DARPA Award#: FA8750-18-2-0117, DARPA-D3M - Award UCB-00009528, Google Research Awards, gifts from Facebook and Netflix, and ARO# W911NF-12-1-0241 and W911NF-15-1-0484.

1. Han-Jia Ye and Hexiang Hu have contributed equally to this study.

Authors and Affiliations

1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

   Han-Jia Ye & De-Chuan Zhan

2. University of Southern California, Los Angeles, USA

   Hexiang Hu

Corresponding author

Correspondence to Han-Jia Ye.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Implementation Details

1.1 A.1 Pre-training Strategy

In particular, on MiniImageNet, we add a linear layer on the backbone output and optimize a 64-way classification problem on the meta-training set with the cross-entropy loss function. Stochastic gradient descent with initial learning rate 0.1 and momentum 0.9 is used to complete such optimization. The 16 classes in MiniImageNet for model selection also assist the choice of the pre-trained model. After each epoch, we use the current embedding and measures the nearest neighbor based few-shot classification performance on the sampled few-shot tasks from these 16 classes. The most suitable embedding function is recorded. After that, such a learned backbone is used to initialize the embedding part \(\phi \) of the whole model. The same strategy is also applied to the meta-training set of the TieredImageNet, Heterogeneous, and Office-Home datasets, where a 351-way, 100-way, and 25-way classifiers are pre-trained.

1.2 A.2 Feature Network Specification

We follow Qiao et al. (2018); Rusu et al. (2019) when investigating the multi-domain GFSL, where images are resized to \(84\times 84\times 3\). In concrete words, three residual blocks are used after an initial convolutional layer (with stride 1 and padding 1) over the image, which have channels 160/320/640, stride 2, and padding 2. After a global average pooling layer, it leads to a 640 dimensional embedding. While for the benchmark experiments on MiniImageNet and TieredImageNet, we follow Lee et al. (2019) to set the architecture of ResNet, which contains 12 layers and uses the DropBlock (Ghiasi et al. 2018) to prevent over-fitting.

We use the pre-trained backbone to initialize the embedding part \(\phi \) of a model for Castle/a Castle and our re-implemented comparison methods such as MC+kNN, ProtoNet+ProtoNet, MC+ProtoNet, L2ML (Wang et al. 2017), and DFSL (Gidaris and Komodakis 2018). When there exists a backbone initialization, we set the initial learning rate as 1e-4 and optimize the model with Momentum SGD. The learning rate will be halved after optimizing 2,000 mini-batches. During meta-learning, all methods are optimized over 5-way few-shot tasks, where the number of shots in a task is consistent with the inference (meta-test) stage. For example, if the goal is a 1-shot 5-way model, we sample 1-shot 5-way \({{\mathcal {D}}}^{{\mathcal {S}}}_\mathbf {train}\) during meta-training, together with 15 instances per class in \({{\mathcal {D}}}^{{\mathcal {S}}}_\mathbf {test}\).

For Castle/a Castle, we take advantage of the multi-classifier training technique to improve learning efficiency. We randomly sample a 24-way task from \({{\mathcal {S}}}\) in each mini-batch, and re-sample 64 5-way tasks from it. It is notable that all instances in the 24-way task are encoded by the ResNet backbone with the same parameters in advance. Therefore, by embedding the synthesized 5-way few-shot classifiers into the global many-shot classifier, it results in 64 different configurations of the generalized few-shot classifiers. To evaluate the classifier, we randomly sample instances with batch size 128 from \({{\mathcal {S}}}\) and compute the GFSL objective in Eq. 5.

1.3 A.3 Baselines for GFSL Benchmarks

Here we describe some baseline approaches compared in the GFSL benchmarks in detail.

(1) Multiclass Classifier (MC) + k NN A \(|{{\mathcal {S}}}|\)-way classifier is trained on the seen classes in a supervised learning manner as standard many-shot classification (He et al. 2016). During the inference, test examples of \({{\mathcal {S}}}\) categories are evaluated based on the \(|{{\mathcal {S}}}|\)-way classifiers and \(|{{\mathcal {U}}}|\) categories are evaluated using the support embeddings from \({{\mathcal {D}}}_{\mathbf {train}}^{\;{{\mathcal {U}}}}\) with a nearest neighbor classifier. To evaluate the generalized few-shot classification task, we take the union of multi-class classifiers’ confidence and nearest neighbor confidence [the normalized negative distance values as in Snell et al. (2017)] as joint classification scores on \({{\mathcal {S}}} \cup {{\mathcal {U}}}\).

(2) ProtoNet + ProtoNet We train a few-shot classifier (initialized by the MC classifier’s feature mapping) using the Prototypical Network (Snell et al. 2017) (a.k.a. ProtoNet), pretending they were few-shot. When evaluated on the seen categories, we randomly sample 100 training instances per category to compute the class prototypes. The class prototypes of unseen classes are computed based on the sampled few-shot training set. During the inference of generalized few-shot learning, the confidence of a test instances is jointly determined by its (negative) distance to both seen and unseen class prototypes.

(3) MC + ProtoNet We combine the learning objective of the previous two baselines ((1) and (2)) to jointly learn the MC classifier and feature embedding. Since there are two objectives for many-shot (cross-entropy loss on all seen classes) and few-shot (ProtoNet meta-learning objective) classification respectively, it trades off between many-shot and few-shot learning. Therefore, this learned model can be used as multi-class linear classifiers on the head categories, and used as ProtoNet on the tail categories. During the inference, the model predicts instances from seen class \({{\mathcal {S}}}\) with the MC classifier, while takes advantage of the few-shot prototypes to discern unseen class instances. To evaluate the generalized few-shot classification task, we take the union of multi-class classifiers’ confidence and ProtoNet confidence as joint classification scores on \({{\mathcal {S}}} \cup {{\mathcal {U}}}\).

(4) L2ML Wang et al. (2017) propose learning to model the “tail” (L2ML) by connecting a few-shot classifier with the corresponding many-shot classifier. The method is designed to learn classifier dynamics from data-poor “tail” classes to the data-rich head classes. Since L2ML is originally designed to learn with both seen and unseen classes in a transductive manner. In our experiment, we adaptive it to our setting. Therefore, we learn a classifier mapping based on the sampled few-shot tasks from seen class set \({{\mathcal {S}}}\), which transforms a few-shot classifier in unseen class set \({{\mathcal {U}}}\) inductively. Following Wang et al. (2017), we first train a many-shot classifier W upon the ResNet backbone on the seen class set \({{\mathcal {S}}}\). We use the same residual architecture as in Wang et al. (2017) to implement the classifier mapping f, which transforms a few-shot classifier to a many-shot classifier. During the meta-learning stage, a \({{\mathcal {S}}}\)-way few-shot task is sampled in each mini-batch, which produces a \({{\mathcal {S}}}\)-way linear few-shot classifier \({\hat{W}}\) based on the fixed pre-trained embedding. The objective of L2ML not only regresses the mapped few-shot classifier \(f({\hat{W}})\) close to the many-shot one W measured by square loss, but also minimizes the classification loss of \(f({\hat{W}})\) over a randomly sampled instances from \({{\mathcal {S}}}\). Therefore, L2ML uses a pre-trained multi-class classifier W for those head categories and used the predicted few-shot classifiers with f for the tail categories.

Appendix B: More Analysis on GFSL Benchmarks

In this appendix, we do analyses to show the influence of training a GFSL model by reusing the many-shot classifier and study different implementation choices in the proposed methods. We mainly investigate and provide the results over Castle on MiniImageNet. We observe the results on a Castle and other datasets reveal similar trends.

1.1 B.1 Reusing the Many-Shot Classifier Facilitates the Calibration for GFSL

We compare the strategy to train Castle from scratch and fine-tune based on the many-shot classifier. We show both the results of 1-Shot 5-Way few-shot classification performance and GFSL performance with 5 unseen tasks for Castle when trained from random or with provided initialization. From the results in Table 7, we find training from scratch gets only a bit lower few-shot classification results with the fine-tune strategy, but much lower GFSL harmonic mean accuracy. Therefore, reusing the parameters in the many-shot classifier benefits the predictions on seen and unseen classes of a GFSL model. Therefore, we use the pre-trained embedding to initialize the backbone.

1.2 B.2 Comparison with One-Phase Incremental Learning Methods

The inductive generalized few-shot learning is also related to the one-phase incremental learning (Li and Hoiem 2018; Liu et al. 2020), where a model is required to adapt itself to the open set environment. In other words, after training over the closed set categories, a classifier should be updated based on the data with novel distributions or categories accordingly. One important thread of incremental learning methods relies on the experience replay, where a set of the closed set instances is preserved and the classifier for all classes is optimized based on the saved and novel few-shot data. In our inductive GFSL, the Castle variants do not save seen class instances and rely on the neural dictionary to adapt the classifier for a joint classification. Thus, Castle variants have lower computational (time) costs during the inference stage.

Towards comprehensive comparisons, we also investigate two popular incremental learning methods, i.e., LwF (Li and Hoiem 2018) and iCARL (Li and Hoiem 2018). We randomly save 5 images per seen class for both methods. By combining the stored images and the newly given unseen class images together, the model will be updated based on a cross-entropy loss and a distillation loss (Hinton et al. 2015). We tune the balance weight between the classification and distillation loss, the initial learning rate for fine-tuning, and the optimization steps for both methods over the validation set. The harmonic mean accuracy in various evaluation scenarios over 10,000 tasks are listed in Table 8.

In our empirical evaluations, we find that incremental learning methods can get better results than our baselines since it fine-tunes the model with the distillation loss. However, their results are not stable since there are many hyper-parameters. Compared with these approaches, our Castle variants still keep their superiority over all criteria.

1.3 B.3 Light-Weight Adaptation on Castle Variants

As shown in the previous subsection, directly fine-tuning the whole model is prone to over-fit even with another distillation loss. Inspired by Sun et al. (2019) and Li et al. (2019), we consider a light-weight fine-tune step based on the synthesized classifier by Castle variants. In detail, we reformulate the model \(\mathbf {W}^\top \phi (\mathbf{x})\) as \(\mathbf {W}^\top ((\mathbf{1} + \mathrm{scale})\cdot \phi (\mathbf{x}) + \mathrm{bias})\), where \(\mathbf {W}\) is the classifier output by the neural dictionary, the \(\mathrm{scale}\in {\mathbb {R}}^d\) and \(\mathrm{bias}\in {\mathbb {R}}^d\) are additional learnable vectors, and \(\mathbf{1}\) is a size d vector with all values equal 1.

Given a few-shot task with unseen class instances, the model will be updated in the following ways. 5 images per seen class are randomly selected, after freezing the backbone \(\phi \), the classifier \(\mathbf {W}\), the scale, and the bias are optimized based on a cross-entropy loss over both stored seen and unseen classes images. We tune the initial learning rate and the optimization steps over the validation set.

The results of such model adaptation strategies are listed in Table 9. With further model adaptation, both Castle and a Castle could be improved.

1.4 B.4 Effects on the Neural Dictionary Size \(|{{\mathcal {B}}}|\)

We show the effects of the dictionary size (as the ratio of seen class size 64) for the standard few-shot learning (measured by mean accuracy when there are 5 unseen classes) in Fig. 11. We observe that the neural dictionary with a ratio of 2 or 3 works best amongst all other dictionary sizes. Therefore, we fix the dictionary size as 128 across all experiments. Note that when \(|{{\mathcal {B}}}|=0\), our method degenerates to case optimizing the unified objective in Eq. 5 without using the neural dictionary (the Castle\(^-\) model in Sect. 5).

The 1-shot 5-way accuracy on unseen of MiniImageNet with different size of dictionaries

Full size image

The 64-way multi-class accuracy on seen of MiniImageNet with 1-shot trained model

Full size image

1.5 B.5 How Well is Synthesized Classifiers Comparing with Multi-class Classifiers?

To assess the quality of synthesized classifier, we made a comparison against ProtoNet and also the Multi-class Classifier on the head seen concepts. To do so, we sample few-shot training instances on each seen category to synthesize classifiers (or compute class prototypes for ProtoNet), and then use the synthesized classifiers/class prototypes solely to evaluate multi-class accuracy. The results are shown in Fig. 12. We observe that the learned synthesized classifier outperforms over ProtoNet. Also, the model trained with unified learning objective improves over the vanilla synthesized classifiers. Note that there is still a gap left against multi-class classifiers trained on the entire dataset. It suggests that the classifier synthesis we learned is effective against using sole instance embeddings.

1.6 B.6 Different Choices of the Classifier Synthesis

As in Eq. 6, when there is more than one instance per class in a few-shot task (i.e., \(K > 1\)), Castle compute the averaged embeddings first, and then use the prototype of each class as the input of the neural dictionary to synthesize their corresponding classifiers. Here we explore another choice to deal with multiple instances in each class. We synthesize classifiers based on each instance first, and then average the corresponding synthesized classifiers for each class. This option equals an ensemble strategy to average the prediction results of each instance’s synthesized classifier. We denote the pre-average strategy (the one used in Castle) as “Pre-AVG”, and the post-average strategy as “Post-AVG”. The 5-Shot 5-way classification results on MiniImageNet for these two strategies are shown in Table 10. From the results, “Post-AVG” does not improve the FSL and GFSL performance obviously. Since averaging the synthesized classifiers in a hindsight way costs more memory during meta-training, we choose the “Pre-AVG” option to synthesize classifiers when there are more than 1 shot in each class. In our experiments, the same conclusion also applies to a Castle.

1.7 B.7 How is Multiple Classifiers Learning’s Impact over the Training?

Both Castle and a Castle adopt a multi-classifier training strategy (as described in Sect. 3), i.e., considering multiple GFSL tasks with different combinations of classifiers in a single mini-batch. In Table 11, we show the influence of the multi-classifier training method based on their GFSL performance (harmonic mean). It shows that with a large number of classifiers during the training, the performance of Castle asymptotically converges to its upper-bound. We find a Castle shares a similar trend.

1.8 B.8 The Gap to the Performance “Upper Bound” (UB)

We focus on the (generalized) few-shot learning scenario where there are only budgeted examples in the unseen class tasks. To show the potential improvement space in such tasks, we also investigate a kind of upper bound model where all the available images are used to build the unseen class classifier during the inference stage.

We implement the upper bound model based on the ProtoNet, and the results are in Table 12. Specifically, in the FSL classification scenario, all the unseen class images except those preserved for evaluation are used to build more precise prototypes, and the mean accuracy over 10,000 tasks are recorded; in the GFSL classification scenario, the many-shot unseen class images are utilized as well, and the calibrated harmonic mean is used as the performance measure.

Since the upper bound takes advantage of all the available training images for the few-shot categories, it performs better than the few-shot Castle and a Castle in all the scenarios. The gap between the few-shot learning methods and the upper bound becomes larger when more unseen classes (ways) are involved.

Reprints and permissions