research-article

Semi-Supervised Low-Rank Semantics Grouping for Zero-Shot Learning

Authors:

Zhengming DingAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 30

Pages 2207 - 2219

https://doi.org/10.1109/TIP.2021.3050677

Published: 01 January 2021 Publication History

Abstract

Zero-shot learning has received great interest in visual recognition community. It aims to classify new unobserved classes based on the model learned from observed classes. Most zero-shot learning methods require pre-provided semantic attributes as the mid-level information to discover the intrinsic relationship between observed and unobserved categories. However, it is impractical to annotate the enriched label information of the observed objects in real-world applications, which would extremely hurt the performance of zero-shot learning with limited labeled seen data. To overcome this obstacle, we develop a Low-rank Semantics Grouping (LSG) method for zero-shot learning in a semi-supervised fashion, which attempts to jointly uncover the intrinsic relationship across visual and semantic information and recover the missing label information from seen classes. Specifically, the visual-semantic encoder is utilized as projection model, low-rank semantic grouping scheme is explored to capture the intrinsic attributes correlations and a Laplacian graph is constructed from the visual features to guide the label propagation from labeled instances to unlabeled ones. Experiments have been conducted on several standard zero-shot learning benchmarks, which demonstrate the efficiency of the proposed method by comparing with state-of-the-art methods. Our model is robust to different levels of missing label settings. Also visualized results prove that the LSG can distinguish the test unseen classes more discriminative.

References

[1]

O. Russakovskyet al., “ImageNet large scale visual recognition challenge,” Int. J. Comput. Vis., vol. 115, no. 3, pp. 211–252, Dec. 2015.

Digital Library

[2]

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014, arXiv:1409.1556. [Online]. Available: http://arxiv.org/abs/1409.1556

[3]

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097–1105.

[4]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770–778.

[5]

C. Szegedyet al., “Going deeper with convolutions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 1–9.

[6]

M. Meng and J. Yu, “Zero-shot learning via robust latent representation and manifold regularization,” IEEE Trans. Image Process., vol. 28, no. 4, pp. 1824–1836, Apr. 2019.

Digital Library

[7]

Y. Guo, G. Ding, J. Han, and Y. Gao, “Zero-shot learning with transferred samples,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3277–3290, Jul. 2017.

Digital Library

[8]

M. Norouziet al., “Zero-shot learning by convex combination of semantic embeddings,” 2013, arXiv:1312.5650. [Online]. Available: http://arxiv.org/abs/1312.5650

[9]

B. Romera-Paredes and P. Torr, “An embarrassingly simple approach to zero-shot learning,” in Proc. Int. Conf. Mach. Learn., 2015, pp. 2152–2161.

[10]

Z. Ding and H. Liu, “Marginalized latent semantic encoder for zero-shot learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 6191–6199.

[11]

Z. Ding, M. Shao, and Y. Fu, “Low-rank embedded ensemble semantic dictionary for zero-shot learning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 2050–2058.

[12]

A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, “Describing objects by their attributes,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 1778–1785.

[13]

M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell, “Zero-shot learning with semantic output codes,” in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 1410–1418.

[14]

C. H. Lampert, H. Nickisch, and S. Harmeling, “Attribute-based classification for zero-shot visual object categorization,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 3, pp. 453–465, Mar. 2014.

Digital Library

[15]

Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Label-embedding for attribute-based classification,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 819–826.

[16]

Z. Li, E. Gavves, T. Mensink, and C. G. Snoek, “Attributes make sense on segmented objects,” in Proc. Eur. Conf. Comput. Vis. Springer, 2014, pp. 350–365.

[17]

D. Jayaraman and K. Grauman, “Zero-shot recognition with unreliable attributes,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 3464–3472.

[18]

Z. Al-Halah, M. Tapaswi, and R. Stiefelhagen, “Recovering the missing link: Predicting class-attribute associations for unsupervised zero-shot learning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 5975–5984.

[19]

Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—A comprehensive evaluation of the good, the bad and the ugly,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 9, pp. 2251–2265, Sep. 2019.

[20]

C. H. Lampert, H. Nickisch, and S. Harmeling, “Learning to detect unseen object classes by between-class attribute transfer,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 951–958.

[21]

H. Larochelle, D. Erhan, and Y. Bengio, “Zero-data learning of new tasks,” in Proc. Assoc. Advancement Artif. Intell., 2008, vol. 1, no. 2, p. 3.

[22]

S. Rahman, S. Khan, and F. Porikli, “A unified approach for conventional zero-shot, generalized zero-shot, and few-shot learning,” IEEE Trans. Image Process., vol. 27, no. 11, pp. 5652–5667, Nov. 2018.

[23]

V. Ferrari and A. Zisserman, “Learning visual attributes,” in Proc. Adv. Neural Inf. Process. Syst., 2008, pp. 433–440.

[24]

T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka, “Metric learning for large scale image classification: Generalizing to new classes at near-zero cost,” in Proc. Eur. Conf. Comput. Vis. Springer, 2012, pp. 488–501.

[25]

M. Rohrbach, S. Ebert, and B. Schiele, “Transfer learning in a transductive setting,” in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 46–54.

[26]

Y. Fu, T. M. Hospedales, T. Xiang, Z. Fu, and S. Gong, “Transductive multi-view embedding for zero-shot recognition and annotation,” in Proc. Eur. Conf. Comput. Vis. Springer, 2014, pp. 584–599.

[27]

E. Kodirov, T. Xiang, and S. Gong, “Semantic autoencoder for zero-shot learning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3174–3183.

[28]

H. Zhang and P. Koniusz, “Zero-shot kernel learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 7670–7679.

[29]

Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, and B. Schiele, “Latent embeddings for zero-shot classification,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 69–77.

[30]

Z. Zhang and V. Saligrama, “Zero-shot learning via joint latent similarity embedding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 6034–6042.

[31]

S. Changpinyo, W.-L. Chao, B. Gong, and F. Sha, “Synthesized classifiers for zero-shot learning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 5327–5336.

[32]

H. Zhang, Y. Long, Y. Guan, and L. Shao, “Triple verification network for generalized zero-shot learning,” IEEE Trans. Image Process., vol. 28, no. 1, pp. 506–517, Jan. 2019.

Digital Library

[33]

Z. Jia, Z. Zhang, L. Wang, C. Shan, and T. Tan, “Deep unbiased embedding transfer for zero-shot learning,” IEEE Trans. Image Process., vol. 29, pp. 1958–1971, 2020.

Digital Library

[34]

R. Gaoet al., “Zero-VAE-GAN: Generating unseen features for generalized and transductive zero-shot learning,” IEEE Trans. Image Process., vol. 29, pp. 3665–3680, 2020.

Digital Library

[35]

E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata, “Generalized zero-and few-shot learning via aligned variational autoencoders,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 8247–8255.

[36]

X. Li and Y. Guo, “Max-margin zero-shot learning for multi-class classification,” in Artificial Intelligence and Statistics. 2015, pp. 626–634.

[37]

X. Li, Y. Guo, and D. Schuurmans, “Semi-supervised zero-shot classification with label representation learning,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 4211–4219.

[38]

M. Xu and Z.-H. Zhou, “Incomplete label distribution learning,” in Proc. Twenty-Sixth Int. Joint Conf. Artif. Intell., Aug. 2017, pp. 3175–3181.

[39]

J. Fürnkranz, E. Hüllermeier, E. Loza Mencía, and K. Brinker, “Multilabel classification via calibrated label ranking,” Mach. Learn., vol. 73, no. 2, pp. 133–153, Nov. 2008.

Digital Library

[40]

G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma, “Robust recovery of subspace structures by low-rank representation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 171–184, Jan. 2013.

Digital Library

[41]

C. Lu, Z. Lin, and S. Yan, “Smoothed low rank and sparse matrix recovery by iteratively reweighted least squares minimization,” IEEE Trans. Image Process., vol. 24, no. 2, pp. 646–654, Feb. 2015.

Digital Library

[42]

C.-P. Wei, C.-F. Chen, and Y.-C.-F. Wang, “Robust face recognition with structurally incoherent low-rank matrix decomposition,” IEEE Trans. Image Process., vol. 23, no. 8, pp. 3294–3307, Aug. 2014.

[43]

G. Liu and S. Yan, “Latent low-rank representation for subspace segmentation and feature extraction,” in Proc. Int. Conf. Comput. Vis., Nov. 2011, pp. 1615–1622.

[44]

Z. Ding, M. Shao, and Y. Fu, “Deep robust encoder through locality preserving low-rank dictionary,” in Proc. Eur. Conf. Comput. Vis. Springer, 2016, pp. 567–582.

[45]

Z. Ding, M. Shao, and Y. Fu, “Missing modality transfer learning via latent low-rank constraint,” IEEE Trans. Image Process., vol. 24, no. 11, pp. 4322–4334, Nov. 2015.

Digital Library

[46]

X. Cai, F. Nie, W. Cai, and H. Huang, “New graph structured sparsity model for multi-label image annotations,” in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 801–808.

[47]

S. S. Bucak, R. Jin, and A. K. Jain, “Multi-label learning with incomplete class assignments,” in Proc. CVPR, Jun. 2011, pp. 2801–2808.

[48]

F. Zhao and Y. Guo, “Semi-supervised multi-label learning with incomplete labels,” in Proc. IJCAI, 2015, pp. 1–7.

[49]

B. Wu, F. Jia, W. Liu, B. Ghanem, and S. Lyu, “Multi-label learning with missing labels using mixed dependency graphs,” Int. J. Comput. Vis., vol. 126, no. 8, pp. 875–896, Aug. 2018.

Digital Library

[50]

J. Li, Y. Liu, R. Yin, and W. Wang, “Multi-class learning using unlabeled samples: Theory and algorithm,” in Proc. 28th Int. Joint Conf. Artif. Intell., Aug. 2019, pp. 2880–2886.

[51]

A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, “Label propagation for deep semi-supervised learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 5070–5079.

[52]

Z. Qi, M. Yang, Z. Zhang, and Z. Zhang, “Mining partially annotated images,” in Proc. 17th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining (KDD), 2011, pp. 1199–1207.

[53]

Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, “Evaluation of output embeddings for fine-grained image classification,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 2927–2936.

[54]

J. Read, B. Pfahringer, G. Holmes, and E. Frank, “Classifier chains for multi-label classification,” Mach. Learn., vol. 85, no. 3, p. 333, Dec. 2011.

Digital Library

[55]

X. J. Zhu, “Semi-supervised learning literature survey,” Dept. Comput. Sci., Univ. Wisconsin-Madison, Madison, WI, USA, Tech. Rep., 2005.

[56]

Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM J. Imag. Sci., vol. 1, no. 3, pp. 248–272, Jan. 2008.

Digital Library

[57]

J.-F. Cai, E. J. Candès, and Z. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM J. Optim., vol. 20, no. 4, pp. 1956–1982, Jan. 2010.

[58]

R. H. Bartels and G. W. Stewart, “Solution of the matrix equation AX + XB = C [F4],” Commun. ACM, vol. 15, no. 9, pp. 820–826, 1972.

Digital Library

[59]

C. H. Q. Ding, T. Li, and M. I. Jordan, “Convex and semi-nonnegative matrix factorizations,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 45–55, Jan. 2010.

Digital Library

[60]

M. Long, J. Wang, G. Ding, D. Shen, and Q. Yang, “Transfer learning with graph co-regularization,” IEEE Trans. Knowl. Data Eng., vol. 26, no. 7, pp. 1805–1818, Jul. 2014.

[61]

S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.

[62]

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The Caltech-UCSD birds-200-2011 dataset,” Tech. Rep., 2011.

[63]

G. Patterson, C. Xu, H. Su, and J. Hays, “The SUN attribute database: Beyond categories for deeper scene understanding,” Int. J. Comput. Vis., vol. 108, nos. 1–2, pp. 59–81, May 2014.

Digital Library

[64]

Z. Zhang and V. Saligrama, “Zero-shot learning via semantic similarity embedding,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 4166–4174.

Cited By

Lazaros KKoumadorakis DVrahatis AKotsiantis S(2024)A comprehensive review on zero-shot-learning techniquesIntelligent Decision Technologies10.3233/IDT-24029718:2(1001-1028)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.3233/IDT-240297
Xing ZPeng JHe XTian M(2024)Semi-supervised sparse subspace clustering with manifold regularizationApplied Intelligence10.1007/s10489-024-05535-654:9-10(6836-6845)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s10489-024-05535-6
Kumar Sinha AMishra DMoorthi S(2024)Towards Adversarial Robustness and Reducing Uncertainty Bias through Expert Regularized Pseudo-Bidirectional Alignment in Transductive Zero Shot LearningPattern Recognition10.1007/978-3-031-78183-4_21(330-345)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/978-3-031-78183-4_21
Show More Cited By

Recommendations

Learning to self-train for semi-supervised few-shot classification
NIPS'19: Proceedings of the 33rd International Conference on Neural Information Processing Systems

Few-shot classification (FSC) is challenging due to the scarcity of labeled training data (e.g. only one labeled data point per class). Meta-learning has shown to achieve promising results by learning to initialize a classification model for FSC. In this ...
Inductive Semi-supervised Multi-Label Learning with Co-Training
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

In multi-label learning, each training example is associated with multiple class labels and the task is to learn a mapping from the feature space to the power set of label space. It is generally demanding and time-consuming to obtain labels for training ...
Multiview Semi-Supervised Learning with Consensus

Obtaining high-quality and up-to-date labeled data can be difficult in many real-world machine learning applications. Semi-supervised learning aims to improve the performance of a classifier trained with limited number of labeled data by utilizing the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 30, Issue

2021

5053 pages

ISSN:1057-7149

Issue’s Table of Contents

1941-0042 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 January 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lazaros KKoumadorakis DVrahatis AKotsiantis S(2024)A comprehensive review on zero-shot-learning techniquesIntelligent Decision Technologies10.3233/IDT-24029718:2(1001-1028)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.3233/IDT-240297
Xing ZPeng JHe XTian M(2024)Semi-supervised sparse subspace clustering with manifold regularizationApplied Intelligence10.1007/s10489-024-05535-654:9-10(6836-6845)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s10489-024-05535-6
Kumar Sinha AMishra DMoorthi S(2024)Towards Adversarial Robustness and Reducing Uncertainty Bias through Expert Regularized Pseudo-Bidirectional Alignment in Transductive Zero Shot LearningPattern Recognition10.1007/978-3-031-78183-4_21(330-345)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/978-3-031-78183-4_21
Li DZeng Z(2023)CRNet: A Fast Continual Learning Framework With Random TheoryIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.326285345:9(10731-10744)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3262853
Pourpanah FAbdar MLuo YZhou XWang RLim CWang XWu Q(2023)A Review of Generalized Zero-Shot Learning MethodsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.319169645:4(4051-4070)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TPAMI.2022.3191696
Ye ZYang GJin XLiu YHuang K(2023)Rebalanced Zero-Shot LearningIEEE Transactions on Image Processing10.1109/TIP.2023.329573832(4185-4198)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3295738
Saini MSusan S(2023)Tackling class imbalance in computer vision: a contemporary reviewArtificial Intelligence Review10.1007/s10462-023-10557-656:Suppl 1(1279-1335)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1007/s10462-023-10557-6

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents