article

Feature Analysis of Unsupervised Learning for Multi-task Classification Using Convolutional Neural Network

Authors:

Minho LeeAuthors Info & Claims

Neural Processing Letters, Volume 47, Issue 3

Pages 783 - 797

https://doi.org/10.1007/s11063-017-9724-1

Published: 01 June 2018 Publication History

Abstract

This study analyzes the characteristics of unsupervised feature learning using a convolutional neural network (CNN) to investigate its efficiency for multi-task classification and compare it to supervised learning features. We keep the conventional CNN structure and introduce modifications into the convolutional auto-encoder design to accommodate a subsampling layer and make a fair comparison. Moreover, we introduce non-maximum suppression and dropout for a better feature extraction and to impose sparsity constraints. The experimental results indicate the effectiveness of our sparsity constraints. We also analyze the efficiency of unsupervised learning features using the t-SNE and variance ratio. The experimental results show that the feature representation obtained in unsupervised learning is more advantageous for multi-task learning than that obtained in supervised learning.

References

[1]

Bengio Y et al (2009) Learning deep architectures for ai. Foundations and trends$$\textregistered $$®. Mach Learn 2(1):1---127

Digital Library

[2]

Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249---256

[3]

Gülçehre BY (2016) Knowledge matters: importance of prior information for optimization. J Mach Learn Res 17(8):1---32

Digital Library

[4]

Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504---507

[5]

Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Tech. rep., Technical Report 07-49, University of Massachusetts, Amherst

[6]

Kavukcuoglu K, Sermanet P, Boureau YL, Gregor K, Mathieu M, Cun YL (2010) Learning convolutional feature hierarchies for visual recognition. In: Advances in neural information processing systems, pp 1090---1098

Digital Library

[7]

Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Technical report, University of Toronto

[8]

LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278---2324

[9]

LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436---444

[10]

Li S, Liu ZQ, Chan AB (2014) Heterogeneous multi-task learning for human pose estimation with deep convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 482---489

Digital Library

[11]

Liu C, Jang YM, Ozawa S, Lee M (2011) Incremental 2-directional 2-dimensional linear discriminant analysis for multitask pattern recognition. In: The 2011 international joint conference on neural networks (IJCNN). IEEE, pp 2911---2916

[12]

Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models. Proc ICML 30(1)

[13]

Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579---2605

[14]

Makhzani A, Frey BJ (2015) Winner-take-all autoencoders. In: Advances in neural information processing systems, pp 2791---2799

Digital Library

[15]

Masci J, Meier U, Cirean D, Schmidhuber J (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. In: International conference on artificial neural networks. Springer, pp 52---59

Digital Library

[16]

Mika S, Ratsch G, Weston J, Scholkopf B, Mullers KR (1999) Fisher discriminant analysis with kernels. In: Neural networks for signal processing IX, 1999. Proceedings of the 1999 IEEE signal processing society workshop. IEEE, pp 41---48

[17]

Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807---814

Digital Library

[18]

Ng HW, Winkler S (2014) A data-driven approach to cleaning large face datasets. In: 2014 IEEE international conference on image processing (ICIP). IEEE, pp 343---347

[19]

Ozawa S, Roy A, Roussinov D (2009) A multitask learning model for online pattern recognition. IEEE Trans Neural Netw 20(3):430---445

Digital Library

[20]

Paulin M, Douze M, Harchaoui Z, Mairal J, Perronin F, Schmid C (2015) Local convolutional features with unsupervised training for image retrieval. In: Proceedings of the IEEE international conference on computer vision, pp 91---99

Digital Library

[21]

Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint. arXiv:1409.1556v6

[22]

Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929---1958

Digital Library

[23]

Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1701---1708

Digital Library

[24]

Thrun S, Pratt L (2012) Learning to learn. Springer Science & Business Media, Berlin

[25]

Tieleman T (2008) Training restricted boltzmann machines using approximations to the likelihood gradient. In: Proceedings of the 25th international conference on machine learning. ACM, pp 1064---1071

Digital Library

[26]

Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning. ACM, pp 1096---1103

Digital Library

[27]

Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv preprint. arXiv:1411.7923v1

[28]

Zhang C, Zhang Z (2014) Improving multiview face detection with multi-task deep convolutional neural networks. In: 2014 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1036---1041

Cited By

Baek SKim JLee JLee M(2024)Implementation of a Virtual Assistant System Based on Deep Multi-modal Data IntegrationJournal of Signal Processing Systems10.1007/s11265-022-01829-596:3(179-189)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s11265-022-01829-5
Cao ZZhao JSun S(2021)Stick-Breaking Dependent Beta Processes with Variational InferenceNeural Processing Letters10.1007/s11063-020-10392-853:1(339-353)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1007/s11063-020-10392-8
Liu YXu CChen ZChen CZhao HJin X(2020)Deep Dual-Stream Network with Scale Context Selection Attention Module for Semantic SegmentationNeural Processing Letters10.1007/s11063-019-10148-z51:3(2281-2299)Online publication date: 24-Jan-2020
https://dl.acm.org/doi/10.1007/s11063-019-10148-z

Feature Analysis of Unsupervised Learning for Multi-task Classification Using Convolutional Neural Network
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

Deep Adaptive Feature Aggregation in Multi-task Convolutional Neural Networks
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

Convolutional Neural Network (CNN) based multi-task learning methods have been widely used in a variety of applications of computer vision. Towards effective multi-task CNN architectures, recent studies automatically learn the optimal combinations of ...
Deep learning of representations for unsupervised and transfer learning
UTLW'11: Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning workshop - Volume 27

Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features. The objective is ...
Towards dropout training for convolutional neural networks

Recently, dropout has seen increasing use in deep learning. For deep convolutional neural networks, dropout is known to work well in fully-connected layers. However, its effect in convolutional and pooling layers is still not clear. This paper ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Neural Processing Letters

Neural Processing Letters Volume 47, Issue 3

June 2018

525 pages

ISSN:1370-4621

Issue’s Table of Contents

Copyright © Copyright © 2018 Springer Science+Business Media, LLC, part of Springer Nature.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 June 2018

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Baek SKim JLee JLee M(2024)Implementation of a Virtual Assistant System Based on Deep Multi-modal Data IntegrationJournal of Signal Processing Systems10.1007/s11265-022-01829-596:3(179-189)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s11265-022-01829-5
Cao ZZhao JSun S(2021)Stick-Breaking Dependent Beta Processes with Variational InferenceNeural Processing Letters10.1007/s11063-020-10392-853:1(339-353)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1007/s11063-020-10392-8
Liu YXu CChen ZChen CZhao HJin X(2020)Deep Dual-Stream Network with Scale Context Selection Attention Module for Semantic SegmentationNeural Processing Letters10.1007/s11063-019-10148-z51:3(2281-2299)Online publication date: 24-Jan-2020
https://dl.acm.org/doi/10.1007/s11063-019-10148-z

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents