Article

Rectified linear units improve restricted boltzmann machines

Authors:

Geoffrey E. HintonAuthors Info & Claims

ICML'10: Proceedings of the 27th International Conference on International Conference on Machine Learning

Pages 807 - 814

Published: 21 June 2010 Publication History

Abstract

Restricted Boltzmann machines were developed using binary stochastic hidden units. These can be generalized by replacing each binary unit by an infinite number of copies that all have the same weights but have progressively more negative biases. The learning and inference rules for these "Stepped Sigmoid Units" are unchanged. They can be approximated efficiently by noisy, rectified linear units. Compared with binary units, these units learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset. Unlike binary units, rectified linear units preserve information about relative intensities as information travels through multiple layers of feature detectors.

References

[1]

Bengio, Y. and LeCun, Y. Scaling learning algorithms towards AI. 2007.

[2]

Chopra, S., Hadsell, R., and LeCun, Y. Learning a similarity metric discriminatively, with application to face verification. In CVPR, pp. 539-546, Washington, DC, USA, 2005. IEEE Computer Society.

Digital Library

[3]

Freund, Y. and Haussler, D. Unsupervised learning of distributions on binary vectors using two layer networks. Technical report, Santa Cruz, CA, USA, 1994.

Digital Library

[4]

Hahnloser, Richard H. R., Seung, H. Sebastian, and Slotine, Jean-Jacques. Permitted and forbidden sets in symmetric threshold-linear networks. Neural Computation, 15(3):621-638, 2003. ISSN 0899-7667.

Digital Library

[5]

Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8): 1711-1800, 2002.

Digital Library

[6]

Hinton, G. E. and Salakhutdinov, R. Reducing the dimensionality of data with neural networks. Science, 313: 504-507, 2006.

[7]

Hinton, G. E., Sallans, B., and Ghahramani, Z. A hierarchical community of experts. pp. 479-494, 1999.

Digital Library

[8]

Hinton, G. E., Osindero, S., and Teh, Y. A fast learning algorithm for deep belief nets. Neural Computation, 18: 1527-1554, 2006.

Digital Library

[9]

Huang, G. B., Ramesh, M., Berg, T., and Learned-Miller, E. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. Technical Report 07-49, University of Massachusetts, Amherst, 2007.

[10]

Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV'09). IEEE, 2009.

[11]

Kumar, N., Berg, A. C., Belhumeur, P. N., and Nayar, S. K. Attribute and simile classifiers for face verification. In International Conference on Computer Vision, 2009.

[12]

Larochelle, H., Erhan, D., Courville, A., Bergstra, J., and Bengio., Y. An empirical evaluation of deep architectures on problems with many factors of variation. In ICML, pp. 473-480, 2007.

Digital Library

[13]

LeCun, Y., Huang, F. J., and Bottou., L. Learning methods for generic object recognition with invariance to pose and lighting. In CVPR, Washington, D.C., 2004.

Digital Library

[14]

Marks, T. K. and Movellan, J. R. Diffusion networks, products of experts, and factor analysis. Technical Report UCSD MPLab TR 2001.02, 2001.

[15]

Mohamed, A. and Hinton, G. E. Phone recognition using restricted boltzmann machines. In ICASSP, Dallas, TX, USA, 2010.

[16]

Nair, V. and Hinton, G. E. Implicit mixtures of restricted boltzmann machines. In Neural information processing systems, 2008.

Digital Library

[17]

Salakhutdinov, R. and Hinton, G. E. Replicated softmax: an undirected topic model. In Advances in Neural Information Processing Systems 22, 2009.

Digital Library

[18]

Salakhutdinov, R., Mnih, A., and Hinton, G. E. Restricted Boltzmann machines for collaborative filtering. In Proceedings of the International Conference on Machine Learning, volume 24, pp. 791-798, 2007.

Digital Library

[19]

Taylor, G. W., Hinton, G. E., and Roweis, S. Modeling human motion using binary latent variables. In Advances in Neural Information Processing Systems 19, Cambridge, MA, 2006. MIT Press.

Digital Library

[20]

Teh, Y.W. and Hinton, G. E. Rate-coded restricted boltz-mann machines for face recognition. In Advances in Neural Information Processing Systems, volume 13, 2001.

Digital Library

[21]

Wolf, L., Hassner, T., and Taigman, Y. Similarity scores based on background samples. In Asian Conference on Computer Vision, 2009.

Digital Library

Cited By

Qin YPu NWu HSebe N(2025)Margin-aware Noise-robust Contrastive Learning for Partially View-aligned ProblemACM Transactions on Knowledge Discovery from Data10.1145/370764619:1(1-20)Online publication date: 20-Jan-2025
https://dl.acm.org/doi/10.1145/3707646
Thukral MHaresamudram HPlötz T(2025)Cross-Domain HAR: Few-Shot Transfer Learning for Human Activity RecognitionACM Transactions on Intelligent Systems and Technology10.1145/370492116:1(1-35)Online publication date: 20-Jan-2025
https://dl.acm.org/doi/10.1145/3704921
Ma XZhao SYin ZLi W(2025)Clustered Reinforcement LearningFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-3194-119:4Online publication date: 1-Apr-2025
https://dl.acm.org/doi/10.1007/s11704-024-3194-1
Show More Cited By

Rectified linear units improve restricted boltzmann machines
1. Computing methodologies

Recommendations

Symmetric Rectified Linear Units for Fully Connected Deep Models
Knowledge Science, Engineering and Management
Abstract
Rectified Linear Units (ReLU) is one of the key aspects for the success of Deep Learning models. It has been shown that deep networks can be trained efficiently using ReLU without pre-training. In this paper, we compare and analyze various kinds ...
Facial Feature Tracking Under Varying Facial Expressions and Face Poses Based on Restricted Boltzmann Machines
CVPR '13: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition

Facial feature tracking is an active area in computer vision due to its relevance to many applications. It is a nontrivial task, since faces may have varying facial expressions, poses or occlusions. In this paper, we address this problem by proposing a ...
An overview on Restricted Boltzmann Machines

The Restricted Boltzmann Machine (RBM) has aroused wide interest in machine learning fields during the past decade. This review aims to report the recent developments in theoretical research and applications of the RBM. We first give an overview of the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

ICML'10: Proceedings of the 27th International Conference on International Conference on Machine Learning

June 2010

1262 pages

ISBN:9781605589077

Sponsors

NSF: National Science Foundation
Xerox
Microsoft Research: Microsoft Research
Yahoo!
IBM: IBM

Publisher

Omnipress

Madison, WI, United States

Publication History

Published: 21 June 2010

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

934
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Qin YPu NWu HSebe N(2025)Margin-aware Noise-robust Contrastive Learning for Partially View-aligned ProblemACM Transactions on Knowledge Discovery from Data10.1145/370764619:1(1-20)Online publication date: 20-Jan-2025
https://dl.acm.org/doi/10.1145/3707646
Thukral MHaresamudram HPlötz T(2025)Cross-Domain HAR: Few-Shot Transfer Learning for Human Activity RecognitionACM Transactions on Intelligent Systems and Technology10.1145/370492116:1(1-35)Online publication date: 20-Jan-2025
https://dl.acm.org/doi/10.1145/3704921
Ma XZhao SYin ZLi W(2025)Clustered Reinforcement LearningFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-3194-119:4Online publication date: 1-Apr-2025
https://dl.acm.org/doi/10.1007/s11704-024-3194-1
Fu JChen ZZhang HGao YXu HZhang H(2025)FANet: focus-aware lightweight light field salient object detection networkJournal of Real-Time Image Processing10.1007/s11554-024-01581-y22:1Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11554-024-01581-y
Chen SLi YLou YLin K(2025)Aggressive and robust low-level control and trajectory tracking for quadrotors with deep reinforcement learningNeural Computing and Applications10.1007/s00521-024-10675-437:3(1223-1240)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s00521-024-10675-4
Zheng ZYao SWang ZTong XYuan MTang KSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)DPNProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694616(61559-61592)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694616
Zhang YZhang KVan Gool LDanelljan MYu FSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Lightweight image super-resolution via flexible meta pruningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694565(60305-60314)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694565
Yu WLi JZhang SJi XSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Learning scale-aware spatio-temporal implicit representation for event-based motion deblurringProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694443(57527-57543)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694443
Yin YWang YLi PSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)High-dimensional Bayesian optimization via semi-supervised learning with optimized unlabeled data samplingProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694427(57085-57100)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694427
Yang YShi YWang CZhen XShi YXu JSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Reducing fine-tuning memory overhead by approximate and memory-sharing backpropagationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694396(56357-56381)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694396
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten