research-article

Research on Point-wise Gated Deep Networks

Authors:

Yu XueAuthors Info & Claims

Applied Soft Computing, Volume 52, Issue C

Pages 1210 - 1221

https://doi.org/10.1016/j.asoc.2016.08.056

Published: 01 March 2017 Publication History

Abstract

Display Omitted We introduce pgRBMs into DBNs and present Point-wise Gated Deep Belief Networks.Similar to pgDBNs, Point-wise Gated Deep Boltzmann Machines are presented.We introduce dropout and weight uncertainty methods into pgRBMs.We discuss the feasibility of dropout and weight uncertainty in deep networks. Stacking Restricted Boltzmann Machines (RBM) to create deep networks, such as Deep Belief Networks (DBN) and Deep Boltzmann Machines (DBM), has become one of the most important research fields in deep learning. DBM and DBN provide state-of-the-art results in many fields such as image recognition, but they don't show better learning abilities than RBM when dealing with data containing irrelevant patterns. Point-wise Gated Restricted Boltzmann Machines (pgRBM) can effectively find the task-relevant patterns from data containing irrelevant patterns and thus achieve satisfied classification results. For the limitations of the DBN and the DBM in the processing of data containing irrelevant patterns, we introduce the pgRBM into the DBN and the DBM and present Point-wise Gated Deep Belief Networks (pgDBN) and Point-wise Gated Deep Boltzmann Machines (pgDBM). The pgDBN and the pgDBM both utilize the pgRBM instead of the RBM to pre-train the weights connecting the networks' the visible layer and the hidden layer, and apply the pgRBM learning task-relevant data subset for traditional networks. Then, this paper discusses the validity that dropout and weight uncertainty methods are developed to prevent overfitting in pgRBMs, pgDBNs, and pgDBMs networks. Experimental results on MNIST variation datasets show that the pgDBN and the pgDBM are effective deep neural networks learning

References

[1]

G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science, 313 (2006) 504-507.

[2]

I. Guyon, A. Elisseeff, An introduction to variable and feature selection, J. Mach. Learn. Res., 3 (2003) 1157-1182.

Digital Library

[3]

B.K. Sohn, G. Zhou, C. Lee, H. Lee, Learning and Selecting Features Jointly with Point-wise Gated Boltzmann Machines, International Conference on Machine Learning (2013) 217-225.

Digital Library

[4]

R.R. Salakhutdinov, G.E. Hinton, An efficient learning procedure for deep Boltzmann machines, Neural Comput., 24 (2010) 1967-2006.

Digital Library

[5]

N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, R.R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15 (2014) 1929-1958.

Digital Library

[6]

C. Blundell, J. Cornebise, K. Kavukcuoglu, Weight uncertainty in neural networks, in: The 32nd International Conference on Machine Learning, 2015.

Digital Library

[7]

N. Zhang, S.F. Ding, J. Zhang, Multi layer ELM-RBF for multi-label learning, Appl. Soft Comput., 43 (2016) 535-545.

Digital Library

[8]

B. Gu, X. Sun, V.S. Sheng, Structural Minimax Probability Machine, IEEE Trans. Neural Netw. Learn Syst. (2016).

[9]

J.W. Liu, Y. Liu, X.L. Luo, Research and development on Boltzmann machine, J. Comput. Res. Dev., 51 (2014) 1-16.

[10]

P. Smolensky, Information processing in dynamical systems: foundations of harmony theory, MIT Press, Cambridge, 1986.

[11]

G.E. Hinton, Training products of experts by minimizing contrastive divergence, Neural Comput., 14 (2002) 1711-1800.

Digital Library

[12]

B. Chen, H. Shu, G. Coatrieux, G. Chen, X. Sun, J. Coatrieux, Color image analysis by quaternion-type moments, J. Math. Imaging Vis., 51 (2015) 124-144.

Digital Library

[13]

M. Ranzato, G.E. Hinton, Modeling pixel means and covariances using factorized third-order boltzmann machines, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2010) 2551-2558.

[14]

A.C. Courville, J. Bergstra, Y. Bengio, Unsupervised models of images by spike and-Slab RBMs, in: the 28th International Conference on Machine Learning, 2011, pp. 1145-1152.

Digital Library

[15]

H. Lee, R. Grosse, R. Ranganath, A.Y. Ng, Unsupervised learning of hierarchical representations with convolutional deep belief networks, Commun. Acm, 54 (2011) 95-103.

Digital Library

[16]

Y. Tang, R.R. Salakhutdinov, G.E. Hinton, Robust Boltzmann Machines for recognition and denoising, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2012) 2264-2271.

Digital Library

[17]

G. Montfar, J. Morton, Discrete restricted boltzmann machines, J. Mach. Learn. Res., 16 (2015) 653-672.

Digital Library

[18]

G. Montfar, N. Ay, K. Ghazi-Zahedi, Geometry and expressive power of conditional restricted Boltzmann machines, J. Mach. Learn. Res., 16 (2015) 2405-2436.

Digital Library

[19]

T. Tieleman, Training restricted Boltzmann machines using approximations to the likelihood gradient, in: the 25th International Conference on Machine Learning, 2010, pp. 1064-1071.

Digital Library

[20]

T. Tieleman, G.E. Hinton, Using fast weights to improve persistent contrastive divergence, in: the 26th Annual International Conference on Machine Learning, 2009, pp. 1033-1040.

Digital Library

[21]

D. Carlson, Y.P. Hsieh, E. Collins, L. Carin, V. Cevher, Stochastic spectral descent for discrete graphical models, IEEE J. Sel. Top. Signal Process., 10 (2016) 296-311.

[22]

G.E. Hinton, R.R. Salakhutdinov, A better way to pretrain deep Boltzmann machines, Adv. Neural Inf. Process.Syst., 3 (2012) 2447-2455.

Digital Library

[23]

P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.A. Manzagol, Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion, J. Mach. Learn. Res., 11 (2010) 3371-3408.

Digital Library

[24]

N. Zhang, S.F. Ding, Z.Z. Shi, Denoising Laplacian multi-layer extreme learning machine, Neurocomputing, 171 (2016) 1066-1074.

Digital Library

[25]

J. Zhang, S.F. Ding, Y. Xue, Weight uncertainty in Boltzmann Machine, Cogn. Comput. (2016).

[26]

Z.Q. Pan, Y. Zhang, S. Kwong, Efficient motion and disparity estimation optimization for low complexity multiview video coding, IEEE T. Broadcast., 61 (2015) 166-176.

[27]

X. Wen, L. Shao, Y. Xue, W. Fang, A rapid learning algorithm for vehicle classification, Inform. Sci., 295 (2015) 395-406.

Digital Library

[28]

H. Larochelle, D. Erhan, A. Courville, J. Bergstra, Y. Bengio, An empirical evaluation of deep architectures on problems with many factors of variation, in: the 24th international conference on Machine learning, 2007, pp. 473-480.

Digital Library

Cited By

Ding SSun YAn YJia W(2020)Multiple birth support vector machine based on recurrent neural networksApplied Intelligence10.1007/s10489-020-01655-x50:7(2280-2292)Online publication date: 3-Mar-2020
https://dl.acm.org/doi/10.1007/s10489-020-01655-x
Zhang NDing SLiao HJia W(2019)Multimodal correlation deep belief networks for multi-view classificationApplied Intelligence10.1007/s10489-018-1379-849:5(1925-1936)Online publication date: 1-May-2019
https://dl.acm.org/doi/10.1007/s10489-018-1379-8
Yilmaz SSen S(2019)Electric fish optimization: a new heuristic algorithm inspired by electrolocationNeural Computing and Applications10.1007/s00521-019-04641-832:15(11543-11578)Online publication date: 5-Dec-2019
https://dl.acm.org/doi/10.1007/s00521-019-04641-8
Show More Cited By

Research on Point-wise Gated Deep Networks
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

Deep learning the ising model near criticality

It is well established that neural networks with deep architectures perform better than shallow networks for many tasks in machine learning. In statistical physics, while there has been recent interest in representing physical data with generative ...
Boltzmann Machines for Image Denoising
Proceedings of the 23rd International Conference on Artificial Neural Networks and Machine Learning ICANN 2013 - Volume 8131

Image denoising based on a probabilistic model of local image patches has been employed by various researchers, and recently a deep denoising autoencoder has been proposed in [2] and [17] as a good model for this. In this paper, we propose that another ...
Adaptive structure learning method of deep belief network using neuron generation–annihilation and layer generation
Abstract
Recently, deep learning is receiving renewed attention in the field of artificial intelligence. Deep belief network (DBN) has a deep network architecture that can represent multiple features of input patterns hierarchically, using pre-trained ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Applied Soft Computing

Applied Soft Computing Volume 52, Issue C

March 2017

1266 pages

ISSN:1568-4946

Issue’s Table of Contents

Copyright © Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 March 2017

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ding SSun YAn YJia W(2020)Multiple birth support vector machine based on recurrent neural networksApplied Intelligence10.1007/s10489-020-01655-x50:7(2280-2292)Online publication date: 3-Mar-2020
https://dl.acm.org/doi/10.1007/s10489-020-01655-x
Zhang NDing SLiao HJia W(2019)Multimodal correlation deep belief networks for multi-view classificationApplied Intelligence10.1007/s10489-018-1379-849:5(1925-1936)Online publication date: 1-May-2019
https://dl.acm.org/doi/10.1007/s10489-018-1379-8
Yilmaz SSen S(2019)Electric fish optimization: a new heuristic algorithm inspired by electrolocationNeural Computing and Applications10.1007/s00521-019-04641-832:15(11543-11578)Online publication date: 5-Dec-2019
https://dl.acm.org/doi/10.1007/s00521-019-04641-8
Laimeche LMeraoumia ABendjenna H(2019)Enhancing LSB embedding schemes using chaotic maps systemsNeural Computing and Applications10.1007/s00521-019-04523-z32:21(16605-16623)Online publication date: 19-Oct-2019
https://dl.acm.org/doi/10.1007/s00521-019-04523-z
Gupta NJalal A(2019)Integration of textual cues for fine-grained image captioning using deep CNN and LSTMNeural Computing and Applications10.1007/s00521-019-04515-z32:24(17899-17908)Online publication date: 19-Oct-2019
https://dl.acm.org/doi/10.1007/s00521-019-04515-z
Zhang NDing SZhang JXue Y(2018)An overview on Restricted Boltzmann MachinesNeurocomputing10.1016/j.neucom.2017.09.065275:C(1186-1199)Online publication date: 31-Jan-2018
https://dl.acm.org/doi/10.1016/j.neucom.2017.09.065
Li DLi YLi JFu Y(2018)Gesture Recognition Based on BP Neural Network Improved by Chaotic Genetic AlgorithmInternational Journal of Automation and Computing10.1007/s11633-017-1107-615:3(267-276)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s11633-017-1107-6

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents