Units and Layers' Effects on Deep Boltzman Machines
Authors: 
Luhong Diao, Yang Liu, Dong Nan, Yong Qiao, Juan PengAuthors Info & Claims
Article No.: 81, Pages 1 - 5
Abstract
This paper analyzes the units' and layers' effects on deep Boltzman machines. It divides the DBM into two parts and reveals how the two parts affect the DBM's approximation capability. It indicates that the representation power of deep Boltzman machine is not always improved with more units and layers. When a deep Boltzman machine is best already, more units and layers will nearly always lead to worse performance.
References
[1]
Y Bengio, O Delalleau (2011). On the expressive power of deep architectures. Proc of the 14th International Conference on Discovery Science. Berlin: Springer-Verlag, 18--36.
[2]
H Ackley, G E Hinton and T J Sejnowski (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9(1), 147--169.
[3]
P Smolensky (1986). Information processing in dynamical systems: Foun-dations of harmony theory. In Rumelhart, D. E. and McClelland, J. L.,editors, Parallel Distributed Processing: Volume 1: Foundations, pp. 194--281. MIT Press, Cambridge.
[4]
R Salakhutdinov and G E Hinton (2009). Deep Boltzmann machines. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 5, 448--455.
[5]
G E Hinton, S Osindero and Y W Teh. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527--1554.
[6]
T Brosch, R Tam (2015). Efficient Training of Convolutional Deep Belief Networks in the Frequency Domain for Application to High-Resolution 2D and 3D Images. Neural Computation, 27(1), 211--227.
[7]
Na Lu, Tengfei Li, Xiaodong Ren, Hongyu Miao (2017). A Deep Learning Scheme for Motor Imagery Classification based on Restricted Boltzmann Machines. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(6), 566--576.
[8]
Nitish Srivastava, Ruslan Salakhutdinov (2014). Multimodal learning with deep Boltzmann machines. The Journal of Machine Learning Research, 15(1), 2949--2980.
[9]
L Polania, K Barner (2017). Exploiting Restricted Boltzmann Machines and Deep Belief Networks in Compressed Sensing. IEEE Transactions on Signal Processing. 65(17), 4538--4550.
[10]
S Bu, Z Liu, J Han, J Wu, R Ji (2014). Learning High-Level Feature by Deep Belief Networks for 3-D Model Retrieval and Recognition [J]. IEEE Transactions on Multimedia. 16(8), 2154--2167.
[11]
Lingjun Zhao, Huakun Huang, Shuxue Ding and Xiang Li (2018). An Accurate and Efficient Device-Free Localization Approach Based on Gaussian Bernoulli Restricted Boltzmann Machine. 10.1109/SMC.2018.00399.
[12]
Heena A. Chopade, Meera Narvekar (2017). Hybrid auto text summarization using deep neural network and fuzzy logic system. 2017 International Conference on Inventive Computing and Informatics, 52--56, Coimbatore, India.
[13]
T A Liu (2010). Novel text classification approach based on deep belief network[C] Proceedings of Neural Information Processing Theory and Algorithms International Conference, Sydney, Australia, 314--321.
[14]
I Sutskever, G E Hinton (2008). Deep Narrow Sigmoid Belief Networks Are Universal Approximators. Neural Computation. 20(11), 2629--2636.
[15]
NL Roux, Y Bengio (2008). Representational Power of Restricted Boltzmann Machines and Deep Belief Networks. Neural Computation, 20(6), 1631--1649.
[16]
G Montufar (2014). Deep Narrow Boltzmann Machines are Universal Approximators. Proc of International Conference on Learning Representation.
[17]
C K Chui, H N Mhaskar (2016). Deep Nets for Local Manifold Learning, arXiv preprint, arXiv:1607.07110.
[18]
U Shaham, A Cloninger, R R Coifman (2018). Provable Approximation Properties for Deep Neural Networks. Applied and Computational Harmonic Analysis, 44(3), 537--557.
[19]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov (2014). Dropout: A Simple Way to Prevent Neural Networks from overfitting. Journal of Machine Learning Research, 15(1), 1929--1958.
[20]
J Zhang, S Ding, N Zhang. Y Xue (2016). Cogn Comput, 8(6), 1064--1073.
[21]
S F Ding, J Zhang and Z Z Shi (2018). Algorithms of Boltzmann Machines Based on Weight Uncertainty. Chinese Journal of Software, 29(4), 1131--1142. (in Chinese)
Index Terms
- Units and Layers' Effects on Deep Boltzman Machines
Recommendations
Research on Point-wise Gated Deep Networks
Display Omitted We introduce pgRBMs into DBNs and present Point-wise Gated Deep Belief Networks.Similar to pgDBNs, Point-wise Gated Deep Boltzmann Machines are presented.We introduce dropout and weight uncertainty methods into pgRBMs.We discuss the ...
Boltzmann Machines for Image Denoising
Proceedings of the 23rd International Conference on Artificial Neural Networks and Machine Learning ICANN 2013 - Volume 8131Image denoising based on a probabilistic model of local image patches has been employed by various researchers, and recently a deep denoising autoencoder has been proposed in [2] and [17] as a good model for this. In this paper, we propose that another ...
Measuring the usefulness of hidden units in Boltzmann machines with mutual information
Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2019
942 pages
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 October 2019
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- general research projects of Beijing Educations Committee
Conference
CSAE 2019
CSAE 2019: The 3rd International Conference on Computer Science and Application Engineering
October 22 - 24, 2019
Sanya, China
Acceptance Rates
Overall Acceptance Rate 368 of 770 submissions, 48%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 50Total Downloads
- Downloads (Last 12 months)1
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in