Model Selection and Weight Sharing of Multi-layer Perceptrons

We present a method to learn and select a succinct multi-layer perceptron having shared weights. Weight sharing means a weight is allowed to have one of common weights. A near-zero common weight can be eliminated, called weight pruning. Our method iteratively merges and splits common weights based on 2nd-order criteria, escaping local optima through bidirectional clustering. Moreover, our method selects the optimal number of hidden units based on cross-validation. Our experiments showed that the proposed method can perfectly restore the original sharing structure for an artificial data set, and finds a small number of common weights for a real data set.

Authors and Affiliations

1. Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, 466-8555, Japan

   Yusuke Tanahashi & Ryohei Nakano

2. NTT Communication Science Laboratories, NTT Corporation, 2-4 Hikaridai, Seika, Soraku, Kyoto, 619-0237, Japan

   Kazumi Saito

Editors and Affiliations

1. School of Business, La Trobe University, 3086, Melbourne, Victoria, Australia

   Rajiv Khosla

2. Centre for SMART systems Engineering Research Centre, University of Brighton, Moulsecoomb, BN2 4GJ, Brighton, UK

   Robert J. Howlett

3. School of Electrical and Information Engineering, Knowledge Based Intelligent Engineering Systems Centre, University of South Australia, 5095, Mawson Lakes, SA, Australia

   Lakhmi C. Jain

Reprints and permissions

© 2005 Springer-Verlag Berlin Heidelberg

Policies and ethics