EDA based Deep Neural Network Parameter Optimization

Deep neural network has been applied in kinds of areas due to the excellent performance. The capacity of deep neural network relies on the parameter training algorithm which is always based on the gradient information. However, for deep neural network, it is harder and harder for training due to depth of the neural network and a large number of parameters. Therefore, kinds of techniques are proposed to cope with this problem. However, the training algorithm is almost based on the gradient information with inherent defect. Evolutionary algorithm has the advantage of global optimization capability, independent of gradient information etc. Estimation of distribution algorithm (EDA) is a typical evolutionary algorithm which relies on the probability model of population. Therefore, the EDA is adopted to train the weight and bias of deep neural network in his paper. However, the large scale optimization capability of EDA is limited. Thereby, an improved strategy is proposed to enhance the large scale optimization capability of EDA. In the improved scheme, a random selection strategy is carried out to select partial variables for probability modeling instead of all of the variables, in order to reduce the computing time and the probability of combination explosion. A simulation is carried out to exhibit the validity of the improved algorithm.