Abstract
The performance of a deep neural network is affected by its configuration as well as its training process. Determining the configuration of a DNN and training its parameters are challenging tasks due to high-dimensional problems. Therefore, there is a need for methods that can optimize the configuration and parameters of a DNN. Most of the existing DNN optimization research concerns the optimization of DNN parameters, and there are only a few studies discussing the optimization of DNN configuration. In this paper, enhanced harmony search is proposed to optimize the configuration of a fully connected neural network. The proposed harmony search enhancement is conducted by introducing various types of harmony memory consideration rate and various types of harmony memory selection. Four types of harmony memory consideration rate are proposed in this research: constant rate, linear increase rate, linear decrease rate, and sigmoid rate. Two types of harmony memory selection are proposed in this research: rank-based selection and random selection. The combination of types of harmony memory consideration rate and types of selection generates eight harmony search scenarios. The performance of the proposed method is compared to random search and genetic algorithm using 12 datasets of classification problems. The experiment results show that the proposed harmony search outperforms random search in 8 out of 12 problems and approximately has the same performance in 4 problems. Harmony search also outperforms genetic algorithm in five problems, approximately has the same performance in six problems, and has worse performance in one problem. In addition, combining various types of harmony memory consideration rate and rank-based selection increases the performance of the ordinary harmony search. The combination of harmony memory consideration with linear increase rate and rank-based selection performs the best among all combinations. It is better than the ordinary harmony search in seven problems, approximately equal in three problems, and worse in two problems. The results show that the proposed method has some advantages in solving classification problems using a DNN. First, the configuration of the DNN is represented as an optimization problem so that it can be used to find a specific FCNN configuration that is suitable for a specific problem. Second, the approach is a global optimization approach as it tunes the DNN hyperparameter (configuration) as well as the DNN parameter (connection weight). Therefore, it is able to find the best combination of DNN configuration as well as its connection weight. However, there is a need to develop a strategy to balance the hyperparameter tuning and the parameter tuning. Inappropriate balance could lead to a high computational cost. Future research can be directed to balance the hyperparameter and parameter tuning during the solution search.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The dataset used in this article was derived from the UCI machine learning repository and can be downloaded from the following link: https://archive.ics.uci.edu/ml/datasets.php.
References
Agushaka JO, Akinola O, Ezugwu AE, Oyelade ON, Saha AK (2022) Advanced dwarf mongoose optimization for solving CEC 2011 and CEC 2017 benchmark problems. PLoS ONE 17(11):e0275346. https://doi.org/10.1371/journal.pone.0275346
Agushaka JO, Ezugwu AE, Olaide ON, Akinola O, Zitar RA, Abualigah L (2023a) Improved Dwarf mongoose optimization for constrained engineering design problems. J Bionic Eng 20:1263–1295. https://doi.org/10.1007/s42235-022-00316-8
Agushaka JO, Ezugwu AE, Abualigah L (2023b) Gazelle optimization algorithm: a novel nature-inspired metaheuristic optimizer. Neural Comput Appl 35(5):4099–4131. https://doi.org/10.1007/s00521-022-07854-6
Ahmadi M, Taghavirashidizadeh A, Masoumian A, Ghoushchi SJ, Pourasad Y (2022) DQRE-SCnet: a novel hybrid approach for selecting users in federated learning with deep-Q-reinforcement learning based on spectral clustering. J King Saud Univ Comput Inf Sci 34(9):7445–7458. https://doi.org/10.1016/j.jksuci.2021.08.019
Alibrahim H, Ludwig SA (2021) Hyperparameter optimization: comparing genetic algorithm against grid search and bayesian optimization. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp 1551–1559
AT&T Laboratories Cambridge (n.d.) The Database of Faces. Retrieved August 1, 2022, from https://cam-orl.co.uk/facedatabase.html
Badem H, Basturk A, Caliskan A, Yuksel ME (2017) A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited–memory BFGS optimization algorithms. Neurocomputing 266:506–526. https://doi.org/10.1016/j.neucom.2017.05.061
Baldi P (1995) Gradient descent learning algorithm overview: a general dynamical systems perspective. IEEE Trans Neural Networks 6(1):182–195. https://doi.org/10.1109/72.363438
Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13:281–305
Bren School of Information and Computer Science University of California, I (n.d.) UCI machine learning repository. Retrieved August 1, 2022, from https://archive.ics.uci.edu/ml/datasets.php
Davis A, Gill S, Wong R, Tayeb S (2020) Feature selection for deep neural networks in cyber security applications. In: 2020 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), pp 1–7
Deng L, Hinton G, Kingsbury B (2013) New types of deep neural network learning for speech recognition and related applications: an overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp 8599–8603
Dorigo M, di Caro G (1999) Ant colony optimization: a new meta-heuristic. In: Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406), pp 1470–1477
Fernandes Junior FE, Yen GG (2019) Particle swarm optimization of deep neural networks architectures for image classification. Swarm Evol Comput 49:62–74. https://doi.org/10.1016/j.swevo.2019.05.010
Fong S, Deb S, Yang X (2018) How meta-heuristic algorithms contribute to deep learning in the hype of big data analytics. In: Progress in Intelligent Computing Techniques: Theory, Practice, and Applications, Springer, pp 3–25
Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press
Hesamifard E, Takabi H, Ghasemi M (2019) Deep neural networks classification over encrypted data. In: Proceedings of the Ninth ACM Conference on Data and Application Security and Privacy, pp 97–108
Karaboga D, Ozturk C (2009) Neural networks training by artificial bee colony algorithm on pattern classification. Neural Netw World Int J Neural Mass Parallel Comput Inf Syst 19(3):279–292
Ke C, Weng NT, Yang Y, Yang ZM, Sumari P, Abualigah L, Kamel S, Agmadi M, Al-Qaness MAA, Forestiero A, Alsoud AR (2022) Mango varieties classification-based optimization with transfer learning and deep learning approaches in classification applications with deep learning and machine learning technologies. In: Abualigah L (ed), pp 45–65
Kennedy J, Eberhart R (1995). Particle swarm optimization. In: Proceedings of ICNN’95 - International Conference on Neural Networks, pp 1942–1948
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. In: 3rd International Conference for Learning Representations. http://arxiv.org/abs/1412.6980
Koopialipoor M, Tootoonchi H, Jahed Armaghani D, Tonnizam Mohamad E, Hedayat A (2019) Application of deep neural networks in predicting the penetration rate of tunnel boring machines. Bull Eng Geol Env 78(8):6347–6360. https://doi.org/10.1007/s10064-019-01538-7
Kumar P, Batra S, Raman B (2021) Deep neural network hyper-parameter tuning through twofold genetic approach. Soft Comput 25(13):8747–8771. https://doi.org/10.1007/s00500-021-05770-w
Kwon O, Kim HG, Ham MJ, Kim W, Kim G-H, Cho J-H, Kim N, Kim K (2018) A deep neural network for classification of melt-pool images in metal additive manufacturing. J Intell Manuf 31(2):375–386. https://doi.org/10.1007/s10845-018-1451-6
Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791
Liashchynskyi P, Liashchynskyi P (2019) Grid search, random search, genetic algorithm: a big comparison for NAS
Luo X, Qin W, Dong A, Sedraoui K, Zhou M (2021) Efficient and high-quality recommendations via momentum-incorporated parallel stochastic gradient descent-based learning. IEEE/CAA J Autom Sin 8(2):402–411. https://doi.org/10.1109/JAS.2020.1003396
Nasrabadi NM (2007) Pattern recognition and machine learning. J Electron Imaging 16(4):049901. https://doi.org/10.1117/1.2819119
Ndiaye E, Le T, Fercoq O, Salmon J, Takeuchi I (2018) Safe grid search with optimal complexity. In: International Conference on Machine Learning
Purnomo HD, Kristianto B, Somya R (2020) The use of local information sharing on soccer game optimization. Soft Comput 24(23):18057–18072. https://doi.org/10.1007/s00500-020-05060-x
Rajeena PPF, Orban R, Vadivel KS, Subramanian M, Muthusamy S, Elminaam DSA, Nabil A, Abulaigh L, Ahmadi M, Ali MAS (2022) A novel method for the classification of butterfly species using pre-trained CNN models. Electronics 11:2016. https://doi.org/10.3390/electronics11132016
Robbins H, Monro S (1951) A stochastic approximation method. Ann Math Stat 22:400–407
Saleem N, Khattak MI (2020) Deep neural networks for speech enhancement in complex-noisy environments. Int J Inter Multimed Artif Intell 6(1):84–90. https://doi.org/10.9781/ijimai.2019.06.001
Scikit-learn’s Development and Maintenance (n.d.) MLPClassifier. Retrieved August 1, 2022, from https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html
Sultan HH, Salem NM, Al-Atabany W (2019) Multi-classification of brain tumor images using deep neural network. IEEE Access 7:69215–69225. https://doi.org/10.1109/ACCESS.2019.2919122
Sun T, Vasarhalyi MA (2020) Predicting credit card delinquencies: an application of deep neural networks. In: Handbook of financial econometrics, mathematics, statistics, and machine learning, World Scientific, vol 25, pp 4349–4381
Talbi E-G (2009) Metaheuristics: from design to implementation. Wiley
Teodorovic D, Lucic P, Markovic G, Orco MD (2006) Bee colony optimization: principles and applications. In: 2006 8th Seminar on Neural Network Applications in Electrical Engineering, pp 151–156
Vigneswaran RK, Vinayakumar R, Soman KP, Poornachandran P (2018) Evaluating shallow and deep neural networks for network intrusion detection systems in cyber security. In: 2018 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp 1–6
Wang Y, Zhang H, Zhang G (2019) cPSO-CNN: an efficient PSO-based algorithm for fine-tuning hyper-parameters of convolutional neural networks. Swarm Evol Comput 49:114–123. https://doi.org/10.1016/j.swevo.2019.06.002
Yang L, Shami A (2020) On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing 415:295–316. https://doi.org/10.1016/j.neucom.2020.07.061
Yi D, Ji S, Bu S (2019) An enhanced optimization scheme based on gradient descent methods for machine learning. Symmetry 11(7):942. https://doi.org/10.3390/sym11070942
Yuan V, Chen C, Lei X, Yuan Y, Adnan RM (2018) Monthly runoff forecasting based on LSTM-ALO model. Stoch Env Res Risk Assess 32(8):2199–2212. https://doi.org/10.1007/s00477-018-1560-y
Zahedi L, Mohammadi FG, Rezapour S, Ohland MW, Amini MH (2021) Search algorithms for automated hyper-parameter tuning
Zhang T, Geem ZW (2019) Review of harmony search with respect to algorithm structure. Swarm Evol Comput 48:31–43. https://doi.org/10.1016/j.swevo.2019.03.012
Zhou B, Han C, Guo T (2021) Convergence of stochastic gradient descent in deep neural network. Acta Math Appl Sin Engl Ser 37(1):126–136. https://doi.org/10.1007/s10255-021-0991-2
Acknowledgements
This research was funded by Sophia Lecturing-Research Grant (STEC) program, Sophia University, Japan and Vice-Rector of Research, Innovation and Entrepreneurship at Satya Wacana Christian University, Indonesia.
Funding
This research was funded by Sophia Lecturing-Research Grant (STEC) program, Sophia University, Japan and Vice-Rector of Research, Innovation and Entrepreneurship at Satya Wacana Christian University, Indonesia.
Author information
Authors and Affiliations
Contributions
Hindriyanto Dwi Purnomo: made substantial contribution to the research design, interpretation of the experiment results for the article, and the writing of manuscript. Tad Gonsalves: made substantial contribution to the research design. Teguh Wahyono: made substantial contribution to the experiment design and implementation. Pratyaksa Ocsa Nugraha Saian: made substantial contribution to the experiment implementation and the writing of the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
This article was part of a research project on Neuroevolution funded by Sophia Lecturing-Research Grant (STEC) program, Sophia University, Japan and Vice-Rector of Research, Innovation and Entrepreneurship at Satya Wacana Christian University, Indonesia.
Ethical approval
This declaration is not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Purnomo, H.D., Gonsalves, T., Wahyono, T. et al. Enhanced harmony search for hyperparameter tuning of deep neural networks. Soft Comput 28, 9905–9919 (2024). https://doi.org/10.1007/s00500-024-09840-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-024-09840-7