Hyper-Parameter Optimization using Metaheuristic Algorithms

Abstract

Abstract: Machine learning algorithms are widely used in various applications. To properly implement them, their hyper-parameters need to be tuned. It is often necessary to know the ins and outs of ML learning algorithms as well as the proper hyper-parameter techniques. This paper presents two metaheuristic algorithms namely, Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) that can be used to improve the performance of machine learning algorithms. In this paper, we evaluated optimized algorithms for various machine learning algorithms namely, K-Nearest Neighbour (KNN), Support Vector Machine (SVM), Decision Tree (DT), and Random Forest (RF). For conducting experiments, we used four benchmark datasets namely, Breast cancer, Iris, Digits, Wine datasets from sklearn library are considered.  Experimental results show that PSO is performed well for optimizing ML models based on large search space. And it is observed that Decision-Tree technique performed poorly for ‘Digits’ dataset.

Index Terms: Optimization, Machine-Learning Models, Hyper-Parameters, Genetic Algorithm, Particle Swarm Optimization.