Machine learning notes. Hyperparameter tuning
Introduction on how to tune hyperparameters in machine learning models
Machine learning models typically involve a parameter set that it is learnt during the training process. However, machine learning models also contain parameters that are not learnable and must be specified before the training process begins. In this notebook, we will review some commonly used methods to establish good hyperparameters values. In particular, we will review the following approaches
- Grid search
- Random search
- Informed search
Machine learning models typically consist of a set of parameters that define their learnability. It is these parameters that somehow we try to estimate during training. However, most machine learning models also contain parameters that we need to establish before training begins. These parameters affect the learning process but they are not learnable themselves. We call these parameters as hyperparameters.
Hyperparameters are typically set before the modeling process begins. For example the number of clusters is a hyperparameter that the application needs to establish before running the algorithm. Thus, the crucial elemet that distinguishes parameters from hyperparameters is that the former are learnt by the model whilst the latter are set by the application.
In the sequel, we will differentiate hyperparameters into two categories. Namely, parameters that affect the model performance and parameters that do not. an example of the latter category is the number of cpu cores that we want to use when training the model. Although, this impacts the training time, it does not impact how the model performs on unseen data or in other words the model's performance.
Now that we have the needed definitions out of the way, we turn our attention to the main topic of this notebook. Namely, how do we set the optimal values for the model hyperparameters. Unfortunately, there is not a clear answer to this question. Hyperparameters are specific to each algorithm. However, there are available some general guidelines and tips that we can follow. Let's review some of the top tips.
First, we need to identify which hyperparameters values are in conflict. For example, if we are using sklearn's logistic regression model, the solver and penalty parameters have options that may be in conflict; the elasticnet penalty is only supported by the saga solver.
Another point to be aware of is that some hyperparameter values are simply silly. For example, setting the number of clusters equal to one when performing K-means clustering or equal to the number of points in the dataset, does not sound very meaningful. Similarly, setting the number of neighbors in a kNN algorithm equal to one is not very wise.
Below, we will review the following methods for hyperparamter tuning
- Grid search
- Random search
- Informed search
Let's start with grid search.
Grid search performs an exhaustive search over a specified parameter values for an estimator. We can visualize this as a two dimensional grid. At each point of the grid, a different combination of parameters is examined. For example, consider an artificial model with three hyperparamters $a, b$ and $c$. Each of these parameters has the following values sets; $a \in [a_1, a_2, a_3], b \in [b_1, b_2], c \in [c_1, c_2, c_3]$. Grid search performs exhaustive search by forming all the possible triplets and fitting the model using the identified values. Let's see how to perform grid search in sklearn. Overall the steps of using grid search in sklearn are as follows
- Choose the algorithm to tune the hyperparameters (
estimator) - Define which hyperparameters to tune (
param_grid) - Define the range of values for each hyperparameter
- Decide of the cross-validation scheme to use (
cv) - Define the score function to be used when deciding which model is the best (
scoring)
The following example, taken from here, shows how to use grid search
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from sklearn.svm import SVC
digits = datasets.load_digits()
# To apply an classifier on this data, we need to flatten the image, to
# turn the data in a (samples, feature) matrix:
n_samples = len(digits.images)
X = digits.images.reshape((n_samples, -1))
y = digits.target
# Split the dataset in two equal parts
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.5, random_state=0)
tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],
'C': [1, 10, 100, 1000]},
{'kernel': ['linear'], 'C': [1, 10, 100, 1000]},]
clf = GridSearchCV(estimator=SVC(), param_grid=tuned_parameters, scoring='precision_macro')
clf.fit(X_train, y_train)
print("Best parameters set found on development set:")
print()
print(clf.best_params_)
print()
print("Grid scores on development set:")
print()
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))
print()
print("Detailed classification report:")
print()
print("The model is trained on the full development set.")
print("The scores are computed on the full evaluation set.")
print()
y_true, y_pred = y_test, clf.predict(X_test)
print(classification_report(y_true, y_pred))
print()
The output from GridSearchCV can be categorized into three different groups
- Results log:
cv_results_ - Best results:
best_index_,best_params_andbest_score_ - Other extra information such as
refit_time_, andscorer_
The next method we will review is random search. According to wikipedia, random search (RS) is a family of numerical optimization methods that do not require the gradient of the problem to be optimized, and RS can hence be used on functions that are not continuous or differentiable. Such optimization methods are also known as direct-search, derivative-free, or black-box methods.
In general, random search is similar to the grid search approach we reviewed above. In particular,
- We have to define an estimator
- The parameters to be tuned and their range of values
- Establish a cross-validation scheme
- Establish a scoring functon
We won't go into the details of why random search works. Instead let's see how to perform random search with scikit-learn. The following example, is a copy verbatim from scikit-learn
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform
iris = load_iris()
logistic = LogisticRegression(solver='saga', tol=1e-2, max_iter=2, random_state=0)
distributions = dict(C=uniform(loc=0, scale=4), penalty=['l2', 'l1'])
clf = RandomizedSearchCV(logistic, distributions, random_state=0)
search = clf.fit(iris.data, iris.target)
search.best_params_
Grid search performs exhaustive search by trying out all possible combinations. On the other hand, random search selects a subset of combinations. Given this, it requires that we establish a sampling methodology. Grid search is more computationally expensive than random search. However, it is guaranteed to find the best score in the sample space. Random search is not guaranteed to find the best score, but it is likely to find a good one faster than grid search.
Both grid and random search algorithms are uninformed search algorithms. What this means is that algorithms do not use any form of information in order to improve the searching. In this section, we will review some informed search algorithms
In this approach, we start with a rough estimate and iteratively we refine our search.This approach utilizes both grid and random search. Here are the general steps we can follow towards this direction.
- Start with random search
- Find the areas in the sampling space that look promising
- Do a grid search in these smaller areas
- Continue until a good or optimal score is achieved
Other approaches to informed search include Bayesian statistics and genetic algorithms. However, we won't go into these directions as they can easily form the subject of whole books on their own. You can check the article A Conceptual Explanation of Bayesian Hyperparameter Optimization for Machine Learning to get a conceptual view about the subject.
-
Hyperparameter tuning in Pythoncourse from Datacamp - Hyperparameter optimization
- Random search