Comparing Randomized Search and Grid Search for Hyperparameter Estimation in Scikit Learn
Last Updated : 26 Apr, 2025
Hyperparameters are the parameters that determine the behavior and performance of a machine-learning model. These parameters are not learned during training but are instead set prior to training. The process of finding the optimal values for these hyperparameters is known as hyperparameter optimization, and it is an important step in the development of any Machine Learning model.
There are many different methods for performing hyperparameter optimization, but two of the most commonly used methods are grid search and randomized search. In this blog post, we will compare these two methods and provide examples of how to implement them using the Scikit Learn library in Python.
Grid Search Hyperparameter Estimation
Grid search is a method for hyperparameter optimization that involves specifying a list of values for each hyperparameter that you want to optimize, and then training a model for each combination of these values. For example, if you want to optimize two hyperparameters, alpha and beta, with grid search, you would specify a list of values for alpha and a separate list of values for the beta. The grid search algorithm would then train a model using every combination of these values and evaluate the performance of each model. The optimal values for the hyperparameters are then chosen based on the performance of the models.
Additionally, it is recommended to use cross-validation when performing hyperparameter optimization with either grid search or randomized search. Cross-validation is a technique that involves splitting the training data into multiple sets and training the model multiple times, each time using a different subset of the data as the validation set. This can provide a more accurate estimate of the model’s performance and help to avoid overfitting.
Here is an example of how to implement grid search in Scikit Learn:
Python3
from sklearn.model_selection import GridSearchCV
param_grid = {
'alpha': [0.01, 0.1, 1.0, 10.0],
'beta': [0.01, 0.1, 1.0, 10.0]
}
model = SomeModel()
grid_search = GridSearchCV(model, param_grid)
grid_search.fit(X, y)
print(grid_search.best_params_)
|
In this example, we define a dictionary called param_grid that specifies the possible values for the hyperparameters alpha and beta. We then use the GridSearchCV class from sklearn.model_selection module to perform grid search using these values. The fit method is used to train the model with the different combinations of hyperparameters, and the best_params_ attribute is used to access the optimal values for the hyperparameters.
One disadvantage of grid search is that it can be computationally expensive, especially when optimizing many hyperparameters or when the list of possible values for each hyperparameter is large. This is because grid search trains a separate model for every combination of hyperparameter values, which can quickly become infeasible as the number of combinations grows.
Randomized Search for Hyperparameter Estimation
Randomized search is another method for hyperparameter optimization that can be more efficient than grid search in some cases. With randomized search, instead of specifying a list of values for each hyperparameter, you specify a distribution for each hyperparameter. The randomized search algorithm will then sample values for each hyperparameter from its corresponding distribution and train a model using the sampled values. This process is repeated a specified number of times, and the optimal values for the hyperparameters are chosen based on the performance of the models.
Here is an example of how to implement randomized search in Scikit Learn:
Python3
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform
param_distributions = {
'alpha': uniform(0.01, 10.0),
'beta': uniform(0.01, 10.0)
}
model = SomeModel()
random_search = RandomizedSearchCV(model,
param_distributions)
random_search.fit(X, y)
print(random_search.best_params_)
|
In this example, we define a dictionary called param_distributions that specifies the distributions for the hyperparameters alpha and beta. We use the uniform distribution from the scipy.stats module, which specifies a range of values for each hyperparameter. We then use the RandomizedSearchCV class from the sklearn.model_selection module to perform a randomized search using these distributions. The fit method is used to train the model with the different combinations of hyperparameters, and the best_params_ attribute is used to access the optimal values for the hyperparameters.
One advantage of randomized search is that it can be more efficient than grid search in some cases since it does not train a separate model for every combination of hyperparameter values. Instead, it trains a specified number of models using random samples from the hyperparameter distributions. This can be more efficient when optimizing many hyperparameters or when the distributions for the hyperparameters have a large range of values.
It is important to note that both grid search and randomized search are only effective if the hyperparameter space is properly defined. In other words, it is crucial to specify a reasonable range of values for the hyperparameters that you want to optimize. If the range of values is too small, the optimal values may not be found, while if the range of values is too large, the search process may be inefficient and take a long time to complete.
Comparing Grid Search and Randomized Search
In this example, we are using a dataset consisting of 200 samples with 10 features each, and a binary target variable. We are fitting a Random Forest classifier with a variety of hyperparameters: the number of trees in the forest (n_estimators), the maximum depth of each tree (max_depth), the minimum number of samples required to split an internal node (min_samples_split), and whether or not to use bootstrapped samples when building the trees (bootstrap).
To compare RandomizedSearchCV and GridSearchCV, we define a search space for the hyperparameters using a dictionary, and then pass this dictionary to both RandomizedSearchCV and GridSearchCV along with the model and the number of cross-validation folds (cv). For RandomizedSearchCV, we also specify the number of iterations (n_iter) to sample from the search space.
Python3
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV,\
GridSearchCV
from scipy.stats import uniform
X = np.random.rand(200, 10)
y = np.random.randint(2, size=200)
|
RandomizedSearchCV works by sampling random combinations of hyperparameters from the search space and evaluating them using cross-validation. It continues sampling and evaluating until it has tried the specified number of iterations. This means that it does not explore the entire search space, but rather a randomly selected subset of it. This can be more efficient than GridSearchCV, which explores the entire search space, but it may also be less likely to find the true optimal set of hyperparameters.
Python3
model = RandomForestClassifier()
param_grid = {
'n_estimators': [10, 50, 100, 200],
'max_depth': [None, 5, 10, 15],
'min_samples_split': np.linspace(0.1, 1, 11),
'bootstrap': [True, False]
}
random_search = RandomizedSearchCV(
model,
param_grid,
cv=5,
n_iter=10,
random_state=42)
random_search.fit(X, y)
|
On the other hand, GridSearchCV exhaustively searches the entire search space by trying every possible combination of hyperparameters. This can be very computationally expensive, especially if the search space is large or if the model takes a long time to fit. However, it is guaranteed to find the optimal set of hyperparameters if it is able to complete the search.
Python3
grid_search = GridSearchCV(model, param_grid, cv=5)
grid_search.fit(X, y)
print(f"Best hyperparameters found by RandomizedSearchCV: {random_search.best_params_}")
print(f"Best hyperparameters found by GridSearchCV: {grid_search.best_params_}")
|
Output:
Best hyperparameters found by RandomizedSearchCV: {'n_estimators': 100, 'min_samples_split': 0.5499999999999999, 'max_depth': 10, 'bootstrap': True} Best hyperparameters found by GridSearchCV: {'bootstrap': False, 'max_depth': None, 'min_samples_split': 1.0, 'n_estimators': 10}Why RandomizedSearchCV is better than GridSearchCV?
One advantage of RandomizedSearchCV over GridSearchCV is that RandomizedSearchCV can be more efficient if the search space is large since it only samples a subset of the possible combinations rather than evaluating them all. This can be especially useful if the model is computationally expensive to fit, or if the hyperparameters have continuous values rather than discrete ones. In these cases, it may not be feasible to explore the entire search space using GridSearchCV.
Another advantage of RandomizedSearchCV is that it can be more robust to the risk of overfitting since it does not exhaustively search the entire search space. If the hyperparameter search space is very large and the model is relatively simple, it is possible that GridSearchCV could overfit to the training data by finding a set of hyperparameters that works well on the training set but not as well on unseen data. RandomizedSearchCV can help mitigate this risk by sampling randomly from the search space rather than evaluating every combination.
It is worth noting that both RandomizedSearchCV and GridSearchCV can be computationally expensive, especially if the model is complex and the search space is large. In these cases, it may be necessary to use techniques such as parallelization or early stopping to speed up the search process.
Conclusion
In conclusion, grid search and randomized search are two commonly used methods for hyperparameter optimization in machine learning. Both methods have their strengths and weaknesses, and which one is more suitable for a given problem will depend on the specific circumstances. It is important to properly define the hyperparameter space and to use cross-validation when performing hyperparameter optimization to obtain the best results.
Similar Reads
Demonstration of multi-metric evaluation on cross_val_score and GridSearchCV in Scikit Learn
In scikit learn, we can demonstrate multi-metric evaluation with the help of two functions cross_val_score and GridSearchCV. They help you check the performance of the model based on multiple metrics with a single click rather than writing repetitive code. In this article, we will first discuss the
4 min read
CatBoost Grid search and random search
Finding the best model and setting it up for optimum performance may be difficult in the realm of machine learning. Thankfully, methods like Grid Search and Random Search may be used to help. We shall clarify these techniques in this article, focusing on CatBoost, a potential gradient-boosting libra
8 min read
Shrinkage Covariance Estimation in Scikit Learn
The Ledoit and Wolf proposed a formula for shrinkage which is generally used for regularizing the usual maximum likelihood estimation. This formula is called the Ledoit-Wolf covariance estimation formula. This formula is able to compute asymptotically optimal shrinkage parameters by minimizing the m
3 min read
Comparing various online solvers in Scikit Learn
Scikit Learn is a popular Python library that provides a wide range of machine-learning algorithms and tools. One of the key features of Scikit Learn is the ability to solve optimization problems using various online solvers. In this article, we will compare some of the most commonly used online sol
4 min read
Recursive Feature Elimination with Cross-Validation in Scikit Learn
In this article, we will earn how to implement recursive feature elimination with cross-validation using scikit learn package in Python. What is Recursive Feature Elimination (RFE)? Recursive Feature Elimination (RFE) is a feature selection algorithm that is used to select a subset of the most relev
5 min read
CatBoost Parameters and Hyperparameters
For gradient boosting on decision trees, CatBoost is a well-liked open-source toolkit. It was created by Yandex and may be applied to a range of machine-learning issues, including classification, regression, ranking, and more. Compared to other boosting libraries, CatBoost has a number of benefits,
12 min read
Building a Custom Estimator for Scikit-learn: A Comprehensive Guide
Scikit-learn is a powerful machine learning library in Python that offers a wide range of tools for data analysis and modeling. One of its best features is the ease with which you can create custom estimators, allowing you to meet specific needs. In this article, we will walk through the process of
5 min read
Normal and Shrinkage Linear Discriminant Analysis for Classification in Scikit Learn
In this article, we will try to understand the difference between Normal and Shrinkage Linear Discriminant Analysis for Classification. We will try to implement the same using sci-kit learn library in Python. But first, let's try to understand what is LDA. What is Linear discriminant analysis (LDA)?
4 min read
Compare Stochastic Learning Strategies for MLPClassifier in Scikit Learn
A stochastic learning strategy is a method for training a Machine Learning model using stochastic optimization algorithms. These algorithms update the model's weights and biases using a randomly selected subset of the training data, rather than using the entire dataset. This can improve convergence
5 min read
How to tune a Decision Tree in Hyperparameter tuning
Decision trees are powerful models extensively used in machine learning for classification and regression tasks. The structure of decision trees resembles the flowchart of decisions helps us to interpret and explain easily. However, the performance of decision trees highly relies on the hyperparamet
14 min read