GridSearchCV and RandomizedSearchCV
Algorithm ඕනම එකකට tune කරන knobs තියෙනවා (SVM එකේ C වගේ). Combination ඔක්කොම අතින් try කරන එක හෙමින්, error prone — ඒ search එක නිසි ලෙස automate කරන්නෙ මෙහෙමයි.
Every algorithm has knobs to tune (like C in SVM). Trying every combination by hand is slow and error-prone — this is how you automate that search properly.
Lesson 44 of 64 · Part 4 · Evaluating & Choosing Models · Section 8 · bilingual notes (English + සිංහල)
Why this matters. Every algorithm has knobs to tune (like C in SVM). Trying every combination by hand is slow and error-prone — this is how you automate that search properly.
සිංහලෙන්: Algorithm ඕනම එකකට tune කරන knobs තියෙනවා (SVM එකේ C වගේ). Combination ඔක්කොම අතින් try කරන එක හෙමින්, error prone — ඒ search එක නිසි ලෙස automate කරන්නෙ මෙහෙමයි.
What you'll use this for. This is the real-world workflow: build a baseline model, then let GridSearchCV or RandomizedSearchCV squeeze out the extra few percent of accuracy automatically.
සිංහලෙන්: සැබෑ ලෝකේ workflow එකයි මේක: baseline model එකක් හදනවා, එහෙනම් extra percentage ටික automatic ම GridSearchCV හෝ RandomizedSearchCV එකෙන් squeeze කරගන්නවා.
Key principles
- Grid search is exhaustive, random search is a smart shortcut. Grid search tries every single combination (thorough but slow); random search samples combinations randomly (faster, and often nearly as good with many hyperparameters).
Grid search එක exhaustive, random search එක smart shortcut එකක්
Walkthrough
Hyperparameter Tuning:
- GridSearchCV
- RandomizedSearchCV
ExamplePython
# importing the dependencies
import numpy as np
import pandas as pd
import sklearn.datasets
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCVWe will be working on the breast cancer dataset
ExamplePython
# loading the data from sklearn
breast_cancer_dataset = sklearn.datasets.load_breast_cancer()ExamplePython
print(breast_cancer_dataset)Output
Output
{'data': array([[1.799e+01, 1.038e+01, 1.228e+02, ..., 2.654e-01, 4.601e-01,
1.189e-01],
[2.057e+01, 1.777e+01, 1.329e+02, ..., 1.860e-01, 2.750e-01,
8.902e-02],
[1.969e+01, 2.125e+01, 1.300e+02, ..., 2.430e-01, 3.613e-01,
8.758e-02],
...,
[1.660e+01, 2.808e+01, 1.083e+02, ..., 1.418e-01, 2.218e-01,
7.820e-02],
[2.060e+01, 2.933e+01, 1.401e+02, ..., 2.650e-01, 4.087e-01,
1.240e-01],
[7.760e+00, 2.454e+01, 4.792e+01, ..., 0.000e+00, 2.871e-01,
7.039e-02]]), 'target': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0,
1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0,
1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1,
1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0,
… (29 more lines)ExamplePython
# loading the data to a data frame
data_frame = pd.DataFrame(breast_cancer_dataset.data, columns = breast_cancer_dataset.feature_names)ExamplePython
# print the first 5 rows of the dataframe
data_frame.head()| mean radius | mean texture | mean perimeter | mean area | mean smoothness | mean compactness | mean concavity | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 17.99 | 10.38 | 122.80 | 1001.0 | 0.11840 | 0.27760 | 0.3001 | … |
| 1 | 20.57 | 17.77 | 132.90 | 1326.0 | 0.08474 | 0.07864 | 0.0869 | … |
| 2 | 19.69 | 21.25 | 130.00 | 1203.0 | 0.10960 | 0.15990 | 0.1974 | … |
| 3 | 11.42 | 20.38 | 77.58 | 386.1 | 0.14250 | 0.28390 | 0.2414 | … |
| 4 | 20.29 | 14.34 | 135.10 | 1297.0 | 0.10030 | 0.13280 | 0.1980 | … |
ExamplePython
# adding the 'target' column to the data frame
data_frame['label'] = breast_cancer_dataset.targetExamplePython
# print the first 5 rows of the dataframe
data_frame.head()| mean radius | mean texture | mean perimeter | mean area | mean smoothness | mean compactness | mean concavity | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 17.99 | 10.38 | 122.80 | 1001.0 | 0.11840 | 0.27760 | 0.3001 | … |
| 1 | 20.57 | 17.77 | 132.90 | 1326.0 | 0.08474 | 0.07864 | 0.0869 | … |
| 2 | 19.69 | 21.25 | 130.00 | 1203.0 | 0.10960 | 0.15990 | 0.1974 | … |
| 3 | 11.42 | 20.38 | 77.58 | 386.1 | 0.14250 | 0.28390 | 0.2414 | … |
| 4 | 20.29 | 14.34 | 135.10 | 1297.0 | 0.10030 | 0.13280 | 0.1980 | … |
ExamplePython
# number of rows and Columns in this dataset
data_frame.shapeOutput
Output
(569, 31)ExamplePython
# checking for missing values
data_frame.isnull().sum()Output
Output
mean radius 0
mean texture 0
mean perimeter 0
mean area 0
mean smoothness 0
mean compactness 0
mean concavity 0
mean concave points 0
mean symmetry 0
mean fractal dimension 0
radius error 0
texture error 0
perimeter error 0
area error 0
smoothness error 0
compactness error 0
concavity error 0
concave points error 0
… (14 more lines)ExamplePython
# checking the distribution of Target Varibale
data_frame['label'].value_counts()Output
Output
1 357
0 212
Name: label, dtype: int641 --> Benign
0 --> Malignant
Separating the features and target
ExamplePython
X = data_frame.drop(columns='label', axis=1)
Y = data_frame['label']ExamplePython
print(X)Output
Output
mean radius mean texture ... worst symmetry worst fractal dimension
0 17.99 10.38 ... 0.4601 0.11890
1 20.57 17.77 ... 0.2750 0.08902
2 19.69 21.25 ... 0.3613 0.08758
3 11.42 20.38 ... 0.6638 0.17300
4 20.29 14.34 ... 0.2364 0.07678
.. ... ... ... ... ...
564 21.56 22.39 ... 0.2060 0.07115
565 20.13 28.25 ... 0.2572 0.06637
566 16.60 28.08 ... 0.2218 0.07820
567 20.60 29.33 ... 0.4087 0.12400
568 7.76 24.54 ... 0.2871 0.07039
[569 rows x 30 columns]ExamplePython
print(Y)Output
Output
0 0
1 0
2 0
3 0
4 0
..
564 0
565 0
566 0
567 0
568 1
Name: label, Length: 569, dtype: int64ExamplePython
X = np.asarray(X)
Y = np.asarray(Y)#GridSearchCV
GridSearchCV is used for determining the best parameters for our model
ExamplePython
# loading the SVC model
model = SVC()ExamplePython
# hyperparameters
parameters = {
'kernel':['linear','poly','rbf','sigmoid'],
'C':[1, 5, 10, 20]
}ExamplePython
# grid search
classifier = GridSearchCV(model, parameters, cv=5)ExamplePython
# fitting the data to our model
classifier.fit(X, Y)Output
Output
GridSearchCV(cv=5, estimator=SVC(),
param_grid={'C': [1, 5, 10, 20],
'kernel': ['linear', 'poly', 'rbf', 'sigmoid']})ExamplePython
classifier.cv_results_Output
Output
{'mean_fit_time': array([1.70692692e+00, 3.91397476e-03, 5.49354553e-03, 1.57924175e-02,
2.89461923e+00, 3.88875008e-03, 4.88286018e-03, 1.51048660e-02,
5.15353546e+00, 4.06928062e-03, 4.58331108e-03, 1.43068314e-02,
6.89355717e+00, 4.32796478e-03, 4.61492538e-03, 1.44116879e-02]),
'mean_score_time': array([0.00086455, 0.00099292, 0.00180144, 0.00392737, 0.00079465,
0.00090632, 0.00151749, 0.00350952, 0.00087113, 0.00088983,
0.00135775, 0.00334759, 0.00077076, 0.00086527, 0.00133314,
0.00336313]),
'mean_test_score': array([0.94553641, 0.90866325, 0.91217202, 0.46044093, 0.95081509,
0.92272939, 0.93150132, 0.41117839, 0.95256948, 0.920975 ,
0.92271386, 0.40239093, 0.9490607 , 0.91922062, 0.92094395,
0.39886664]),
'param_C': masked_array(data=[1, 1, 1, 1, 5, 5, 5, 5, 10, 10, 10, 10, 20, 20, 20, 20],
mask=[False, False, False, False, False, False, False, False,
False, False, False, False, False, False, False, False],
fill_value='?',
dtype=object),
'param_kernel': masked_array(data=['linear', 'poly', 'rbf', 'sigmoid', 'linear', 'poly',
… (56 more lines)ExamplePython
# best parameters
best_parameters = classifier.best_params_
print(best_parameters)Output
Output
{'C': 10, 'kernel': 'linear'}ExamplePython
# higest accuracy
highest_accuracy = classifier.best_score_
print(highest_accuracy)Output
Output
0.9525694767893185ExamplePython
# loading the results to pandas dataframe
result = pd.DataFrame(classifier.cv_results_)ExamplePython
result.head()| mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_C | param_kernel | params | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 1.706927 | 0.594114 | 0.000865 | 0.000041 | 1 | linear | {'C': 1, 'kernel': 'linear'} | … |
| 1 | 0.003914 | 0.000251 | 0.000993 | 0.000027 | 1 | poly | {'C': 1, 'kernel': 'poly'} | … |
| 2 | 0.005494 | 0.000258 | 0.001801 | 0.000090 | 1 | rbf | {'C': 1, 'kernel': 'rbf'} | … |
| 3 | 0.015792 | 0.000438 | 0.003927 | 0.000274 | 1 | sigmoid | {'C': 1, 'kernel': 'sigmoid'} | … |
| 4 | 2.894619 | 0.492362 | 0.000795 | 0.000013 | 5 | linear | {'C': 5, 'kernel': 'linear'} | … |
ExamplePython
grid_search_result = result[['param_C','param_kernel','mean_test_score']]ExamplePython
grid_search_result| param_C | param_kernel | mean_test_score | |
|---|---|---|---|
| 0 | 1 | linear | 0.945536 |
| 1 | 1 | poly | 0.908663 |
| 2 | 1 | rbf | 0.912172 |
| 3 | 1 | sigmoid | 0.460441 |
| 4 | 5 | linear | 0.950815 |
| 5 | 5 | poly | 0.922729 |
… 10 more rows
Highest Accuracy = 95.2%
Best Parameters = {'C':10, 'kernel':'linear'}
#RandomizedSearchCV
ExamplePython
# loading the SVC model
model = SVC()ExamplePython
# hyperparameters
parameters = {
'kernel':['linear','poly','rbf','sigmoid'],
'C':[1, 5, 10, 20]
}ExamplePython
# grid search
classifier = RandomizedSearchCV(model, parameters, cv=5)ExamplePython
# fitting the data to our model
classifier.fit(X, Y)Output
Output
RandomizedSearchCV(cv=5, estimator=SVC(),
param_distributions={'C': [1, 5, 10, 20],
'kernel': ['linear', 'poly', 'rbf',
'sigmoid']})ExamplePython
classifier.cv_results_Output
Output
{'mean_fit_time': array([1.30171299e-02, 2.58466721e-02, 7.38779540e+00, 5.75504303e-03,
3.93733978e-03, 5.13038635e-03, 1.67533398e-02, 1.37300825e+00,
1.45676136e-02, 4.33041320e+00]),
'mean_score_time': array([0.00338869, 0.00527906, 0.00092034, 0.00193195, 0.00099864,
0.00169048, 0.00426598, 0.0008255 , 0.00337229, 0.00076871]),
'mean_test_score': array([0.91922062, 0.41117839, 0.9490607 , 0.91217202, 0.92272939,
0.93150132, 0.46044093, 0.94553641, 0.39886664, 0.95256948]),
'param_C': masked_array(data=[20, 5, 20, 1, 5, 5, 1, 1, 20, 10],
mask=[False, False, False, False, False, False, False, False,
False, False],
fill_value='?',
dtype=object),
'param_kernel': masked_array(data=['poly', 'sigmoid', 'linear', 'rbf', 'poly', 'rbf',
'sigmoid', 'linear', 'sigmoid', 'linear'],
mask=[False, False, False, False, False, False, False, False,
False, False],
fill_value='?',
dtype=object),
… (29 more lines)ExamplePython
# best parameters
best_parameters = classifier.best_params_
print(best_parameters)Output
Output
{'kernel': 'linear', 'C': 10}ExamplePython
# higest accuracy
highest_accuracy = classifier.best_score_
print(highest_accuracy)Output
Output
0.9525694767893185ExamplePython
# loading the results to pandas dataframe
result = pd.DataFrame(classifier.cv_results_)ExamplePython
result.head()| mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_kernel | param_C | params | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.013017 | 0.004477 | 0.003389 | 0.003481 | poly | 20 | {'kernel': 'poly', 'C': 20} | … |
| 1 | 0.025847 | 0.004386 | 0.005279 | 0.000147 | sigmoid | 5 | {'kernel': 'sigmoid', 'C': 5} | … |
| 2 | 7.387795 | 2.442252 | 0.000920 | 0.000237 | linear | 20 | {'kernel': 'linear', 'C': 20} | … |
| 3 | 0.005755 | 0.000210 | 0.001932 | 0.000060 | rbf | 1 | {'kernel': 'rbf', 'C': 1} | … |
| 4 | 0.003937 | 0.000350 | 0.000999 | 0.000033 | poly | 5 | {'kernel': 'poly', 'C': 5} | … |
ExamplePython
randomized_search_result = result[['param_C','param_kernel','mean_test_score']]ExamplePython
randomized_search_result| param_C | param_kernel | mean_test_score | |
|---|---|---|---|
| 0 | 20 | poly | 0.919221 |
| 1 | 5 | sigmoid | 0.411178 |
| 2 | 20 | linear | 0.949061 |
| 3 | 1 | rbf | 0.912172 |
| 4 | 5 | poly | 0.922729 |
| 5 | 5 | rbf | 0.931501 |
… 4 more rows
Highest Accuracy = 95.2%
Best Parameters = {'C':10, 'kernel':'linear'}
Keep going: ← Lesson 43 · Lesson 45 → · Track it in the Self Guide
Finished this one?
Part of Self Guide · Stage 16: ML 4 — Evaluating & Choosing Models. Progress is saved in this browser.
