Model Selection in ML with Python

Ekම algorithm එකක්ම dataset ඕනම එකකට දිනන්නෙ නෑ — කුමන model එක පාවිච්චි කරන්නද guess කරනවා වෙනුවට, එකම data එකට models කිහිපයක් systematic ව compare කරන එකයි මේ පාඩම.

No single algorithm wins on every dataset — this lesson is about systematically comparing several models on the same data instead of guessing which one to use.

16 May 20265 min readSelf Guide · Stage 16#Evaluating & Choosing Models#Section 8#Machine learning

Lesson 45 of 64 · Part 4 · Evaluating & Choosing Models · Section 8 · bilingual notes (English + සිංහල)

Why this matters. No single algorithm wins on every dataset — this lesson is about systematically comparing several models on the same data instead of guessing which one to use.

සිංහලෙන්: Ekම algorithm එකක්ම dataset ඕනම එකකට දිනන්නෙ නෑ — කුමන model එක පාවිච්චි කරන්නද guess කරනවා වෙනුවට, එකම data එකට models කිහිපයක් systematic ව compare කරන එකයි මේ පාඩම.

What you'll use this for. In any real project, this comparison step — not gut feeling — is what tells you whether to ship Logistic Regression, SVM, or something else entirely.

සිංහලෙන්: සැබෑ project ඕනම එකක, Logistic Regression, SVM, නැත්නම් වෙන දෙයක් ship කරන්නද කියලා කියන්නෙ, hunch එකකින් නෙවෙයි, මේ comparison step එකෙන්.

Key principles

Compare models on the same metric, on the same test set, every time. Comparing one model's accuracy to another model's F1-score is meaningless — a fair comparison holds everything constant except the algorithm itself.
Model compare කරන්නෙ එකම metric එකෙන්, එකම test set එකෙන්, හැම වෙලාවෙම

Walkthrough

Importing the dependencies

ExamplePython

import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV

ExamplePython

# importing the models
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier

We will be working on the Heart Disease dataset

ExamplePython

# loading the csv data to a Pandas DataFrame
heart_data = pd.read_csv('/content/heart.csv')

ExamplePython

# print first 5 rows of the dataset
heart_data.head()

	age	sex	cp	trestbps	chol	fbs	restecg	…
0	63	1	3	145	233	1	0	…
1	37	1	2	130	250	0	1	…
2	41	0	1	130	204	0	0	…
3	56	1	1	120	236	0	1	…
4	57	0	0	120	354	0	1	…

ExamplePython

# number of rows and columns in the dataset
heart_data.shape

Output

Output

(303, 14)

ExamplePython

# checking for missing values
heart_data.isnull().sum()

Output

Output

age         0
sex         0
cp          0
trestbps    0
chol        0
fbs         0
restecg     0
thalach     0
exang       0
oldpeak     0
slope       0
ca          0
thal        0
target      0
dtype: int64

ExamplePython

# checking the distribution of Target Variable
heart_data['target'].value_counts()

Output

Output

1    165
0    138
Name: target, dtype: int64

1 --> Defective Heart

0 --> Healthy Heart

Splitting the Features and Target

ExamplePython

X = heart_data.drop(columns='target', axis=1)
Y = heart_data['target']

ExamplePython

print(X)

Output

Output

     age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  \
0     63    1   3       145   233    1        0      150      0      2.3   
1     37    1   2       130   250    0        1      187      0      3.5   
2     41    0   1       130   204    0        0      172      0      1.4   
3     56    1   1       120   236    0        1      178      0      0.8   
4     57    0   0       120   354    0        1      163      1      0.6   
..   ...  ...  ..       ...   ...  ...      ...      ...    ...      ...   
298   57    0   0       140   241    0        1      123      1      0.2   
299   45    1   3       110   264    0        1      132      0      1.2   
300   68    1   0       144   193    1        1      141      0      3.4   
301   57    1   0       130   131    0        1      115      1      1.2   
302   57    0   1       130   236    0        0      174      0      0.0   

     slope  ca  thal  
0        0   0     1  
1        0   0     2  
2        2   0     2  
3        2   0     2  
… (9 more lines)

ExamplePython

print(Y)

Output

Output

0      1
1      1
2      1
3      1
4      1
      ..
298    0
299    0
300    0
301    0
302    0
Name: target, Length: 303, dtype: int64

ExamplePython

 X = np.asarray(X)
 Y = np.asarray(Y)

Model Selection

Comparing the models with default hyperparameter values using Cross Validation

ExamplePython

# list of models
models = [LogisticRegression(max_iter=1000), SVC(kernel='linear'), KNeighborsClassifier(), RandomForestClassifier(random_state=0)]

ExamplePython

def compare_models_cross_validation():

  for model in models:

    cv_score = cross_val_score(model, X, Y, cv=5)
    mean_accuracy = sum(cv_score)/len(cv_score)
    mean_accuracy = mean_accuracy*100
    mean_accuracy = round(mean_accuracy, 2)

    print('Cross Validation accuracies for the',model,'=', cv_score)
    print('Acccuracy score of the ',model,'=',mean_accuracy,'%')
    print('---------------------------------------------------------------')

ExamplePython

compare_models_cross_validation()

Output

Output

Cross Validation accuracies for the LogisticRegression(max_iter=1000) = [0.80327869 0.8852459  0.85245902 0.86666667 0.75      ]
Acccuracy score of the  LogisticRegression(max_iter=1000) = 83.15 %
---------------------------------------------------------------
Cross Validation accuracies for the SVC(kernel='linear') = [0.81967213 0.8852459  0.80327869 0.86666667 0.76666667]
Acccuracy score of the  SVC(kernel='linear') = 82.83 %
---------------------------------------------------------------
Cross Validation accuracies for the KNeighborsClassifier() = [0.60655738 0.6557377  0.57377049 0.73333333 0.65      ]
Acccuracy score of the  KNeighborsClassifier() = 64.39 %
---------------------------------------------------------------
Cross Validation accuracies for the RandomForestClassifier(random_state=0) = [0.85245902 0.90163934 0.81967213 0.81666667 0.8       ]
Acccuracy score of the  RandomForestClassifier(random_state=0) = 83.81 %
---------------------------------------------------------------

Inference: For the Heart Disease dataset, Random Forest Classifier has the Highest accuracy value with default hyperparameter values

2. Comparing the models with different Hyperparameter values using GridSearchCV

ExamplePython

# list of models
models_list = [LogisticRegression(max_iter=10000), SVC(), KNeighborsClassifier(), RandomForestClassifier(random_state=0)]

ExamplePython

# creating a dictionary that contains hyperparameter values for the above mentioned models

model_hyperparameters = {
    

    'log_reg_hyperparameters': {
        
        'C' : [1,5,10,20]
    },

    'svc_hyperparameters': {
        
        'kernel' : ['linear','poly','rbf','sigmoid'],
        'C' : [1,5,10,20]
    },

    'KNN_hyperparameters' : {
        
        'n_neighbors' : [3,5,10]
    },

    'random_forest_hyperparameters' : {
        
        'n_estimators' : [10, 20, 50, 100]
    }
}

ExamplePython

type(model_hyperparameters)

Output

Output

dict

ExamplePython

print(model_hyperparameters.keys())

Output

Output

dict_keys(['log_reg_hyperparameters', 'svc_hyperparameters', 'KNN_hyperparameters', 'random_forest_hyperparameters'])

ExamplePython

model_hyperparameters['log_reg_hyperparameters']

Output

Output

{'C': [1, 5, 10, 20]}

ExamplePython

model_keys = list(model_hyperparameters.keys())
print(model_keys)

Output

Output

['log_reg_hyperparameters', 'svc_hyperparameters', 'KNN_hyperparameters', 'random_forest_hyperparameters']

ExamplePython

model_keys[0]

Output

Output

'log_reg_hyperparameters'

ExamplePython

model_hyperparameters[model_keys[0]]

Output

Output

{'C': [1, 5, 10, 20]}

Applying GridSearchCV

ExamplePython

def ModelSelection(list_of_models, hyperparameters_dictionary):

  result = []

  i = 0

  for model in list_of_models:

    key = model_keys[i]

    params = hyperparameters_dictionary[key]

    i += 1

    print(model)
    print(params)
    print('---------------------------------')

    classifier = GridSearchCV(model, params, cv=5)

    # fitting the data to classifier
    classifier.fit(X,Y)

    result.append({
        'model used' : model,
        'highest score' : classifier.best_score_,
        'best hyperparameters' : classifier.best_params_
    })

  result_dataframe = pd.DataFrame(result, columns = ['model used','highest score','best hyperparameters'])

  return result_dataframe

ExamplePython

ModelSelection(models_list, model_hyperparameters)

Output

Output

LogisticRegression(max_iter=10000)
{'C': [1, 5, 10, 20]}
---------------------------------
SVC()
{'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'C': [1, 5, 10, 20]}
---------------------------------
KNeighborsClassifier()
{'n_neighbors': [3, 5, 10]}
---------------------------------
RandomForestClassifier(random_state=0)
{'n_estimators': [10, 20, 50, 100]}
---------------------------------

	model used	highest score	best hyperparameters
0	LogisticRegression(max_iter=10000)	0.831585	{'C': 5}
1	SVC()	0.828306	{'C': 1, 'kernel': 'linear'}
2	KNeighborsClassifier()	0.643880	{'n_neighbors': 5}
3	RandomForestClassifier(random_state=0)	0.838087	{'n_estimators': 100}

Random Forest Classifier with n_estimators = 100 has the highest accuracy

Keep going: ← Lesson 44 · Lesson 46 → · Track it in the Self Guide

Finished this one?

Part of Self Guide · Stage 16: ML 4 — Evaluating & Choosing Models. Progress is saved in this browser.

Guide

Previous Next

Projects in this lesson

ML & Vision

Machine Learning Playbook & Deep Learning Lab

Interactive machine learning educational playground and hands-on laboratory covering computer vision, regression, and deep neural nets.

Model Selection in ML with Python

Key principles

Walkthrough

Importing the dependencies

We will be working on the Heart Disease dataset

Splitting the Features and Target

2. Comparing the models with different Hyperparameter values using GridSearchCV

Applying GridSearchCV

Random Forest Classifier with n_estimators = 100 has the highest accuracy

Machine Learning Playbook & Deep Learning Lab

Teaching Networking, DevOps and ML in Sinhala

Zero-Cost Hosting for InFocus SLUG 2025 — and 25 Hosting & Domain Options Compared

Life Vision: The Distributed Life

	age	sex	cp	trestbps	chol	fbs	restecg	…
0	63	1	3	145	233	1	0	…
1	37	1	2	130	250	0	1	…
2	41	0	1	130	204	0	0	…
3	56	1	1	120	236	0	1	…
4	57	0	0	120	354	0	1	…

	age	sex	cp	trestbps	chol	fbs	restecg	…
0	63	1	3	145	233	1	0	…
1	37	1	2	130	250	0	1	…
2	41	0	1	130	204	0	0	…
3	56	1	1	120	236	0	1	…
4	57	0	0	120	354	0	1	…

	age	sex	cp	trestbps	chol	fbs	restecg	…
0	63	1	3	145	233	1	0	…
1	37	1	2	130	250	0	1	…
2	41	0	1	130	204	0	0	…
3	56	1	1	120	236	0	1	…
4	57	0	0	120	354	0	1	…