Galle, LK--:--5/6 sites live

Precision, Recall, F1 Score Python implementation

Accuracy සහ confusion matrix එකෙන් මොකද වුනේ කියලා කියනවා; precision, recall, F1 වලින් ඔයාට ඇත්තටම worry කරන failure mode එකට, model එක "කොච්චර හොඳද" කියලා compare කරන්න පුළුවන් number එකක් ලැබෙනවා.

Accuracy and the confusion matrix tell you what happened; precision, recall, and F1 give you a single, comparable number for "how good" a model is at the failure mode you actually care about.

16 May 20267 min readSelf Guide · Stage 16#Evaluating & Choosing Models#Model Evaluation#Machine learning

Lesson 47 of 64 · Part 4 · Evaluating & Choosing Models · Model Evaluation · bilingual notes (English + සිංහල)

Why this matters. Accuracy and the confusion matrix tell you what happened; precision, recall, and F1 give you a single, comparable number for "how good" a model is at the failure mode you actually care about.

සිංහලෙන්: Accuracy සහ confusion matrix එකෙන් මොකද වුනේ කියලා කියනවා; precision, recall, F1 වලින් ඔයාට ඇත්තටම worry කරන failure mode එකට, model එක "කොච්චර හොඳද" කියලා compare කරන්න පුළුවන් number එකක් ලැබෙනවා.

What you'll use this for. In the medical projects (heart disease, diabetes), recall is usually prioritized over precision — this lesson is what lets you make and justify that exact choice.

සිංහලෙන්: Medical projects වල (heart disease, diabetes), සාමාන්‍යයෙන් precision එකට වඩා recall එකට priority දෙනවා — ඒ choice එක ගන්නත්, justify කරන්නත් ඉඩ දෙන්නෙ මේ පාඩම.

Key principles

  • Precision asks "of what I flagged, how much was right?"; recall asks "of what was actually true, how much did I catch?". A model can have perfect precision by flagging only one, very obvious case — and terrible recall because it missed everything else. F1 balances the two into one number.
    Precision අහන්නෙ "මම flag කරපු ඒවායින් කොච්චරද හරි?"; recall අහන්නෙ "ඇත්තටම තිබ්බ ඒවායින් කොච්චරක් මම අල්ලගත්තද?"

Walkthrough

Importing the Dependencies

ExamplePython
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

Data Collection & Processing

ExamplePython
# loading the csv data to a Pandas DataFrame
heart_data = pd.read_csv('heart.csv')
ExamplePython
# print first 5 rows of the dataset
heart_data.head()
agesexcptrestbpscholfbsrestecg…
0631314523310…
1371213025001…
2410113020400…
3561112023601…
4570012035401…
ExamplePython
# print last 5 rows of the dataset
heart_data.tail()
agesexcptrestbpscholfbsrestecg…
298570014024101…
299451311026401…
300681014419311…
301571013013101…
302570113023600…
ExamplePython
# number of rows and columns in the dataset
heart_data.shape

Output

Output
(303, 14)
ExamplePython
# getting some info about the data
heart_data.info()

Output

Output
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 303 entries, 0 to 302
Data columns (total 14 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   age       303 non-null    int64  
 1   sex       303 non-null    int64  
 2   cp        303 non-null    int64  
 3   trestbps  303 non-null    int64  
 4   chol      303 non-null    int64  
 5   fbs       303 non-null    int64  
 6   restecg   303 non-null    int64  
 7   thalach   303 non-null    int64  
 8   exang     303 non-null    int64  
 9   oldpeak   303 non-null    float64
 10  slope     303 non-null    int64  
 11  ca        303 non-null    int64  
 12  thal      303 non-null    int64  
… (3 more lines)
ExamplePython
# checking for missing values
heart_data.isnull().sum()

Output

Output
age         0
sex         0
cp          0
trestbps    0
chol        0
fbs         0
restecg     0
thalach     0
exang       0
oldpeak     0
slope       0
ca          0
thal        0
target      0
dtype: int64
ExamplePython
# statistical measures about the data
heart_data.describe()
agesexcptrestbpscholfbsrestecg…
count303.000000303.000000303.000000303.000000303.000000303.000000303.000000…
mean54.3663370.6831680.966997131.623762246.2640260.1485150.528053…
std9.0821010.4660111.03205217.53814351.8307510.3561980.525860…
min29.0000000.0000000.00000094.000000126.0000000.0000000.000000…
25%47.5000000.0000000.000000120.000000211.0000000.0000000.000000…
50%55.0000001.0000001.000000130.000000240.0000000.0000001.000000…

… 2 more rows

ExamplePython
# checking the distribution of Target Variable
heart_data['target'].value_counts()

Output

Output
1    165
0    138
Name: target, dtype: int64

1 --> Defective Heart

0 --> Healthy Heart

Splitting the Features and Target

ExamplePython
X = heart_data.drop(columns='target', axis=1)
Y = heart_data['target']
ExamplePython
print(X)

Output

Output
     age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  \
0     63    1   3       145   233    1        0      150      0      2.3   
1     37    1   2       130   250    0        1      187      0      3.5   
2     41    0   1       130   204    0        0      172      0      1.4   
3     56    1   1       120   236    0        1      178      0      0.8   
4     57    0   0       120   354    0        1      163      1      0.6   
..   ...  ...  ..       ...   ...  ...      ...      ...    ...      ...   
298   57    0   0       140   241    0        1      123      1      0.2   
299   45    1   3       110   264    0        1      132      0      1.2   
300   68    1   0       144   193    1        1      141      0      3.4   
301   57    1   0       130   131    0        1      115      1      1.2   
302   57    0   1       130   236    0        0      174      0      0.0   

     slope  ca  thal  
0        0   0     1  
1        0   0     2  
2        2   0     2  
3        2   0     2  
… (9 more lines)
ExamplePython
print(Y)

Output

Output
0      1
1      1
2      1
3      1
4      1
      ..
298    0
299    0
300    0
301    0
302    0
Name: target, Length: 303, dtype: int64

Splitting the Data into Training data & Test Data

ExamplePython
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, stratify=Y, random_state=2)
ExamplePython
print(X.shape, X_train.shape, X_test.shape)

Output

Output
(303, 13) (242, 13) (61, 13)

Model Training

ExamplePython
model = LogisticRegression(max_iter=1000)
ExamplePython
# training the LogisticRegression model with Training data
model.fit(X_train, Y_train)

Output

Output
LogisticRegression(max_iter=1000)

Model Evaluation

Accuracy Score

ExamplePython
from sklearn.metrics import accuracy_score
ExamplePython
# accuracy on training data
X_train_prediction = model.predict(X_train)
training_data_accuracy = accuracy_score(Y_train, X_train_prediction)
print(training_data_accuracy)

Output

Output
0.8553719008264463
ExamplePython
print('Accuracy on Training data : ', round(training_data_accuracy*100, 2), '%')

Output

Output
Accuracy on Training data :  85.54 %
ExamplePython
# accuracy on test data
X_test_prediction = model.predict(X_test)
test_data_accuracy = accuracy_score(Y_test, X_test_prediction)
print(test_data_accuracy)

Output

Output
0.8032786885245902
ExamplePython
print('Accuracy on Test data : ', round(test_data_accuracy*100, 2), '%')

Output

Output
Accuracy on Test data :  80.33 %

Precision

Precision is the ratio of number of True Positive to the total number of Predicted Positive. It measures, out of the total predicted positive, how many are actually positive.

ExamplePython
from sklearn.metrics import precision_score
ExamplePython
# precision for training data predictions
precision_train = precision_score(Y_train, X_train_prediction)
print('Training data Precision =', precision_train)

Output

Output
Training data Precision = 0.8299319727891157
ExamplePython
# precision for test data predictions
precision_test = precision_score(Y_test, X_test_prediction)
print('Test data Precision =', precision_test)

Output

Output
Test data Precision = 0.8181818181818182

Recall

Recall is the ratio of number of True Positive to the total number of Actual Positive. It measures, out of the total actual positive, how many are predicted as True Positive.

ExamplePython
from sklearn.metrics import recall_score
ExamplePython
# recall for training data predictions
recall_train = recall_score(Y_train, X_train_prediction)
print('Training data Recall =', recall_train)

Output

Output
Training data Recall = 0.9242424242424242
ExamplePython
# recall for test data predictions
recall_test = recall_score(Y_test, X_test_prediction)
print('Test data Recall =', recall_test)

Output

Output
Test data Recall = 0.8181818181818182

F1 Score

F1 Score is an important evaluation metric for binary classification that combines Precision & Recall. F1 Score is the harmonic mean of Precision & Recall.

ExamplePython
from sklearn.metrics import f1_score
ExamplePython
# F1 score for training data predictions
f1_score_train = f1_score(Y_train, X_train_prediction)
print('Training data F1 Score =', f1_score_train)

Output

Output
Training data F1 Score = 0.8745519713261649
ExamplePython
# F1 Score for test data predictions
f1_score_test = recall_score(Y_test, X_test_prediction)
print('Test data F1 Score =', f1_score_test)

Output

Output
Test data F1 Score = 0.8181818181818182

Precision, Recall, & F1 Score - function

ExamplePython
def precision_recall_f1_score(true_labels, pred_labels):

  precision_value = precision_score(true_labels, pred_labels)
  recall_value = recall_score(true_labels, pred_labels)
  f1_score_value = f1_score(true_labels, pred_labels)

  print('Precision =',precision_value)
  print('Recall =',recall_value)
  print('F1 Score =',f1_score_value)
ExamplePython
# classification metrics for training data
precision_recall_f1_score(Y_train, X_train_prediction)

Output

Output
Precision = 0.8299319727891157
Recall = 0.9242424242424242
F1 Score = 0.8745519713261649
ExamplePython
# classification metrics for test data
precision_recall_f1_score(Y_test, X_test_prediction)

Output

Output
Precision = 0.8181818181818182
Recall = 0.8181818181818182
F1 Score = 0.8181818181818182

Keep going: ← Lesson 46 · Lesson 48 → · Track it in the Self Guide