Train Test Split
Model එකක් ඉගෙන ගත්ත ඒ දත්තමයෙන්ම test කලොත්, ඇත්තටම perfect විදිහට පේනවා — ඒක intelligence නෙවෙයි, memorization එකක්. ඒක catch කරන්නෙ මෙහෙමයි.
If you test a model on the exact same data it learned from, of course it looks perfect — that's not intelligence, that's memorization. This is how you catch that.
Lesson 17 of 64 · Part 1 · Python & Data Basics · Data Pre-processing · bilingual notes (English + සිංහල)
Why this matters. If you test a model on the exact same data it learned from, of course it looks perfect — that's not intelligence, that's memorization. This is how you catch that.
සිංහලෙන්: Model එකක් ඉගෙන ගත්ත ඒ දත්තමයෙන්ම test කලොත්, ඇත්තටම perfect විදිහට පේනවා — ඒක intelligence නෙවෙයි, memorization එකක්. ඒක catch කරන්නෙ මෙහෙමයි.
What you'll use this for. You will run this exact function — train_test_split — at the start of almost every single ML project in this entire course.
සිංහලෙන්: මුළු course එකේම ML project ඕනම එකක ආරම්භයේදී, හරියටම මේ train_test_split function එක run කරනවා.
Key principles
- The test set is a preview of the future, not the past. The model must never see the test set during training — that's the entire point. It's your one honest measure of how the model handles data it's never met.
Test set එක කියන්නෙ අනාගතයේ preview එකක්, අතීතේ එකක් නෙවෙයි
Walkthrough
Importing the Dependencies
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn import svm
from sklearn.metrics import accuracy_scoreData Collection and Analysis
PIMA Diabetes Dataset
# loading the diabetes dataset to a pandas DataFrame
diabetes_dataset = pd.read_csv('/content/diabetes.csv')# printing the first 5 rows of the dataset
diabetes_dataset.head()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 6 | 148 | 72 | 35 | 0 | 33.6 | 0.627 | … |
| 1 | 1 | 85 | 66 | 29 | 0 | 26.6 | 0.351 | … |
| 2 | 8 | 183 | 64 | 0 | 0 | 23.3 | 0.672 | … |
| 3 | 1 | 89 | 66 | 23 | 94 | 28.1 | 0.167 | … |
| 4 | 0 | 137 | 40 | 35 | 168 | 43.1 | 2.288 | … |
# number of rows and Columns in this dataset
diabetes_dataset.shapeOutput
(768, 9)# getting the statistical measures of the data
diabetes_dataset.describe()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | … | |
|---|---|---|---|---|---|---|---|---|
| count | 768.000000 | 768.000000 | 768.000000 | 768.000000 | 768.000000 | 768.000000 | 768.000000 | … |
| mean | 3.845052 | 120.894531 | 69.105469 | 20.536458 | 79.799479 | 31.992578 | 0.471876 | … |
| std | 3.369578 | 31.972618 | 19.355807 | 15.952218 | 115.244002 | 7.884160 | 0.331329 | … |
| min | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.078000 | … |
| 25% | 1.000000 | 99.000000 | 62.000000 | 0.000000 | 0.000000 | 27.300000 | 0.243750 | … |
| 50% | 3.000000 | 117.000000 | 72.000000 | 23.000000 | 30.500000 | 32.000000 | 0.372500 | … |
… 2 more rows
diabetes_dataset['Outcome'].value_counts()Output
0 500
1 268
Name: Outcome, dtype: int640 --> Non-Diabetic
1 --> Diabetic
diabetes_dataset.groupby('Outcome').mean()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | Age | |
|---|---|---|---|---|---|---|---|---|
| Outcome | ||||||||
| 0 | 3.298000 | 109.980000 | 68.184000 | 19.664000 | 68.792000 | 30.304200 | 0.429734 | 31.190000 |
| 1 | 4.865672 | 141.257463 | 70.824627 | 22.164179 | 100.335821 | 35.142537 | 0.550500 | 37.067164 |
# separating the data and labels
X = diabetes_dataset.drop(columns = 'Outcome', axis=1)
Y = diabetes_dataset['Outcome']print(X)Output
Pregnancies Glucose BloodPressure ... BMI DiabetesPedigreeFunction Age
0 6 148 72 ... 33.6 0.627 50
1 1 85 66 ... 26.6 0.351 31
2 8 183 64 ... 23.3 0.672 32
3 1 89 66 ... 28.1 0.167 21
4 0 137 40 ... 43.1 2.288 33
.. ... ... ... ... ... ... ...
763 10 101 76 ... 32.9 0.171 63
764 2 122 70 ... 36.8 0.340 27
765 5 121 72 ... 26.2 0.245 30
766 1 126 60 ... 30.1 0.349 47
767 1 93 70 ... 30.4 0.315 23
[768 rows x 8 columns]print(Y)Output
0 1
1 0
2 1
3 0
4 1
..
763 0
764 0
765 0
766 1
767 0
Name: Outcome, Length: 768, dtype: int64Data Standardization
scaler = StandardScaler()scaler.fit(X)Output
StandardScaler(copy=True, with_mean=True, with_std=True)standardized_data = scaler.transform(X)print(standardized_data)Output
[[ 0.63994726 0.84832379 0.14964075 ... 0.20401277 0.46849198
1.4259954 ]
[-0.84488505 -1.12339636 -0.16054575 ... -0.68442195 -0.36506078
-0.19067191]
[ 1.23388019 1.94372388 -0.26394125 ... -1.10325546 0.60439732
-0.10558415]
...
[ 0.3429808 0.00330087 0.14964075 ... -0.73518964 -0.68519336
-0.27575966]
[-0.84488505 0.1597866 -0.47073225 ... -0.24020459 -0.37110101
1.17073215]
[-0.84488505 -0.8730192 0.04624525 ... -0.20212881 -0.47378505
-0.87137393]]X = standardized_data
Y = diabetes_dataset['Outcome']print(X)
print(Y)Output
[[ 0.63994726 0.84832379 0.14964075 ... 0.20401277 0.46849198
1.4259954 ]
[-0.84488505 -1.12339636 -0.16054575 ... -0.68442195 -0.36506078
-0.19067191]
[ 1.23388019 1.94372388 -0.26394125 ... -1.10325546 0.60439732
-0.10558415]
...
[ 0.3429808 0.00330087 0.14964075 ... -0.73518964 -0.68519336
-0.27575966]
[-0.84488505 0.1597866 -0.47073225 ... -0.24020459 -0.37110101
1.17073215]
[-0.84488505 -0.8730192 0.04624525 ... -0.20212881 -0.47378505
-0.87137393]]
0 1
1 0
2 1
3 0
4 1
… (7 more lines)SPLITTING THE DATA INTO TRAINING DATA & TESTING DATA
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=2)print(X.shape, X_train.shape, X_test.shape)Output
(768, 8) (614, 8) (154, 8)Keep going: ← Lesson 16 · Lesson 18 → · Track it in the Self Guide
Finished this one?
Part of Self Guide · Stage 13: ML 1 — Python & Data Basics. Progress is saved in this browser.
