Numerical Dataset Pre Processing Use Case
මේ section එකේ ඉගෙන ගත්ත ඔක්කොම — missing values, standardization, encoding, splitting — එකට එකතු කරලා, මින් ඉස්සරහට ඕනම real dataset එකකට repeat කරන එකම pipeline එකට ගේනවා මේ පාඩම.
This lesson strings together everything from this section — missing values, standardization, encoding, splitting — into the one pipeline you'll repeat on every real dataset from now on.
Lesson 20 of 64 · Part 1 · Python & Data Basics · Data Pre-processing · bilingual notes (English + සිංහල)
Why this matters. This lesson strings together everything from this section — missing values, standardization, encoding, splitting — into the one pipeline you'll repeat on every real dataset from now on.
සිංහලෙන්: මේ section එකේ ඉගෙන ගත්ත ඔක්කොම — missing values, standardization, encoding, splitting — එකට එකතු කරලා, මින් ඉස්සරහට ඕනම real dataset එකකට repeat කරන එකම pipeline එකට ගේනවා මේ පාඩම.
What you'll use this for. Whenever you start a brand-new project, this is the checklist you mentally run through before writing a single line of model code.
සිංහලෙන්: අලුත් project එකක් පටන් ගන්නකොට, model code එකක් ලියන්නත් කලින් හිතින්ම run කරන checklist එකයි මේක.
Key principles
- Order matters: clean, then encode, then split, then scale. Splitting before cleaning risks leaking information between train and test sets — this pipeline exists to get that order right, every time.
පිළිවෙළ වැදගත්: clean කරන්න, එහෙනම් encode, එහෙනම් split, එහෙනම් scale
Walkthrough
Importing the Dependencies
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_splitData Collection & Pre-Processing
# loading the data from csv file to a pandas dataframe
diabetes_data = pd.read_csv('/content/diabetes.csv')# first 5 rows of the dataframe
diabetes_data.head()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 6 | 148 | 72 | 35 | 0 | 33.6 | 0.627 | … |
| 1 | 1 | 85 | 66 | 29 | 0 | 26.6 | 0.351 | … |
| 2 | 8 | 183 | 64 | 0 | 0 | 23.3 | 0.672 | … |
| 3 | 1 | 89 | 66 | 23 | 94 | 28.1 | 0.167 | … |
| 4 | 0 | 137 | 40 | 35 | 168 | 43.1 | 2.288 | … |
# number of rows & columns
diabetes_data.shapeOutput
(768, 9)diabetes_data.describe()| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | … | |
|---|---|---|---|---|---|---|---|---|
| count | 768.000000 | 768.000000 | 768.000000 | 768.000000 | 768.000000 | 768.000000 | 768.000000 | … |
| mean | 3.845052 | 120.894531 | 69.105469 | 20.536458 | 79.799479 | 31.992578 | 0.471876 | … |
| std | 3.369578 | 31.972618 | 19.355807 | 15.952218 | 115.244002 | 7.884160 | 0.331329 | … |
| min | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.078000 | … |
| 25% | 1.000000 | 99.000000 | 62.000000 | 0.000000 | 0.000000 | 27.300000 | 0.243750 | … |
| 50% | 3.000000 | 117.000000 | 72.000000 | 23.000000 | 30.500000 | 32.000000 | 0.372500 | … |
… 2 more rows
Separating Features and Target
X = diabetes_data.drop(columns='Outcome', axis =1)
Y = diabetes_data['Outcome']print(X)Output
Pregnancies Glucose BloodPressure ... BMI DiabetesPedigreeFunction Age
0 6 148 72 ... 33.6 0.627 50
1 1 85 66 ... 26.6 0.351 31
2 8 183 64 ... 23.3 0.672 32
3 1 89 66 ... 28.1 0.167 21
4 0 137 40 ... 43.1 2.288 33
.. ... ... ... ... ... ... ...
763 10 101 76 ... 32.9 0.171 63
764 2 122 70 ... 36.8 0.340 27
765 5 121 72 ... 26.2 0.245 30
766 1 126 60 ... 30.1 0.349 47
767 1 93 70 ... 30.4 0.315 23
[768 rows x 8 columns]print(Y)Output
0 1
1 0
2 1
3 0
4 1
..
763 0
764 0
765 0
766 1
767 0
Name: Outcome, Length: 768, dtype: int640 --> Non - Diabetic
1 --> Diabetic
Data Standardization
scaler = StandardScaler()standardized_data = scaler.fit_transform(X)print(standardized_data)Output
[[ 0.63994726 0.84832379 0.14964075 ... 0.20401277 0.46849198
1.4259954 ]
[-0.84488505 -1.12339636 -0.16054575 ... -0.68442195 -0.36506078
-0.19067191]
[ 1.23388019 1.94372388 -0.26394125 ... -1.10325546 0.60439732
-0.10558415]
...
[ 0.3429808 0.00330087 0.14964075 ... -0.73518964 -0.68519336
-0.27575966]
[-0.84488505 0.1597866 -0.47073225 ... -0.24020459 -0.37110101
1.17073215]
[-0.84488505 -0.8730192 0.04624525 ... -0.20212881 -0.47378505
-0.87137393]]X = standardized_dataprint(X)Output
[[ 0.63994726 0.84832379 0.14964075 ... 0.20401277 0.46849198
1.4259954 ]
[-0.84488505 -1.12339636 -0.16054575 ... -0.68442195 -0.36506078
-0.19067191]
[ 1.23388019 1.94372388 -0.26394125 ... -1.10325546 0.60439732
-0.10558415]
...
[ 0.3429808 0.00330087 0.14964075 ... -0.73518964 -0.68519336
-0.27575966]
[-0.84488505 0.1597866 -0.47073225 ... -0.24020459 -0.37110101
1.17073215]
[-0.84488505 -0.8730192 0.04624525 ... -0.20212881 -0.47378505
-0.87137393]]print(Y)Output
0 1
1 0
2 1
3 0
4 1
..
763 0
764 0
765 0
766 1
767 0
Name: Outcome, Length: 768, dtype: int64Splitting the dataset into Training data & Testing Data
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=2)print(X.shape, X_train.shape, X_test.shape)Output
(768, 8) (614, 8) (154, 8)Keep going: ← Lesson 19 · Lesson 21 → · Track it in the Self Guide
Finished this one?
Part of Self Guide · Stage 13: ML 1 — Python & Data Basics. Progress is saved in this browser.
