Data Standardization
"age: 30" සහ "salary: 50000" එකට දැක්කම, model එකකට වැරදියට හිතෙනවා salary එක 1000ගුණයක් වැදගත් කියලා — ඇත්තටම වැදගත් නිසා නෙවෙයි, scale එක නිසා විතරයි.
A model that sees "age: 30" and "salary: 50000" together will wrongly think salary matters 1000x more, purely because of its scale, not its actual importance.
Lesson 15 of 64 · Part 1 · Python & Data Basics · Data Pre-processing · bilingual notes (English + සිංහල)
Why this matters. A model that sees "age: 30" and "salary: 50000" together will wrongly think salary matters 1000x more, purely because of its scale, not its actual importance.
සිංහලෙන්: "age: 30" සහ "salary: 50000" එකට දැක්කම, model එකකට වැරදියට හිතෙනවා salary එක 1000ගුණයක් වැදගත් කියලා — ඇත්තටම වැදගත් නිසා නෙවෙයි, scale එක නිසා විතරයි.
What you'll use this for. Distance-based algorithms (KNN, SVM) and neural networks are especially sensitive to this — you'll standardize data before almost every one of them.
සිංහලෙන්: Distance-based algorithms (KNN, SVM) සහ neural networks මේකට ගොඩක් sensitive — ඒ ඔක්කොම වලට කලින් වගේම data එක standardize කරනවා.
Key principles
- Standardizing puts every feature on the same footing. After standardization, every column has mean 0 and standard deviation 1 — so the model judges features by their actual relationship to the outcome, not their units.
Standardize කරන එකෙන් හැම feature එකකටම එකම footing එකක් ලැබෙනවා
Walkthrough
Data Standardization:
The process of standardizing the data to a common format and common range
import numpy as np
import pandas as pd
import sklearn.datasets
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split# loading the dataset
dataset = sklearn.datasets.load_breast_cancer()print(dataset)Output
{'data': array([[1.799e+01, 1.038e+01, 1.228e+02, ..., 2.654e-01, 4.601e-01,
1.189e-01],
[2.057e+01, 1.777e+01, 1.329e+02, ..., 1.860e-01, 2.750e-01,
8.902e-02],
[1.969e+01, 2.125e+01, 1.300e+02, ..., 2.430e-01, 3.613e-01,
8.758e-02],
...,
[1.660e+01, 2.808e+01, 1.083e+02, ..., 1.418e-01, 2.218e-01,
7.820e-02],
[2.060e+01, 2.933e+01, 1.401e+02, ..., 2.650e-01, 4.087e-01,
1.240e-01],
[7.760e+00, 2.454e+01, 4.792e+01, ..., 0.000e+00, 2.871e-01,
7.039e-02]]), 'target': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0,
1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0,
1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1,
1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0,
… (29 more lines)# loading the data to a pandas dataframe
df = pd.DataFrame(dataset.data, columns=dataset.feature_names)df.head()| mean radius | mean texture | mean perimeter | mean area | mean smoothness | mean compactness | mean concavity | … | |
|---|---|---|---|---|---|---|---|---|
| 0 | 17.99 | 10.38 | 122.80 | 1001.0 | 0.11840 | 0.27760 | 0.3001 | … |
| 1 | 20.57 | 17.77 | 132.90 | 1326.0 | 0.08474 | 0.07864 | 0.0869 | … |
| 2 | 19.69 | 21.25 | 130.00 | 1203.0 | 0.10960 | 0.15990 | 0.1974 | … |
| 3 | 11.42 | 20.38 | 77.58 | 386.1 | 0.14250 | 0.28390 | 0.2414 | … |
| 4 | 20.29 | 14.34 | 135.10 | 1297.0 | 0.10030 | 0.13280 | 0.1980 | … |
df.shapeOutput
(569, 30)X = df
Y = dataset.targetprint(X)Output
mean radius mean texture ... worst symmetry worst fractal dimension
0 17.99 10.38 ... 0.4601 0.11890
1 20.57 17.77 ... 0.2750 0.08902
2 19.69 21.25 ... 0.3613 0.08758
3 11.42 20.38 ... 0.6638 0.17300
4 20.29 14.34 ... 0.2364 0.07678
.. ... ... ... ... ...
564 21.56 22.39 ... 0.2060 0.07115
565 20.13 28.25 ... 0.2572 0.06637
566 16.60 28.08 ... 0.2218 0.07820
567 20.60 29.33 ... 0.4087 0.12400
568 7.76 24.54 ... 0.2871 0.07039
[569 rows x 30 columns]print(Y)Output
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 1 0 1 1 1 1 1 0 0 1 0 0 1 1 1 1 0 1 0 0 1 1 1 1 0 1 0 0
1 0 1 0 0 1 1 1 0 0 1 0 0 0 1 1 1 0 1 1 0 0 1 1 1 0 0 1 1 1 1 0 1 1 0 1 1
1 1 1 1 1 1 0 0 0 1 0 0 1 1 1 0 0 1 0 1 0 0 1 0 0 1 1 0 1 1 0 1 1 1 1 0 1
1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 1 0 1 1 0 0 1 1 0 0 1 1 1 1 0 1 1 0 0 0 1 0
1 0 1 1 1 0 1 1 0 0 1 0 0 0 0 1 0 0 0 1 0 1 0 1 1 0 1 0 0 0 0 1 1 0 0 1 1
1 0 1 1 1 1 1 0 0 1 1 0 1 1 0 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 1 1 1 1 1 1 0 1 0 1 1 0 1 1 0 1 0 0 1 1 1 1 1 1 1 1 1 1 1 1
1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 0 1 1 1 1 0 0 0 1 1
1 1 0 1 0 1 0 1 1 1 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 0
0 1 0 0 1 1 1 1 1 0 1 1 1 1 1 0 1 1 1 0 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1
1 0 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 0 1 0 1 1 1 1 1 0 1 1
0 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 0 1
1 1 1 1 1 1 0 1 0 1 1 0 1 1 1 1 1 0 0 1 0 1 0 1 1 1 1 1 0 1 1 0 1 0 1 0 0
1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 0 0 0 0 0 0 1]Splitting the data into training data and test data
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=3)print(X.shape, X_train.shape, X_test.shape)Output
(569, 30) (455, 30) (114, 30)Standardize the data
print(dataset.data.std())Output
228.29740508276657scaler = StandardScaler()scaler.fit(X_train)Output
StandardScaler(copy=True, with_mean=True, with_std=True)X_train_standardized = scaler.transform(X_train)print(X_train_standardized)Output
[[ 1.40381088 1.79283426 1.37960065 ... 1.044121 0.52295995
0.64990763]
[ 1.16565505 -0.14461158 1.07121375 ... 0.5940779 0.44153782
-0.85281516]
[-0.0307278 -0.77271123 -0.09822185 ... -0.64047556 -0.31161687
-0.69292805]
...
[ 1.06478904 0.20084323 0.89267396 ... 0.01694621 3.06583565
-1.29952679]
[ 1.51308238 2.3170559 1.67987211 ... 1.14728703 -0.16599653
0.82816016]
[-0.73678981 -1.02636686 -0.74380549 ... -0.31826862 -0.40713129
-0.38233653]]X_test_standardized = scaler.transform(X_test)print(X_train_standardized.std())Output
1.0print(X_test_standardized.std())Output
0.8654541077212674Keep going: ← Lesson 14 · Lesson 16 → · Track it in the Self Guide
Finished this one?
Part of Self Guide · Stage 13: ML 1 — Python & Data Basics. Progress is saved in this browser.
