Handling imbalanced Dataset

Lesson 18 of 64 · Part 1 · Python & Data Basics · Data Pre-processing · bilingual notes (English + සිංහල)

Key principles

• Oversampling and undersampling fix the training data, not the truth. Techniques like SMOTE create more examples of the rare class so the model is forced to actually learn its pattern, instead of ignoring it.
  Oversampling සහ undersampling fix කරන්නෙ training data එක, ඇත්තම truth එක නෙවෙයි

Walkthrough

Imbalanced Dataset:

A dataset with an unequal class distribution

# importing the dependencies
import numpy as np
import pandas as pd

# loading the dataset to pandas DataFrame
credit_card_data = pd.read_csv('/content/credit_data.csv')

# first 5 rows of the dataframe
credit_card_data.head()

credit_card_data.tail()

# distribution of the two classes
credit_card_data['Class'].value_counts()

Output

0    284315
1       492
Name: Class, dtype: int64

This is Highly Imbalanced Dataset

0 --> Legit Transactions

1 --> Fraudulent Transactions

# separating the legit and fraudulent transactions
legit = credit_card_data[credit_card_data.Class == 0]
fraud = credit_card_data[credit_card_data.Class == 1]

print(legit.shape)
print(fraud.shape)

Output

(284315, 31)
(492, 31)

Under-sampling

Build a sample dataset containing similar distribution of Legit & Fraudulent Transactionds

Number of Fraudulent Transactions --> 492

legit_sample = legit.sample(n=492)

print(legit_sample.shape)

Output

(492, 31)

Concatenate the Two DataFrames

new_dataset = pd.concat([legit_sample, fraud], axis = 0)

new_dataset.head()

new_dataset.tail()

new_dataset['Class'].value_counts()

Output

1    492
0    492
Name: Class, dtype: int64

Keep going: ← Lesson 17 · Lesson 19 → · Track it in the Self Guide