Text Data Pre Processing Use Case

Models වලට තේරෙන්නෙ numbers, sentences නෙවෙයි. මේක තමයි ඒ පාලම — raw text එකක් algorithm එකකට වැඩ කරන්න පුළුවන් numeric form එකකට හරවන එක.

Models understand numbers, not sentences. This is the bridge — turning raw text into a numeric form an algorithm can actually work with.

8 May 20265 min readSelf Guide · Stage 13#Python & Data Basics#Data Pre-processing#Machine learning

Lesson 21 of 64 · Part 1 · Python & Data Basics · Data Pre-processing · bilingual notes (English + සිංහල)

Why this matters. Models understand numbers, not sentences. This is the bridge — turning raw text into a numeric form an algorithm can actually work with.

සිංහලෙන්: Models වලට තේරෙන්නෙ numbers, sentences නෙවෙයි. මේක තමයි ඒ පාලම — raw text එකක් algorithm එකකට වැඩ කරන්න පුළුවන් numeric form එකකට හරවන එක.

What you'll use this for. Any project involving reviews, emails, or messages (like the Spam Mail project later) depends on this exact pre-processing pipeline.

සිංහලෙන්: Reviews, emails, messages එක්ක වැඩ කරන project ඕනම එකක් (පස්සෙ එන Spam Mail project එක වගේ) මේ pre-processing pipeline එක උඩ තමයි රඳා පවතින්නෙ.

Key principles

Lowercasing and removing punctuation reduce noise before counting. Without this step, "Free" and "free!" are treated as two completely different words — which quietly hurts model accuracy.
Count කරන්නත් කලින් lowercase කරන එකයි punctuation අයින් කරන එකයි noise අඩු කරනවා

Walkthrough

Importing the Dependencies

ExamplePython

import numpy as np
import pandas as pd
import re
import nltk 
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split

ExamplePython

nltk.download('stopwords')

Output

Output

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

Output

Output

True

ExamplePython

# printing the stopwords
print(stopwords.words('english'))

Output

Output

['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'hi

Data Pre-Processing

ExamplePython

# load the data to a pandas dataframe
news_data = pd.read_csv('/content/drive/MyDrive/Datasets/Fake News Dataset/train.csv')

ExamplePython

# first 5 rows of the dataset
news_data.head()

	id	title	author	text	label
0	0	House Dem Aide: We Didn’t Even See Comey’s Let...	Darrell Lucus	House Dem Aide: We Didn’t Even See Comey’s Let...	1
1	1	FLYNN: Hillary Clinton, Big Woman on Campus - ...	Daniel J. Flynn	Ever get the feeling your life circles the rou...	0
2	2	Why the Truth Might Get You Fired	Consortiumnews.com	Why the Truth Might Get You Fired October 29, ...	1
3	3	15 Civilians Killed In Single US Airstrike Hav...	Jessica Purkiss	Videos 15 Civilians Killed In Single US Airstr...	1
4	4	Iranian woman jailed for fictional unpublished...	Howard Portnoy	Print \nAn Iranian woman has been sentenced to...	1

0 --> Real News

1 --> Fake News

ExamplePython

news_data.shape

Output

Output

(20800, 5)

ExamplePython

# checking for missing values
news_data.isnull().sum()

Output

Output

id           0
title      558
author    1957
text        39
label        0
dtype: int64

ExamplePython

# replacing the missing values with null string
news_data = news_data.fillna('')

ExamplePython

# merging th eauthor name and news title
news_data['content'] = news_data['author']+' '+news_data['title']

ExamplePython

# first 5 rows of the dataset
news_data.head()

	id	title	author	text	label	content
0	0	House Dem Aide: We Didn’t Even See Comey’s Let...	Darrell Lucus	House Dem Aide: We Didn’t Even See Comey’s Let...	1	Darrell Lucus House Dem Aide: We Didn’t Even S...
1	1	FLYNN: Hillary Clinton, Big Woman on Campus - ...	Daniel J. Flynn	Ever get the feeling your life circles the rou...	0	Daniel J. Flynn FLYNN: Hillary Clinton, Big Wo...
2	2	Why the Truth Might Get You Fired	Consortiumnews.com	Why the Truth Might Get You Fired October 29, ...	1	Consortiumnews.com Why the Truth Might Get You...
3	3	15 Civilians Killed In Single US Airstrike Hav...	Jessica Purkiss	Videos 15 Civilians Killed In Single US Airstr...	1	Jessica Purkiss 15 Civilians Killed In Single ...
4	4	Iranian woman jailed for fictional unpublished...	Howard Portnoy	Print \nAn Iranian woman has been sentenced to...	1	Howard Portnoy Iranian woman jailed for fictio...

ExamplePython

# separating feature and target
X = news_data.drop(columns='label', axis =1)
Y = news_data['label']

ExamplePython

print(X)

Output

Output

          id  ...                                            content
0          0  ...  Darrell Lucus House Dem Aide: We Didn’t Even S...
1          1  ...  Daniel J. Flynn FLYNN: Hillary Clinton, Big Wo...
2          2  ...  Consortiumnews.com Why the Truth Might Get You...
3          3  ...  Jessica Purkiss 15 Civilians Killed In Single ...
4          4  ...  Howard Portnoy Iranian woman jailed for fictio...
...      ...  ...                                                ...
20795  20795  ...  Jerome Hudson Rapper T.I.: Trump a ’Poster Chi...
20796  20796  ...  Benjamin Hoffman N.F.L. Playoffs: Schedule, Ma...
20797  20797  ...  Michael J. de la Merced and Rachel Abrams Macy...
20798  20798  ...  Alex Ansary NATO, Russia To Hold Parallel Exer...
20799  20799  ...            David Swanson What Keeps the F-35 Alive

[20800 rows x 5 columns]

ExamplePython

print(Y)

Output

Output

0        1
1        0
2        1
3        1
4        1
        ..
20795    0
20796    0
20797    0
20798    1
20799    1
Name: label, Length: 20800, dtype: int64

Stemming:

Stemming is the process of reducing a word to its Root Word

ExamplePython

port_stem = PorterStemmer()

ExamplePython

def stemming(content):
  stemmed_content = re.sub('[^a-zA-Z]',' ',content)
  stemmed_content = stemmed_content.lower()
  stemmed_content = stemmed_content.split()
  stemmed_content = [port_stem.stem(word) for word in stemmed_content if not word in stopwords.words('english')]
  stemmed_content = ' '.join(stemmed_content)
  return stemmed_content

ExamplePython

news_data['content'] = news_data['content'].apply(stemming)

ExamplePython

print(news_data['content'])

Output

Output

0        darrel lucu hous dem aid even see comey letter...
1        daniel j flynn flynn hillari clinton big woman...
2                   consortiumnew com truth might get fire
3        jessica purkiss civilian kill singl us airstri...
4        howard portnoy iranian woman jail fiction unpu...
                               ...                        
20795    jerom hudson rapper trump poster child white s...
20796    benjamin hoffman n f l playoff schedul matchup...
20797    michael j de la merc rachel abram maci said re...
20798    alex ansari nato russia hold parallel exercis ...
20799                            david swanson keep f aliv
Name: content, Length: 20800, dtype: object

ExamplePython

X = news_data['content'].values
Y = news_data['label'].values

ExamplePython

print(X)

Output

Output

['darrel lucu hous dem aid even see comey letter jason chaffetz tweet'
 'daniel j flynn flynn hillari clinton big woman campu breitbart'
 'consortiumnew com truth might get fire' ...
 'michael j de la merc rachel abram maci said receiv takeov approach hudson bay new york time'
 'alex ansari nato russia hold parallel exercis balkan'
 'david swanson keep f aliv']

ExamplePython

print(Y)

Output

Output

[1 0 1 ... 0 1 1]

ExamplePython

Y.shape

Output

Output

(20800,)

ExamplePython

# converting the textual data to feature vectors
vectorizer = TfidfVectorizer()
vectorizer.fit(X)

X = vectorizer.transform(X)

ExamplePython

print(X)

Output

Output

  (0, 15686)	0.28485063562728646
  (0, 13473)	0.2565896679337957
  (0, 8909)	0.3635963806326075
  (0, 8630)	0.29212514087043684
  (0, 7692)	0.24785219520671603
  (0, 7005)	0.21874169089359144
  (0, 4973)	0.233316966909351
  (0, 3792)	0.2705332480845492
  (0, 3600)	0.3598939188262559
  (0, 2959)	0.2468450128533713
  (0, 2483)	0.3676519686797209
  (0, 267)	0.27010124977708766
  (1, 16799)	0.30071745655510157
  (1, 6816)	0.1904660198296849
  (1, 5503)	0.7143299355715573
  (1, 3568)	0.26373768806048464
  (1, 2813)	0.19094574062359204
  (1, 2223)	0.3827320386859759
… (33 more lines)