Feature extraction of Text data using Tf idf Vectorizer

සරල word counts වලදී "the" කියන එකයි "diagnosis" කියන එකයි එකම වැදගත්කමින් treat වෙනවා. TF-IDF තමයි fix එක — common words ගණිතමය ව අඩු weight එකක් දෙනවා, document එකක් අනිත් එකෙන් වෙන් කරන වචන වලට වැඩි weight එකක් දෙනවා.

Simple word counts treat "the" and "diagnosis" as equally important. TF-IDF is the fix — it mathematically down-weights common words and up-weights the ones that actually distinguish one document from another.

7 May 20265 min readSelf Guide · Stage 13#Python & Data Basics#Data Pre-processing#Machine learning

Lesson 19 of 64 · Part 1 · Python & Data Basics · Data Pre-processing · bilingual notes (English + සිංහල)

Why this matters. Simple word counts treat "the" and "diagnosis" as equally important. TF-IDF is the fix — it mathematically down-weights common words and up-weights the ones that actually distinguish one document from another.

සිංහලෙන්: සරල word counts වලදී "the" කියන එකයි "diagnosis" කියන එකයි එකම වැදගත්කමින් treat වෙනවා. TF-IDF තමයි fix එක — common words ගණිතමය ව අඩු weight එකක් දෙනවා, document එකක් අනිත් එකෙන් වෙන් කරන වචන වලට වැඩි weight එකක් දෙනවා.

What you'll use this for. This is the exact technique used to turn spam-mail text into features for the Spam Mail Prediction project coming up.

සිංහලෙන්: ඊළඟට එන Spam Mail Prediction project එකේදී, spam mail text එක features බවට හරවන්නෙ හරියටම මේ technique එකෙන්.

Key principles

Rare-but-repeated words are the real signal. A word appearing in every single email tells you nothing useful for classification. A word appearing often in only spam emails tells you a lot — TF-IDF captures exactly that difference.
දුර්ලභ ඒත් නැවත නැවත එන වචන තමයි ඇත්තම signal එක

Walkthrough

About the Dataset:

id: unique id for a news article
title: the title of a news article
author: author of the news article
text: the text of the article; could be incomplete
label: a label that marks whether the news article is real or fake: 1: Fake news 0: real News

Importing the Dependencies

ExamplePython

import numpy as np
import pandas as pd
import re
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer

ExamplePython

import nltk
nltk.download('stopwords')

Output

Output

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.

Output

Output

True

ExamplePython

# printing the stopwords in English
print(stopwords.words('english'))

Output

Output

['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'hi

Data Pre-processing

ExamplePython

# loading the dataset to a pandas DataFrame
news_dataset = pd.read_csv('/content/train.csv')

ExamplePython

news_dataset.shape

Output

Output

(20800, 5)

ExamplePython

# print the first 5 rows of the dataframe
news_dataset.head()

	id	title	author	text	label
0	0	House Dem Aide: We Didn’t Even See Comey’s Let...	Darrell Lucus	House Dem Aide: We Didn’t Even See Comey’s Let...	1
1	1	FLYNN: Hillary Clinton, Big Woman on Campus - ...	Daniel J. Flynn	Ever get the feeling your life circles the rou...	0
2	2	Why the Truth Might Get You Fired	Consortiumnews.com	Why the Truth Might Get You Fired October 29, ...	1
3	3	15 Civilians Killed In Single US Airstrike Hav...	Jessica Purkiss	Videos 15 Civilians Killed In Single US Airstr...	1
4	4	Iranian woman jailed for fictional unpublished...	Howard Portnoy	Print \nAn Iranian woman has been sentenced to...	1

ExamplePython

# counting the number of missing values in the dataset
news_dataset.isnull().sum()

Output

Output

id           0
title      558
author    1957
text        39
label        0
dtype: int64

ExamplePython

# replacing the null values with empty string
news_dataset = news_dataset.fillna('')

ExamplePython

# merging the author name and news title
news_dataset['content'] = news_dataset['author']+' '+news_dataset['title']

ExamplePython

print(news_dataset['content'])

Output

Output

0        Darrell Lucus House Dem Aide: We Didn’t Even S...
1        Daniel J. Flynn FLYNN: Hillary Clinton, Big Wo...
2        Consortiumnews.com Why the Truth Might Get You...
3        Jessica Purkiss 15 Civilians Killed In Single ...
4        Howard Portnoy Iranian woman jailed for fictio...
                               ...                        
20795    Jerome Hudson Rapper T.I.: Trump a ’Poster Chi...
20796    Benjamin Hoffman N.F.L. Playoffs: Schedule, Ma...
20797    Michael J. de la Merced and Rachel Abrams Macy...
20798    Alex Ansary NATO, Russia To Hold Parallel Exer...
20799              David Swanson What Keeps the F-35 Alive
Name: content, Length: 20800, dtype: object

ExamplePython

# separating the data & label
X = news_dataset.drop(columns='label', axis=1)
Y = news_dataset['label']

ExamplePython

print(X)
print(Y)

Output

Output

          id  ...                                            content
0          0  ...  Darrell Lucus House Dem Aide: We Didn’t Even S...
1          1  ...  Daniel J. Flynn FLYNN: Hillary Clinton, Big Wo...
2          2  ...  Consortiumnews.com Why the Truth Might Get You...
3          3  ...  Jessica Purkiss 15 Civilians Killed In Single ...
4          4  ...  Howard Portnoy Iranian woman jailed for fictio...
...      ...  ...                                                ...
20795  20795  ...  Jerome Hudson Rapper T.I.: Trump a ’Poster Chi...
20796  20796  ...  Benjamin Hoffman N.F.L. Playoffs: Schedule, Ma...
20797  20797  ...  Michael J. de la Merced and Rachel Abrams Macy...
20798  20798  ...  Alex Ansary NATO, Russia To Hold Parallel Exer...
20799  20799  ...            David Swanson What Keeps the F-35 Alive

[20800 rows x 5 columns]
0        1
1        0
2        1
3        1
… (8 more lines)

Stemming:

Stemming is the process of reducing a word to its Root word

example: actor, actress, acting --> act

ExamplePython

port_stem = PorterStemmer()

ExamplePython

def stemming(content):
    stemmed_content = re.sub('[^a-zA-Z]',' ',content)
    stemmed_content = stemmed_content.lower()
    stemmed_content = stemmed_content.split()
    stemmed_content = [port_stem.stem(word) for word in stemmed_content 
                       if not word in stopwords.words('english')]
    stemmed_content = ' '.join(stemmed_content)
    return stemmed_content

ExamplePython

news_dataset['content'] = news_dataset['content'].apply(stemming)

ExamplePython

print(news_dataset['content'])

Output

Output

0        darrel lucu hous dem aid even see comey letter...
1        daniel j flynn flynn hillari clinton big woman...
2                   consortiumnew com truth might get fire
3        jessica purkiss civilian kill singl us airstri...
4        howard portnoy iranian woman jail fiction unpu...
                               ...                        
20795    jerom hudson rapper trump poster child white s...
20796    benjamin hoffman n f l playoff schedul matchup...
20797    michael j de la merc rachel abram maci said re...
20798    alex ansari nato russia hold parallel exercis ...
20799                            david swanson keep f aliv
Name: content, Length: 20800, dtype: object

ExamplePython

#separating the data and label
X = news_dataset['content'].values
Y = news_dataset['label'].values

ExamplePython

print(X)

Output

Output

['darrel lucu hous dem aid even see comey letter jason chaffetz tweet'
 'daniel j flynn flynn hillari clinton big woman campu breitbart'
 'consortiumnew com truth might get fire' ...
 'michael j de la merc rachel abram maci said receiv takeov approach hudson bay new york time'
 'alex ansari nato russia hold parallel exercis balkan'
 'david swanson keep f aliv']

ExamplePython

print(Y)

Output

Output

[1 0 1 ... 0 1 1]

ExamplePython

Y.shape

Output

Output

(20800,)

Tf-Idf

ExamplePython

# convert the textual data to Feature Vectors
vectorizer = TfidfVectorizer()

ExamplePython

vectorizer.fit(X)

X = vectorizer.transform(X)

ExamplePython

print(X)

Output

Output

  (0, 15686)	0.28485063562728646
  (0, 13473)	0.2565896679337957
  (0, 8909)	0.3635963806326075
  (0, 8630)	0.29212514087043684
  (0, 7692)	0.24785219520671603
  (0, 7005)	0.21874169089359144
  (0, 4973)	0.233316966909351
  (0, 3792)	0.2705332480845492
  (0, 3600)	0.3598939188262559
  (0, 2959)	0.2468450128533713
  (0, 2483)	0.3676519686797209
  (0, 267)	0.27010124977708766
  (1, 16799)	0.30071745655510157
  (1, 6816)	0.1904660198296849
  (1, 5503)	0.7143299355715573
  (1, 3568)	0.26373768806048464
  (1, 2813)	0.19094574062359204
  (1, 2223)	0.3827320386859759
… (33 more lines)