Implementing Lasso Regression from Scratch
L1-penalty theory එක, මෙතන run වෙන code එකකට හරවනවා, feature weights ටිකක් හරියටම zero දක්වා shrink වෙනවා ඔයාට පේනවා — automatic feature selection එක වැඩ කරන ඇස් ඉදිරියේ පේන සාක්ෂියක්.
This turns the L1-penalty theory into running code, and you'll actually watch some feature weights shrink all the way to exactly zero — visible proof of automatic feature selection at work.
Lesson 42 of 64 · Part 3 · Building Models from Scratch · Algorithms from Scratch · bilingual notes (English + සිංහල)
Why this matters. This turns the L1-penalty theory into running code, and you'll actually watch some feature weights shrink all the way to exactly zero — visible proof of automatic feature selection at work.
සිංහලෙන්: L1-penalty theory එක, මෙතන run වෙන code එකකට හරවනවා, feature weights ටිකක් හරියටම zero දක්වා shrink වෙනවා ඔයාට පේනවා — automatic feature selection එක වැඩ කරන ඇස් ඉදිරියේ පේන සාක්ෂියක්.
What you'll use this for. This closes out the "algorithms from scratch" section — you now understand not just how to use four core ML algorithms, but why each one makes the decisions it does.
සිංහලෙන්: "Algorithms from scratch" section එක අවසන් වෙන්නෙ මෙතනින් — Core ML algorithms හතරක් පාවිච්චි කරන්නෙ කොහොමද කියලා විතරක් නෙවෙයි, එකිනෙක එහෙම decisions ගන්නෙ ඇයි කියලාත් දැන් ඔයාට තේරෙනවා.
Key principles
- A weight that becomes exactly zero means that feature was dropped, not just weakened. This is the key difference from other regularization methods (like Ridge) — Lasso can produce true zeros, effectively performing feature selection as a side effect of training.
හරියටම zero වුන weight එකක් කියන්නෙ, ඒ feature එක අඩු කලේ නෙවෙයි, සම්පූර්ණයෙන්ම drop කරපු එකක්
Walkthrough
Lasso Regression:
Y = wX + b
Y --> Dependent Variable
X --> Independent Variable
w --> weight
b --> bias
Gradient Descent:
Gradient Descent is an optimization algorithm used for minimizing the loss function in various machine learning algorithms. It is used for updating the parameters of the learning model.
w = w - α*dw
b = b - α*db
Learning Rate:
Learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.
Gradient for Weights
Gradient for Bias
Importing the Dependencies
import numpy as npLasso Regression
# creating a class for Lasso Regression
class Lasso_Regression():
#initiating the hyperparameters
def __init__(self, learning_rate, no_of_iterations, lambda_parameter):
self.learning_rate = learning_rate
self.no_of_iterations = no_of_iterations
self.lambda_parameter = lambda_parameter
# fitting the dataset to the Lasso Regression model
def fit(self, X, Y):
# m --> number of Data points --> number of rows
# n --> number of input features --> number of columns
self.m, self.n = X.shape
self.w = np.zeros(self.n)
self.b = 0
self.X = X
self.Y = Y
# implementing Gradient Descent algorithm for Optimization
for i in range(self.no_of_iterations): # missed "self"
self.upadte_weights()
# function for updating the weight & bias value
def upadte_weights(self):
# linear equation of the model
Y_prediction = self.predict(self.X)
# gradients (dw, db)
# gradient for weight
dw = np.zeros(self.n)
for i in range(self.n):
if self.w[i]>0:
dw[i] = (-(2*(self.X[:,i]).dot(self.Y - Y_prediction)) + self.lambda_parameter) / self.m
else :
dw[i] = (-(2*(self.X[:,i]).dot(self.Y - Y_prediction)) - self.lambda_parameter) / self.m
# gradient for bias
db = - 2 * np.sum(self.Y - Y_prediction) / self.m
# updating the weights & bias
self.w = self.w - self.learning_rate*dw
self.b = self.b - self.learning_rate*db
# Predicting the Target variable
def predict(self,X):
return X.dot(self.w) + self.bUsing Linear Regression model for Prediction
# importing the dependencies
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from sklearn import metricsData Pre-Processing
# loading the data from csv file to a pandas dataframe
salary_data = pd.read_csv('/content/salary_data.csv')# printing the first 5 columns of the dataframe
salary_data.head()| YearsExperience | Salary | |
|---|---|---|
| 0 | 1.1 | 39343 |
| 1 | 1.3 | 46205 |
| 2 | 1.5 | 37731 |
| 3 | 2.0 | 43525 |
| 4 | 2.2 | 39891 |
# last 5 rows of the dataframe
salary_data.tail()| YearsExperience | Salary | |
|---|---|---|
| 25 | 9.0 | 105582 |
| 26 | 9.5 | 116969 |
| 27 | 9.6 | 112635 |
| 28 | 10.3 | 122391 |
| 29 | 10.5 | 121872 |
# number of rows & columns in the dataframe
salary_data.shapeOutput
(30, 2)# checking for missing values
salary_data.isnull().sum()Output
YearsExperience 0
Salary 0
dtype: int64Splitting the feature & target
features = salary_data.iloc[:,:-1].values
target = salary_data.iloc[:,1].valuesprint(features)Output
[[ 1.1]
[ 1.3]
[ 1.5]
[ 2. ]
[ 2.2]
[ 2.9]
[ 3. ]
[ 3.2]
[ 3.2]
[ 3.7]
[ 3.9]
[ 4. ]
[ 4. ]
[ 4.1]
[ 4.5]
[ 4.9]
[ 5.1]
[ 5.3]
… (12 more lines)print(target)Output
[ 39343 46205 37731 43525 39891 56642 60150 54445 64445 57189
63218 55794 56957 57081 61111 67938 66029 83088 81363 93940
91738 98273 101302 113812 109431 105582 116969 112635 122391 121872]Splitting the dataset into training & test data
X_train, X_test, Y_train, Y_test = train_test_split(features, target,
test_size=0.33,
random_state = 2)Training the Linear Regression model
model = Lasso_Regression(learning_rate = 0.02, no_of_iterations=1000,
lambda_parameter=200)model.fit(X_train, Y_train)Predict the salary value for test data
test_data_prediction = model.predict(X_test)print(test_data_prediction)Output
[ 36068.90441536 34166.14804787 66513.00629512 58901.98082518
91248.83907243 80783.67905126 101713.9990936 52242.33353898
42728.55170156 88394.7045212 ]from sklearn import metrics
# R squared error
score_1 = metrics.r2_score(Y_test, test_data_prediction)
# Mean Absolute Error
score_2 = metrics.mean_absolute_error(Y_test, test_data_prediction)
print("R squared error : ", score_1)
print('Mean Absolute Error : ', score_2)Output
R squared error : 0.9041366551714738
Mean Absolute Error : 6675.382767802961Comparing the error with SKLearn Lasso Regression
from sklearn.linear_model import Lasso
sk_model = Lasso()sk_model.fit(X_train, Y_train)Output
Lasso()sk_test_data_prediction = sk_model.predict(X_test)# R squared error
score_1_ = metrics.r2_score(Y_test, sk_test_data_prediction)
# Mean Absolute Error
score_2_ = metrics.mean_absolute_error(Y_test, sk_test_data_prediction)
print("R squared error : ", score_1_)
print('Mean Absolute Error : ', score_2_)Output
R squared error : 0.9041817142395875
Mean Absolute Error : 6673.480440148538Keep going: ← Lesson 41 · Lesson 43 → · Track it in the Self Guide
Finished this one?
Part of Self Guide · Stage 15: ML 3 — Building Models from Scratch. Progress is saved in this browser.
