Building Support Vector Machine Classifier from Scratch in Python

SVM එකෙන් අහන්නෙ "මට classes වෙන් කරගන්න පුළුවන්ද?" කියන එකට වඩා smart questionයක් — "මට වෙන් කරගන්න පුළුවන් පළලම margin එක මොකද්ද?" කියලා අහනවා, ඒක සාමාන්‍යයෙන් අලුත් data එකට වඩා හොඳින් generalize වෙනවා.

SVM asks a smarter question than "can I separate the classes?" — it asks "what's the widest possible margin I can separate them by?", which usually generalizes better to new data.

13 May 20263 min readSelf Guide · Stage 15#Building Models from Scratch#Algorithms from Scratch#Machine learning

Lesson 38 of 64 · Part 3 · Building Models from Scratch · Algorithms from Scratch · bilingual notes (English + සිංහල)

Why this matters. SVM asks a smarter question than "can I separate the classes?" — it asks "what's the widest possible margin I can separate them by?", which usually generalizes better to new data.

සිංහලෙන්: SVM එකෙන් අහන්නෙ "මට classes වෙන් කරගන්න පුළුවන්ද?" කියන එකට වඩා smart questionයක් — "මට වෙන් කරගන්න පුළුවන් පළලම margin එක මොකද්ද?" කියලා අහනවා, ඒක සාමාන්‍යයෙන් අලුත් data එකට වඩා හොඳින් generalize වෙනවා.

What you'll use this for. SVMs are especially strong on datasets with clear margins between classes and relatively few features — a common shape in medical and text datasets alike.

සිංහලෙන්: Classes අතර පැහැදිලි margins තියෙන, features සාපේක්ෂව අඩුවෙන් තියෙන datasets වලට SVM ගොඩක් ශක්තිමත් — medical සහ text datasets දෙකෙන්ම common පේන shape එකක් තමයි මේක.

Key principles

Only the closest points (support vectors) actually decide the boundary. Points far from the boundary could be deleted entirely and the decision line wouldn't move at all — which is where the algorithm's name comes from.
ළඟම points (support vectors) විතරයි boundary එක ඇත්තටම තීරණය කරන්නෙ

Walkthrough

SVM Classifier

Equation of the Hyperplane:

y = wx - b

Gradient Descent:

Gradient Descent is an optimization algorithm used for minimizing the loss function in various machine learning algorithms. It is used for updating the parameters of the learning model.

w = w - α*dw

b = b - α*db

Learning Rate:

Learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.

Importing the Dependencies

ExamplePython

# importing numpy library
import numpy as np

Support Vector Machine Classifier

ExamplePython

class SVM_classifier():

  # initiating the hyperparameters
  def __init__(self, learning_rate, no_of_iterations, lambda_parameter):

    self.learning_rate = learning_rate
    self.no_of_iterations = no_of_iterations
    self.lambda_parameter = lambda_parameter

  
  # fitting the dataset to SVM Classifier
  def fit(self, X, Y):

    # m  --> number of Data points --> number of rows
    # n  --> number of input features --> number of columns
    self.m, self.n = X.shape

    # initiating the weight value and bias value

    self.w = np.zeros(self.n)

    self.b = 0

    self.X = X

    self.Y = Y

    # implementing Gradient Descent algorithm for Optimization

    for i in range(self.no_of_iterations):
      self.update_weights()

  # function for updating the weight and bias value
  def update_weights(self):

    # label encoding
    y_label = np.where(self.Y <= 0, -1, 1)

    # gradients ( dw, db)
    for index, x_i in enumerate(self.X):

      condition = y_label[index] * (np.dot(x_i, self.w) - self.b) >= 1

      if (condition == True):

        dw = 2 * self.lambda_parameter * self.w
        db = 0

      else:

        dw = 2 * self.lambda_parameter * self.w - np.dot(x_i, y_label[index])
        db = y_label[index]

      self.w = self.w - self.learning_rate * dw

      self.b = self.b - self.learning_rate * db

  # predict the label for a given input value
  def predict(self, X):

    output = np.dot(X, self.w) - self.b
    
    predicted_labels = np.sign(output)

    y_hat = np.where(predicted_labels <= -1, 0, 1)

    return y_hat