Support Vector Machines

Understanding SVMs and their implementation for classification and regression

What are Support Vector Machines?

A powerful supervised learning algorithm for classification and regression

Support Vector Machines (SVMs) are a set of supervised learning methods used for classification, regression, and outlier detection. The objective of an SVM is to find a hyperplane in an N-dimensional space that distinctly classifies the data points.

Key Concepts in SVMs

Hyperplane: A decision boundary that separates different classes
Support Vectors: Data points closest to the hyperplane that influence its position and orientation
Margin: The distance between the hyperplane and the closest data points (support vectors)
Kernel Trick: A method to transform the input space to a higher-dimensional space where a linear separator might exist

How SVMs Work

SVMs work by finding the hyperplane that maximizes the margin between classes. The algorithm follows these steps:

Map data to a high-dimensional feature space (implicitly using kernels)
Find the optimal hyperplane that maximizes the margin between classes
Identify support vectors (points that lie closest to the hyperplane)
Use the support vectors to define the decision boundary

SVM Kernels

Kernels allow SVMs to handle non-linearly separable data by transforming it into a higher-dimensional space:

Linear Kernel: K(x, y) = x · y (dot product)
Polynomial Kernel: K(x, y) = (γx · y + r)^d
Radial Basis Function (RBF) Kernel: K(x, y) = exp(-γ||x - y||²)
Sigmoid Kernel: K(x, y) = tanh(γx · y + r)

SVM Parameters and Tuning

The performance of an SVM model depends on several key parameters:

C Parameter (Regularization)

Controls the trade-off between having a smooth decision boundary and classifying training points correctly. A small C makes the decision surface smooth but may lead to training errors. A large C aims to classify all training examples correctly but may lead to overfitting.

Gamma Parameter

Defines how far the influence of a single training example reaches. Low gamma means a point has a far reach, while high gamma means the reach is limited to close points. High gamma can lead to overfitting.

Kernel Selection

The choice of kernel depends on the data. Linear kernels work well for linearly separable data, while RBF kernels are versatile for non-linear data. Polynomial kernels can capture more complex relationships.