Radial Basis Function Kernel - Machine Learning
Last Updated : 06 May, 2024
Kernels play a fundamental role in transforming data into higher-dimensional spaces, enabling algorithms to learn complex patterns and relationships. Among the diverse kernel functions, the Radial Basis Function (RBF) kernel stands out as a versatile and powerful tool. In this article, we delve into the intricacies of the RBF kernel, exploring its mathematical formulation, intuitive understanding, practical applications, and its significance in various machine learning algorithms.
What is Kernel Function?
Kernel Function is used to transform n-dimensional input to m-dimensional input, where m is much higher than n then find the dot product in higher dimensional efficiently. The main idea to use kernel is: A linear classifier or regression curve in higher dimensions becomes a Non-linear classifier or regression curve in lower dimensions.
Radial Basis Function Kernel
The Radial Basis Function (RBF) kernel, also known as the Gaussian kernel, is one of the most widely used kernel functions. It operates by measuring the similarity between data points based on their Euclidean distance in the input space. Mathematically, the RBF kernel between two data points, \mathbf{x} and \mathbf{x'}, is defined as:
K(\mathbf{x}, \mathbf{x'}) = \exp\left(-\frac{|\mathbf{x} - \mathbf{x'}|^2}{2\sigma^2}\right)
where,
- |\mathbf{x} - \mathbf{x'}|^2 represents the squared Euclidean distance between the two data points.
- \sigma is a parameter known as the bandwidth or width of the kernel, controlling the smoothness of the decision boundary.
If we expand the above exponential expression, It will go upto infinite power of x and x’, as expansion of e^x contains infinite terms upto infinite power of x hence it involves terms upto infinite powers in infinite dimension.
If we apply any of the algorithms like perceptron Algorithm or linear regression on RBF kernel, actually we would be applying our algorithm to new infinite-dimensional data point we have created. Hence it will give a hyperplane in infinite dimensions, which will give a very strong non-linear classifier or regression curve after returning to our original dimensions.
a_1 x^{\infty} +a_2 x^{\infty-1}+a_3 x^{\infty-2} +\cdots +a_n x +c
So, Although we are applying linear classifier/regression it will give a non-linear classifier or regression line, that will be a polynomial of infinite power. And being a polynomial of infinite power, Radial Basis kernel is a very powerful kernel, which can give a curve fitting any complex dataset.
Why Radial Basis Kernel Is much powerful?
The main motive of the kernel is to do calculations in any d-dimensional space where d > 1, so that we can get a quadratic, cubic or any polynomial equation of large degree for our classification/regression line. Since Radial basis kernel uses exponent and as we know the expansion of e^x gives a polynomial equation of infinite power, so using this kernel, we make our regression/classification line infinitely powerful too.
Some Complex Dataset Fitted Using RBF Kernel easily:
The RBF kernel computes a similarity score between data points based on their distance in the input space. It assigns high similarity values to points that are close to each other and lower values to points that are farther apart. The parameter \sigma determines the scale of the distances over which points are considered similar.
Visually, the RBF kernel creates a "bump" or "hill" around each data point, with the height of the bump decaying exponentially as the distance from the point increases. This behavior captures the local structure of the data, making the RBF kernel particularly effective in capturing nonlinear relationships.
Radial Basis Function Neural Network for XOR Classification
- RBFNN Class:
- The RBFNN class initializes with a parameter sigma, representing the width of the Gaussian radial basis function.
- It contains methods to calculate Gaussian activation functions and to fit the model to data.
- The fit method trains the RBFNN model by computing activations for input data points and solving for the weights using the Moore-Penrose pseudo-inverse.
- The predict method predicts the output for new input data points using the trained model.
- Example Usage:
- The XOR dataset (X) consists of four data points, each with two features.
- Corresponding labels (y) represent the XOR function output for each data point.
- An RBFNN instance is created with a specified sigma value.
- The model is trained using the fit method on the XOR dataset.
- Predictions are obtained for the same dataset using the predict method.
- The mean squared error (MSE) between the predicted and actual outputs is calculated.
- Finally, the results are plotted, showing the predicted outputs colored based on their values, providing a visualization of the RBFNN's predictions for the XOR dataset.
Python import numpy as np import matplotlib.pyplot as plt class RBFNN: def __init__(self, sigma): self.sigma = sigma self.centers = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) self.weights = None def _gaussian(self, x, c): return np.exp(-np.linalg.norm(x - c) ** 2 / (2 * self.sigma ** 2)) def _calculate_activation(self, X): activations = np.zeros((X.shape[0], self.centers.shape[0])) for i, center in enumerate(self.centers): for j, x in enumerate(X): activations[j, i] = self._gaussian(x, center) return activations def fit(self, X, y): # Calculate activations activations = self._calculate_activation(X) # Initialize and solve for weights self.weights = np.linalg.pinv(activations.T @ activations) @ activations.T @ y def predict(self, X): if self.weights is None: raise ValueError("Model not trained yet. Call fit method first.") activations = self._calculate_activation(X) return activations @ self.weights # Example usage: if __name__ == "__main__": # Define XOR dataset X = np.array([[0.1, 0.1], [0.1, 0.9], [0.9, 0.1], [0.9, 0.9]]) y = np.array([0, 1, 1, 0]) # Initialize and train RBFNN rbfnn = RBFNN(sigma=0.1) rbfnn.fit(X, y) # Predict predictions = rbfnn.predict(X) print("Predictions:", predictions) # Calculate mean squared error mse = np.mean((predictions - y) ** 2) print("Mean Squared Error:", mse) # Plot the results plt.scatter(X[:, 0], X[:, 1], c=predictions, cmap='viridis') plt.colorbar(label='Predicted Output') plt.xlabel('X1') plt.ylabel('X2') plt.title('RBFNN Predictions for XOR ') plt.show()
Output:
RBF Applied on XOR OperationPractical Applications of Radial Basis Function Kernel
The versatility and effectiveness of the RBF kernel make it suitable for various machine learning tasks, including:
- Support Vector Machines (SVMs): In SVMs, the RBF kernel is commonly used to map data points into a higher-dimensional space where a linear decision boundary can be constructed to separate classes.
- Kernelized Ridge Regression: In regression tasks, the RBF kernel can be used to perform kernelized ridge regression, allowing the model to capture nonlinear relationships between features and target variables.
- Clustering: The RBF kernel can also be employed in kernelized clustering algorithms such as spectral clustering, where it helps in capturing the local structure of the data for grouping similar data points together.
- Dimensionality Reduction: In manifold learning and nonlinear dimensionality reduction techniques like t-Distributed Stochastic Neighbor Embedding (t-SNE), the RBF kernel is used to define the similarity between data points in the high-dimensional space.