Machine Learning Algorithms
Last Updated : 21 Apr, 2025
Machine learning algorithms are essentially sets of instructions that allow computers to learn from data, make predictions, and improve their performance over time without being explicitly programmed. Machine learning algorithms are broadly categorized into three types:
- Supervised Learning: Algorithms learn from labeled data, where the input-output relationship is known.
- Unsupervised Learning: Algorithms work with unlabeled data to identify patterns or groupings.
- Reinforcement Learning: Algorithms learn by interacting with an environment and receiving feedback in the form of rewards or penalties.
.webp)
Supervised Learning Algorithms
Supervised learning algos are trained on datasets where each example is paired with a target or response variable, known as the label. The goal is to learn a mapping function from input data to the corresponding output labels, enabling the model to make accurate predictions on unseen data. Supervised learning problems are generally categorized into two main types: Classification and Regression. Most widely used supervised learning algorithms are:
1. Linear Regression
Linear regression is used to predict a continuous value by finding the best-fit straight line between input (independent variable) and output (dependent variable)
- Minimizes the difference between actual values and predicted values using a method called "least squares" to to best fit the data.
- Predicting a person’s weight based on their height or predicting house prices based on size.
2. Logistic Regression
Logistic regression predicts probabilities and assigns data points to binary classes (e.g., spam or not spam).
- It uses a logistic function (S-shaped curve) to model the relationship between input features and class probabilities.
- Used for classification tasks (binary or multi-class).
- Outputs probabilities to classify data into categories.
- Example : Predicting whether a customer will buy a product online (yes/no) or diagnosing if a person has a disease (sick/not sick).
Note : Despite its name, logistic regression is used for classification tasks, not regression.
3. Decision Trees
A decision tree splits data into branches based on feature values, creating a tree-like structure.
- Each decision node represents a feature; leaf nodes provide the final prediction.
- The process continues until a final prediction is made at the leaf nodes
- Works for both classification and regression tasks.
For more decision tree algorithms, you can explore:
4. Support Vector Machines (SVM)
SVMs find the best boundary (called a hyperplane) that separates data points into different classes.
- Uses support vectors (critical data points) to define the hyperplane.
- Can handle linear and non-linear problems using kernel functions.
- focuses on maximizing the margin between classes, making it robust for high-dimensional data or complex patterns.
5. k-Nearest Neighbors (k-NN)
KNN is a simple algorithm that predicts the output for a new data point based on the similarity (distance) to its nearest neighbors in the training dataset, used for both classification and regression tasks.
- Calculates distance between point with existing data points in training dataset using a distance metric (e.g., Euclidean, Manhattan, Minkowski)
- identifies k nearest neighbors to new data point based on the calculated distances.
- For classification, algorithm assigns class label that is most common among its k nearest neighbors.
- For regression, the algorithm predicts the value as the average of the values of its k nearest neighbors.
6. Naive Bayes
Based on Bayes' theorem and assumes all features are independent of each other (hence "naive")
- Calculates probabilities for each class and assigns the most likely class to a data point.
- Assumption of feature independence might not hold in all cases ( rarely true in real-world data )
- Works well for high-dimensional data.
- Commonly used in text classification tasks like spam filtering : Naive Bayes
7. Random Forest
Random forest is an ensemble method that combines multiple decision trees.
- Uses random sampling and feature selection for diversity among trees.
- Final prediction is based on majority voting (classification) or averaging (regression).
- Advantages : reduces overfitting compared to individual decision trees.
- Handles large datasets with higher dimensionality.
For in-depth understanding : What is Ensemble Learning? - Two types of ensemble methods in ML
7. Gradient Boosting (e.g., XGBoost, LightGBM, CatBoost)
These algorithms build models sequentially, meaning each new model corrects errors made by previous ones. Combines weak learners (like decision trees) to create a strong predictive model. Effective for both regression and classification tasks. : Gradient Boosting in ML
- XGBoost (Extreme Gradient Boosting) : Advanced version of Gradient Boosting that includes regularization to prevent overfitting. Faster than traditional Gradient Boosting, for large datasets.
- LightGBM (Light Gradient Boosting Machine): Uses a histogram-based approach for faster computation and supports categorical features natively.
- CatBoost: Designed specifically for categorical data, with built-in encoding techniques. Uses symmetric trees for faster training and better generalization.
For more ensemble learning and gradient boosting approaches, explore:
8. Neural Networks ( Including Multilayer Perceptron)
Neural Networks, including Multilayer Perceptrons (MLPs), are considered part of supervised machine learning algorithms as they require labeled data to train and learn the relationship between input and desired output; network learns to minimize the error using backpropagation algorithm to adjust weights during training.
- Multilayer Perceptron (MLP): Neural network with multiple layers of nodes.
- Used for both classification and regression ( Examples: image classification, spam detection, and predicting numerical values like stock prices or house prices)
For in-depth understanding : Supervised multi-layer perceptron model - What is perceptron?
Unsupervised Learning Algorithms
Unsupervised learning algos works with unlabeled data to discover hidden patterns or structures without predefined outputs. These are again divided into three main categories based on their purpose: Clustering, Association Rule Mining, and Dimensionality Reduction. First we'll see algorithms for Clustering, then dimensionality reduction and at last association.
1. Clustering
Clustering algorithms group data points into clusters based on their similarities or differences. The goal is to identify natural groupings in the data. Clustering algorithms are divided into multiple types based on the methods they use to group data. These types include Centroid-based methods, Distribution-based methods, Connectivity-based methods, and Density-based methods. For resources and in-depth understanding, go through the links below.
- Centroid-based Methods: Represent clusters using central points, such as centroids or medoids.
- Distribution-based Methods
- Connectivity based methods
- Density Based methods
2. Dimensionality Reduction
Dimensionality reduction is used to simplify datasets by reducing the number of features while retaining the most important information.
3. Association Rule
Find patterns (called association rules) between items in large datasets, typically in market basket analysis (e.g., finding that people who buy bread often buy butter). It identifies patterns based solely on the frequency of item occurrences and co-occurrences in the dataset.
Reinforcement Learning Algorithms
Reinforcement learning involves training agents to make a sequence of decisions by rewarding them for good actions and penalizing them for bad ones. Broadly categorized into Model-Based and Model-Free methods, these approaches differ in how they interact with the environment.
1. Model-Based Methods
These methods use a model of the environment to predict outcomes and help the agent plan actions by simulating potential results.
2. Model-Free Methods
These methods do not build or rely on an explicit model of the environment. Instead, the agent learns directly from experience by interacting with the environment and adjusting its actions based on feedback. Model-Free methods can be further divided into Value-Based and Policy-Based methods:
- Value-Based Methods: Focus on learning the value of different states or actions, where the agent estimates the expected return from each action and selects the one with the highest value.
- Policy-based Methods: Directly learn a policy (a mapping from states to actions) without estimating values where the agent continuously adjusts its policy to maximize rewards.
Discover the Top 15 Machine Learning Algorithms for Interview Preparation.
Similar Reads
Machine Learning Algorithms
Machine learning algorithms are essentially sets of instructions that allow computers to learn from data, make predictions, and improve their performance over time without being explicitly programmed. Machine learning algorithms are broadly categorized into three types: Supervised Learning: Algorith
8 min read
Top 15 Machine Learning Algorithms Every Data Scientist Should Know in 2025
Machine Learning (ML) Algorithms are the backbone of everything from Netflix recommendations to fraud detection in financial institutions. These algorithms form the core of intelligent systems, empowering organizations to analyze patterns, predict outcomes, and automate decision-making processes. Wi
15 min read
Linear Model Regression
Ordinary Least Squares (OLS) using statsmodels
Ordinary Least Squares (OLS) is a widely used statistical method for estimating the parameters of a linear regression model. It minimizes the sum of squared residuals between observed and predicted values. In this article we will learn how to implement Ordinary Least Squares (OLS) regression using P
3 min read
Linear Regression (Python Implementation)
Linear regression is a statistical method that is used to predict a continuous dependent variable i.e target variable based on one or more independent variables. This technique assumes a linear relationship between the dependent and independent variables which means the dependent variable changes pr
14 min read
ML | Multiple Linear Regression using Python
Linear regression is a fundamental statistical method widely used for predictive analysis. It models the relationship between a dependent variable and a single independent variable by fitting a linear equation to the data. Multiple Linear Regression is an extension of this concept that allows us to
4 min read
Polynomial Regression ( From Scratch using Python )
Prerequisites Linear RegressionGradient DescentIntroductionLinear Regression finds the correlation between the dependent variable ( or target variable ) and independent variables ( or features ). In short, it is a linear model to fit the data linearly. But it fails to fit and catch the pattern in no
5 min read
Bayesian Linear Regression
Linear regression is based on the assumption that the underlying data is normally distributed and that all relevant predictor variables have a linear relationship with the outcome. But In the real world, this is not always possible, it will follows these assumptions, Bayesian regression could be the
11 min read
How to Perform Quantile Regression in Python
In this article, we are going to see how to perform quantile regression in Python. Linear regression is defined as the statistical method that constructs a relationship between a dependent variable and an independent variable as per the given set of variables. While performing linear regression we a
4 min read
Isotonic Regression in Scikit Learn
Isotonic regression is a regression technique in which the predictor variable is monotonically related to the target variable. This means that as the value of the predictor variable increases, the value of the target variable either increases or decreases in a consistent, non-oscillating manner. Mat
6 min read
Stepwise Regression in Python
Stepwise regression is a method of fitting a regression model by iteratively adding or removing variables. It is used to build a model that is accurate and parsimonious, meaning that it has the smallest number of variables that can explain the data. There are two main types of stepwise regression: F
6 min read
Least Angle Regression (LARS)
Regression is a supervised machine learning task that can predict continuous values (real numbers), as compared to classification, that can predict categorical or discrete values. Before we begin, if you are a beginner, I highly recommend this article. Least Angle Regression (LARS) is an algorithm u
3 min read
Linear Model Classification
K-Nearest Neighbors (KNN)
ML | Stochastic Gradient Descent (SGD)
Stochastic Gradient Descent (SGD) is an optimization algorithm in machine learning, particularly when dealing with large datasets. It is a variant of the traditional gradient descent algorithm but offers several advantages in terms of efficiency and scalability, making it the go-to method for many d
8 min read