Gradient Descent in Linear Regression Last Updated : 27 May, 2025 Comments Improve Suggest changes Like Article Like Report Gradient descent is a optimization algorithm used in linear regression to find the best fit line to the data. It works by gradually by adjusting the line’s slope and intercept to reduce the difference between actual and predicted values. This process helps the model make accurate predictions by minimizing errors step by step. In this article we will see more about Gradient Descent and its core concepts in detail.Gradient Descent in Linear RegressionAbove image shows two graphs, left one plots house prices against size to show errors measured by the cost function while right one shows how gradient descent moves downhill on the cost curve to minimize error by updating parameters step by step.Why Use Gradient Descent for Linear Regression?Linear regression finds the best-fit line for a dataset by minimizing the error between the actual and predicted values. This error is measured using the cost function usually Mean Squared Error (MSE). The goal is to find the model parameters i.e. the slope m and the intercept b that minimize this cost function.For simple linear regression, we can use formulas like Normal Equation to find parameters directly. However for large datasets or high-dimensional data these methods become computationally expensive due to:Large matrix computations.Memory limitations.In models like polynomial regression, the cost function becomes highly complex and non-linear, so analytical solutions are not available. That’s where gradient descent plays an important role even for:Large datasets.Complex, high-dimensional problems.How Does Gradient Descent Work in Linear Regression?Lets see various steps involved in the working of Gradient Descent in Linear Regression:1. Initializing Parameters: Start with random initial values for the slope (m) and intercept (b).2. Calculate the Cost Function: Measure the error using the Mean Squared Error (MSE): J(m, b) = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - (mx_i + b) \right)^23. Compute the Gradient: Calculate how much the cost function changes with respect to m and b. For slope m : \frac{\partial J}{\partial m} = -\frac{2}{n} \sum_{i=1}^{n} x_i (y_i - (mx_i + b))For intercept b: \frac{\partial J}{\partial b} = -\frac{2}{n} \sum_{i=1}^{n} (y_i - (mx_i + b))4. Update Parameters: Change m and b to reduce the error:For slope m : m = m - \alpha \cdot \frac{\partial J}{\partial m}For intercept b :b = b - \alpha \cdot \frac{\partial J}{\partial b}Here \alpha is the learning rate that controls the size of each update.5. Repeat: Keep repeating steps 2–4 until the error stops decreasing significantly.Implementation of Gradient Descent in Linear RegressionLet’s implement linear regression step by step. To understand how gradient descent improves the model, we will first build a simple linear regression without using gradient descent and observe its results.Here we will be using Numpy, Pandas, Matplotlib and Sckit learn libraries for this.X, y = make_regression(n_samples=100, n_features=1, noise=15, random_state=42): Generating 100 data points with one feature and some noise for realism.X_b = np.c_[np.ones((m, 1)), X]: Addind a column of ones to X to account for the intercept term in the model.theta = np.array([[2.0], [3.0]]): Initializing model parameters (intercept and slope) with starting values. Python import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import make_regression X, y = make_regression(n_samples=100, n_features=1, noise=15, random_state=42) y = y.reshape(-1, 1) m = X.shape[0] X_b = np.c_[np.ones((m, 1)), X] theta = np.array([[2.0], [3.0]]) plt.figure(figsize=(10, 5)) plt.scatter(X, y, color="blue", label="Actual Data") plt.plot(X, X_b.dot(theta), color="green", label="Initial Line (No GD)") plt.xlabel("Feature") plt.ylabel("Target") plt.title("Linear Regression Without Gradient Descent") plt.legend() plt.show() Output:Linear Regression without Gradient DescentHere the model’s predictions are not accurate and the line does not fit the data well. This happens because the initial parameters are not optimized which prevents the model from finding the best-fit line.Now we will apply gradient descent to improve the model and optimize these parameters.learning_rate = 0.1, n_iterations = 100: Set the learning rate and number of iterations for gradient descent to run respectively.gradients = (2 / m) * X_b.T.dot(y_pred - y): Finding gradients of the cost function with respect to parameters.theta -= learning_rate * gradients: Updating parameters by moving opposite to the gradient direction. Python learning_rate = 0.1 n_iterations = 100 for _ in range(n_iterations): y_pred = X_b.dot(theta) gradients = (2 / m) * X_b.T.dot(y_pred - y) theta -= learning_rate * gradients plt.figure(figsize=(10, 5)) plt.scatter(X, y, color="blue", label="Actual Data") plt.plot(X, X_b.dot(theta), color="red", label="Optimized Line (With GD)") plt.xlabel("Feature") plt.ylabel("Target") plt.title("Linear Regression With Gradient Descent") plt.legend() plt.show() Output:Linear Regression with Gradient DescentLinear Regression with Gradient Descent shows how the model gradually learns to fit the line that minimizes the difference between predicted and actual values by updating parameters step by step.As datasets grow larger and models become more complex, gradient descent will continue to help in building accurate and efficient machine learning systems. Comment More infoAdvertise with us Next Article Gradient Descent in Linear Regression M mohit gupta_omg :) Follow Improve Article Tags : Machine Learning AI-ML-DS ML-Regression python AI-ML-DS With Python +1 More Practice Tags : Machine Learningpython Similar Reads Machine Learning Algorithms Machine learning algorithms are essentially sets of instructions that allow computers to learn from data, make predictions, and improve their performance over time without being explicitly programmed. Machine learning algorithms are broadly categorized into three types: Supervised Learning: Algorith 8 min read Top 15 Machine Learning Algorithms Every Data Scientist Should Know in 2025 Machine Learning (ML) Algorithms are the backbone of everything from Netflix recommendations to fraud detection in financial institutions. These algorithms form the core of intelligent systems, empowering organizations to analyze patterns, predict outcomes, and automate decision-making processes. Wi 14 min read Linear Model RegressionOrdinary Least Squares (OLS) using statsmodelsOrdinary Least Squares (OLS) is a widely used statistical method for estimating the parameters of a linear regression model. It minimizes the sum of squared residuals between observed and predicted values. In this article we will learn how to implement Ordinary Least Squares (OLS) regression using P 3 min read Linear Regression (Python Implementation)Linear regression is a statistical method that is used to predict a continuous dependent variable i.e target variable based on one or more independent variables. This technique assumes a linear relationship between the dependent and independent variables which means the dependent variable changes pr 14 min read Multiple Linear Regression using Python - MLLinear regression is a statistical method used for predictive analysis. It models the relationship between a dependent variable and a single independent variable by fitting a linear equation to the data. Multiple Linear Regression extends this concept by modelling the relationship between a dependen 4 min read Polynomial Regression ( From Scratch using Python )Prerequisites Linear RegressionGradient DescentIntroductionLinear Regression finds the correlation between the dependent variable ( or target variable ) and independent variables ( or features ). In short, it is a linear model to fit the data linearly. But it fails to fit and catch the pattern in no 5 min read Bayesian Linear RegressionLinear regression is based on the assumption that the underlying data is normally distributed and that all relevant predictor variables have a linear relationship with the outcome. But In the real world, this is not always possible, it will follows these assumptions, Bayesian regression could be the 10 min read How to Perform Quantile Regression in PythonIn this article, we are going to see how to perform quantile regression in Python. Linear regression is defined as the statistical method that constructs a relationship between a dependent variable and an independent variable as per the given set of variables. While performing linear regression we a 4 min read Isotonic Regression in Scikit LearnIsotonic regression is a regression technique in which the predictor variable is monotonically related to the target variable. This means that as the value of the predictor variable increases, the value of the target variable either increases or decreases in a consistent, non-oscillating manner. Mat 6 min read Stepwise Regression in PythonStepwise regression is a method of fitting a regression model by iteratively adding or removing variables. It is used to build a model that is accurate and parsimonious, meaning that it has the smallest number of variables that can explain the data. There are two main types of stepwise regression: F 6 min read Least Angle Regression (LARS)Regression is a supervised machine learning task that can predict continuous values (real numbers), as compared to classification, that can predict categorical or discrete values. Before we begin, if you are a beginner, I highly recommend this article. Least Angle Regression (LARS) is an algorithm u 3 min read Linear Model ClassificationLogistic Regression in Machine LearningLogistic Regression is a supervised machine learning algorithm used for classification problems. Unlike linear regression which predicts continuous values it predicts the probability that an input belongs to a specific class. It is used for binary classification where the output can be one of two po 11 min read Understanding Activation Functions in DepthIn artificial neural networks, the activation function of a neuron determines its output for a given input. This output serves as the input for subsequent neurons in the network, continuing the process until the network solves the original problem. Consider a binary classification problem, where the 6 min read RegularizationImplementation of Lasso Regression From Scratch using PythonLasso Regression (Least Absolute Shrinkage and Selection Operator) is a linear regression technique that combines prediction with feature selection. It does this by adding a penalty term to the cost function shrinking less relevant feature's coefficients to zero. This makes it effective for high-dim 7 min read Implementation of Ridge Regression from Scratch using PythonPrerequisites: Linear Regression Gradient Descent Introduction: Ridge Regression ( or L2 Regularization ) is a variation of Linear Regression. In Linear Regression, it minimizes the Residual Sum of Squares ( or RSS or cost function ) to fit the training examples perfectly as possible. The cost funct 4 min read Implementation of Elastic Net Regression From ScratchPrerequisites: Linear RegressionGradient DescentLasso & Ridge RegressionIntroduction: Elastic-Net Regression is a modification of Linear Regression which shares the same hypothetical function for prediction. The cost function of Linear Regression is represented by J. \frac{1}{m} \sum_{i=1}^{m}\l 5 min read K-Nearest Neighbors (KNN)Implementation of Elastic Net Regression From ScratchPrerequisites: Linear RegressionGradient DescentLasso & Ridge RegressionIntroduction: Elastic-Net Regression is a modification of Linear Regression which shares the same hypothetical function for prediction. The cost function of Linear Regression is represented by J. \frac{1}{m} \sum_{i=1}^{m}\l 5 min read Brute Force Approach and its pros and consIn this article, we will discuss the Brute Force Algorithm and what are its pros and cons. What is the Brute Force Algorithm?A brute force algorithm is a simple, comprehensive search strategy that systematically explores every option until a problem's answer is discovered. It's a generic approach to 3 min read Implementation of KNN classifier using Scikit - learn - PythonK-Nearest Neighbors is a most simple but fundamental classifier algorithm in Machine Learning. It is under the supervised learning category and used with great intensity for pattern recognition, data mining and analysis of intrusion. It is widely disposable in real-life scenarios since it is non-par 3 min read Regression using k-Nearest Neighbors in R ProgrammingMachine learning is a subset of Artificial Intelligence that provides a machine with the ability to learn automatically without being explicitly programmed. The machine in such cases improves from the experience without human intervention and adjusts actions accordingly. It is primarily of 3 types: 5 min read Support Vector MachinesSupport Vector Machine (SVM) AlgorithmSupport Vector Machine (SVM) is a supervised machine learning algorithm used for classification and regression tasks. It tries to find the best boundary known as hyperplane that separates different classes in the data. It is useful when you want to do binary classification like spam vs. not spam or 9 min read Classifying data using Support Vector Machines(SVMs) in PythonIntroduction to SVMs: In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. A Support Vector Machine (SVM) is a discriminative classifier 4 min read Support Vector Regression (SVR) using Linear and Non-Linear Kernels in Scikit LearnSupport vector regression (SVR) is a type of support vector machine (SVM) that is used for regression tasks. It tries to find a function that best predicts the continuous output value for a given input value. SVR can use both linear and non-linear kernels. A linear kernel is a simple dot product bet 5 min read Major Kernel Functions in Support Vector Machine (SVM)In previous article we have discussed about SVM(Support Vector Machine) in Machine Learning. Now we are going to learn in detail about SVM Kernel and Different Kernel Functions and its examples.Types of SVM Kernel FunctionsSVM algorithm use the mathematical function defined by the kernel. Kernel Fu 4 min read ML - Stochastic Gradient Descent (SGD) Stochastic Gradient Descent (SGD) is an optimization algorithm in machine learning, particularly when dealing with large datasets. It is a variant of the traditional gradient descent algorithm but offers several advantages in terms of efficiency and scalability, making it the go-to method for many d 8 min read Decision TreeMajor Kernel Functions in Support Vector Machine (SVM)In previous article we have discussed about SVM(Support Vector Machine) in Machine Learning. Now we are going to learn in detail about SVM Kernel and Different Kernel Functions and its examples.Types of SVM Kernel FunctionsSVM algorithm use the mathematical function defined by the kernel. Kernel Fu 4 min read CART (Classification And Regression Tree) in Machine LearningCART( Classification And Regression Trees) is a variation of the decision tree algorithm. It can handle both classification and regression tasks. Scikit-Learn uses the Classification And Regression Tree (CART) algorithm to train Decision Trees (also called “growing†trees). CART was first produced b 11 min read Decision Tree Classifiers in R ProgrammingClassification is the task in which objects of several categories are categorized into their respective classes using the properties of classes. A classification model is typically used to, Predict the class label for a new unlabeled data objectProvide a descriptive model explaining what features ch 4 min read Decision Tree Regression using sklearn - PythonDecision Tree Regression is a method used to predict continuous values like prices or scores by using a tree-like structure. It works by splitting the data into smaller parts based on simple rules taken from the input features. These splits help reduce errors in prediction. At the end of each branch 4 min read Ensemble LearningEnsemble Methods in PythonEnsemble means a group of elements viewed as a whole rather than individually. An Ensemble method creates multiple models and combines them to solve it. Ensemble methods help to improve the robustness/generalizability of the model. In this article, we will discuss some methods with their implementat 11 min read Random Forest Regression in PythonA random forest is an ensemble learning method that combines the predictions from multiple decision trees to produce a more accurate and stable prediction. It is a type of supervised learning algorithm that can be used for both classification and regression tasks.In regression task we can use Random 7 min read ML | Extra Tree Classifier for Feature SelectionPrerequisites: Decision Tree Classifier Extremely Randomized Trees Classifier(Extra Trees Classifier) is a type of ensemble learning technique which aggregates the results of multiple de-correlated decision trees collected in a "forest" to output it's classification result. In concept, it is very si 6 min read Implementing the AdaBoost Algorithm From ScratchAdaBoost means Adaptive Boosting which is a ensemble learning technique that combines multiple weak classifiers to create a strong classifier. It works by sequentially adding classifiers to correct the errors made by previous models giving more weight to the misclassified data points. In this articl 4 min read XGBoostTraditional machine learning models like decision trees and random forests are easy to interpret but often struggle with accuracy on complex datasets. XGBoost short form for eXtreme Gradient Boosting is an advanced machine learning algorithm designed for efficiency, speed and high performance.It is 6 min read CatBoost in Machine LearningWhen working with machine learning we often deal with datasets that include categorical data. We use techniques like One-Hot Encoding or Label Encoding to convert these categorical features into numerical values. However One-Hot Encoding can lead to sparse matrix and cause overfitting. This is where 5 min read LightGBM (Light Gradient Boosting Machine)LightGBM is an open-source high-performance framework developed by Microsoft. It is an ensemble learning framework that uses gradient boosting method which constructs a strong learner by sequentially adding weak learners in a gradient descent manner.It's designed for efficiency, scalability and high 7 min read Stacking in Machine LearningStacking is a ensemble learning technique where the final model known as the “stacked model" combines the predictions from multiple base models. The goal is to create a stronger model by using different models and combining them.Architecture of StackingStacking architecture is like a team of models 3 min read Like