Skip to content
geeksforgeeks
  • Tutorials
    • Python
    • Java
    • Data Structures & Algorithms
    • ML & Data Science
    • Interview Corner
    • Programming Languages
    • Web Development
    • CS Subjects
    • DevOps And Linux
    • School Learning
    • Practice Coding Problems
  • Courses
    • DSA to Development
    • Get IBM Certification
    • Newly Launched!
      • Master Django Framework
      • Become AWS Certified
    • For Working Professionals
      • Interview 101: DSA & System Design
      • Data Science Training Program
      • JAVA Backend Development (Live)
      • DevOps Engineering (LIVE)
      • Data Structures & Algorithms in Python
    • For Students
      • Placement Preparation Course
      • Data Science (Live)
      • Data Structure & Algorithm-Self Paced (C++/JAVA)
      • Master Competitive Programming (Live)
      • Full Stack Development with React & Node JS (Live)
    • Full Stack Development
    • Data Science Program
    • All Courses
  • Data Science
  • Data Science Projects
  • Data Analysis
  • Data Visualization
  • Machine Learning
  • ML Projects
  • Deep Learning
  • NLP
  • Computer Vision
  • Artificial Intelligence
Open In App
Next Article:
Expectation-Maximization Algorithm - ML
Next article icon

Expectation-Maximization Algorithm - ML

Last Updated : 16 May, 2025
Comments
Improve
Suggest changes
Like Article
Like
Report

Expectation-Maximization (EM) algorithm is a iterative method used in unsupervised machine learning to find unknown values in statistical models. It helps to find the best values for unknown parameters especially when some data is missing or hidden. It works in two steps:

  • E-step (Expectation Step): Estimates missing or hidden values using current parameter estimates.
  • M-step (Maximization Step): Updates model parameters to maximize the likelihood based on the estimated values from the E-step.

This process repeats until the model reaches a stable solution as it improve accuracy with each iteration. It is widely used in clustering like Gaussian Mixture Models and handling missing data.

Expectation-Maximization in EM Algorithm-Geeksforgeeks
Expectation-Maximization in EM Algorithm

By iteratively repeating these steps the EM algorithm seeks to maximize the likelihood of the observed data.

Key Terms in Expectation-Maximization (EM) Algorithm

Lets understand about some of the most commonly used key terms in the Expectation-Maximization (EM) Algorithm:

  • Latent Variables: These are hidden parts of the data that we can’t see directly but they still affect what we do see. We try to guess their values using the visible data.
  • Likelihood: This refers to the probability of seeing the data we have based on certain assumptions or parameters. The EM algorithm tries to find the best parameters that make the data most likely.
  • Log-Likelihood: This is just the natural log of the likelihood function. It's used to make calculations easier and measure how well the model fits the data. The EM algorithm tries to maximize the log-likelihood to improve the model fit.
  • Maximum Likelihood Estimation (MLE): This is a method to find the best values for a model’s settings called parameters. It looks for the values that make the data we observed most likely to happen.
  • Posterior Probability: In Bayesian methods this is the probability of the parameters given both prior knowledge and the observed data. In EM it helps estimate the "best" parameters when there's uncertainty about the data.
  • Expectation (E) Step: In this step the algorithm estimates the missing or hidden information (latent variables) based on the observed data and current parameters. It calculates probabilities for the hidden values given what we can see.
  • Maximization (M) Step: This step update the parameters by finding the values that maximize the likelihood based on the estimates from the E-step.
  • Convergence: Convergence happens when the algorithm has reached a stable point. This is checked by seeing if the changes in the model's parameters or the log-likelihood are small enough to stop the process.

Working of Expectation-Maximization (EM) Algorithm

So far, we've discussed the key terms in the EM algorithm. Now, let's dive into how the EM algorithm works. Here's a step-by-step breakdown of the process:

EM Algorithm Flow chart-Geeksforgeeks
EM Algorithm Flowchart

1. Initialization: The algorithm starts with initial parameter values and assumes the observed data comes from a specific model.

2. E-Step (Expectation Step):

  • Find the missing or hidden data based on the current parameters.
  • Calculate the posterior probability of each latent variable based on the observed data.
  • Compute the log-likelihood of the observed data using the current parameter estimates.

3. M-Step (Maximization Step):

  • Update the model parameters by maximize the log-likelihood.
  • The better the model the higher this value.

4. Convergence:

  • Check if the model parameters are stable and converging.
  • If the changes in log-likelihood or parameters are below a set threshold, stop. If not repeat the E-step and M-step until convergence is reached

Implementation of Expectation-Maximization Algorithm

Step 1 : Import the necessary libraries

First we will import the necessary Python libraries like NumPy, Seaborn, Matplotlib and SciPy.

Python 
import numpy as np import seaborn as sns from scipy.stats import norm from scipy.stats import gaussian_kde import matplotlib.pyplot as plt

Step 2 : Generate a dataset with two Gaussian components

We generate two sets of data values from two different normal distributions:

  • One centered around 2 (with more spread).
  • Another around -1 (with less spread).

These two sets are then combined to form a single dataset. We plot this dataset to visualize how the values are distributed.

Python 
mu1, sigma1 = 2, 1 mu2, sigma2 = -1, 0.8 X1 = np.random.normal(mu1, sigma1, size=200) X2 = np.random.normal(mu2, sigma2, size=600) X = np.concatenate([X1, X2])  sns.kdeplot(X) plt.xlabel('X') plt.ylabel('Density') plt.title('Density Estimation of X') plt.show()

Output:

Density Plot-Geeksforgeeks
Density Plot

Step 3: Initialize parameters

We make initial guesses for each group’s:

  • Mean (average),
  • Standard deviation (spread),
  • Proportion (how much each group contributes to the total data).
Python 
mu1_hat, sigma1_hat = np.mean(X1), np.std(X1) mu2_hat, sigma2_hat = np.mean(X2), np.std(X2) pi1_hat, pi2_hat = len(X1) / len(X), len(X2) / len(X)

Step 4: Perform EM algorithm

We run a loop for 20 rounds called epochs. In each round:

  • The E-step calculates the responsibilities (gamma values) by evaluating the Gaussian probability densities for each component and weighting them by the corresponding proportions.
  • The M-step updates the parameters by computing the weighted mean and standard deviation for each component

We also calculate the log-likelihood in each round to check if the model is getting better. This is a measure of how well the model explains the data.

Python 
num_epochs = 20 log_likelihoods = []  for epoch in range(num_epochs):     # E-step: Compute responsibilities     gamma1 = pi1_hat * norm.pdf(X, mu1_hat, sigma1_hat)     gamma2 = pi2_hat * norm.pdf(X, mu2_hat, sigma2_hat)     total = gamma1 + gamma2     gamma1 /= total     gamma2 /= total          # M-step: Update parameters     mu1_hat = np.sum(gamma1 * X) / np.sum(gamma1)     mu2_hat = np.sum(gamma2 * X) / np.sum(gamma2)     sigma1_hat = np.sqrt(np.sum(gamma1 * (X - mu1_hat)**2) / np.sum(gamma1))     sigma2_hat = np.sqrt(np.sum(gamma2 * (X - mu2_hat)**2) / np.sum(gamma2))     pi1_hat = np.mean(gamma1)     pi2_hat = np.mean(gamma2)          # Compute log-likelihood     log_likelihood = np.sum(np.log(pi1_hat * norm.pdf(X, mu1_hat, sigma1_hat)                                    + pi2_hat * norm.pdf(X, mu2_hat, sigma2_hat)))     log_likelihoods.append(log_likelihood)   plt.plot(range(1, num_epochs+1), log_likelihoods) plt.xlabel('Epoch') plt.ylabel('Log-Likelihood') plt.title('Log-Likelihood vs. Epoch') plt.show()

Output:

Epoch vs Log-likelihood-Geeksforgeeks
Epoch vs Log-likelihood

Step 5: Visualize the Final Result

Now we will finally visualize the curve which compare the final estimated curve (in red) with the original data’s smooth curve (in green).

Python 
X_sorted = np.sort(X) density_estimation = pi1_hat*norm.pdf(X_sorted,                                         mu1_hat,                                          sigma1_hat) + pi2_hat * norm.pdf(X_sorted,                                                                          mu2_hat,                                                                           sigma2_hat)   plt.plot(X_sorted, gaussian_kde(X_sorted)(X_sorted), color='green', linewidth=2) plt.plot(X_sorted, density_estimation, color='red', linewidth=2) plt.xlabel('X') plt.ylabel('Density') plt.title('Density Estimation of X') plt.legend(['Kernel Density Estimation','Mixture Density']) plt.show()

Output:

Estimated density-Geeksforgeeks
 Estimated density

The above image compares Kernel Density Estimation (green) and Mixture Density (red) for variable X. Both show similar patterns with a main peak near -1.5 and a smaller bump around 2 indicate two data clusters. The red curve is slightly smoother and sharper than the green one.

Advantages of EM algorithm

  • Always improves results – With each step, the algorithm improves the likelihood (chances) of finding a good solution.
  • Simple to implement – The two steps (E-step and M-step) are often easy to code for many problems.
  • Quick math solutions – In many cases, the M-step has a direct mathematical solution (closed-form), making it efficient

Disadvantages of EM algorithm

  • Takes time to finish: It converges slowly meaning it may take many iterations to reach the best solution.
  • Gets stuck in local best: Instead of finding the absolute best solution, it might settle for a "good enough" one.
  • Needs extra probabilities: Unlike some optimization methods that only need forward probability, EM requires both forward and backward probabilities making it slightly more complex.

The EM algorithm iteratively estimates missing data and updates model parameters to improve accuracy. By alternating between the E-step and M-step it refines the model until it converges making it widely used tool for handling hidden or incomplete data in machine learning.


Next Article
Expectation-Maximization Algorithm - ML

R
raman_257
Improve
Article Tags :
  • Machine Learning
  • AI-ML-DS
  • ML-Clustering
  • AI-ML-DS With Python
Practice Tags :
  • Machine Learning

Similar Reads

    Machine Learning Algorithms
    Machine learning algorithms are essentially sets of instructions that allow computers to learn from data, make predictions, and improve their performance over time without being explicitly programmed. Machine learning algorithms are broadly categorized into three types: Supervised Learning: Algorith
    8 min read
    Top 15 Machine Learning Algorithms Every Data Scientist Should Know in 2025
    Machine Learning (ML) Algorithms are the backbone of everything from Netflix recommendations to fraud detection in financial institutions. These algorithms form the core of intelligent systems, empowering organizations to analyze patterns, predict outcomes, and automate decision-making processes. Wi
    14 min read

    Linear Model Regression

    Ordinary Least Squares (OLS) using statsmodels
    Ordinary Least Squares (OLS) is a widely used statistical method for estimating the parameters of a linear regression model. It minimizes the sum of squared residuals between observed and predicted values. In this article we will learn how to implement Ordinary Least Squares (OLS) regression using P
    3 min read
    Linear Regression (Python Implementation)
    Linear regression is a statistical method that is used to predict a continuous dependent variable i.e target variable based on one or more independent variables. This technique assumes a linear relationship between the dependent and independent variables which means the dependent variable changes pr
    14 min read
    Multiple Linear Regression using Python - ML
    Linear regression is a statistical method used for predictive analysis. It models the relationship between a dependent variable and a single independent variable by fitting a linear equation to the data. Multiple Linear Regression extends this concept by modelling the relationship between a dependen
    4 min read
    Polynomial Regression ( From Scratch using Python )
    Prerequisites Linear RegressionGradient DescentIntroductionLinear Regression finds the correlation between the dependent variable ( or target variable ) and independent variables ( or features ). In short, it is a linear model to fit the data linearly. But it fails to fit and catch the pattern in no
    5 min read
    Bayesian Linear Regression
    Linear regression is based on the assumption that the underlying data is normally distributed and that all relevant predictor variables have a linear relationship with the outcome. But In the real world, this is not always possible, it will follows these assumptions, Bayesian regression could be the
    10 min read
    How to Perform Quantile Regression in Python
    In this article, we are going to see how to perform quantile regression in Python. Linear regression is defined as the statistical method that constructs a relationship between a dependent variable and an independent variable as per the given set of variables. While performing linear regression we a
    4 min read
    Isotonic Regression in Scikit Learn
    Isotonic regression is a regression technique in which the predictor variable is monotonically related to the target variable. This means that as the value of the predictor variable increases, the value of the target variable either increases or decreases in a consistent, non-oscillating manner. Mat
    6 min read
    Stepwise Regression in Python
    Stepwise regression is a method of fitting a regression model by iteratively adding or removing variables. It is used to build a model that is accurate and parsimonious, meaning that it has the smallest number of variables that can explain the data. There are two main types of stepwise regression: F
    6 min read
    Least Angle Regression (LARS)
    Regression is a supervised machine learning task that can predict continuous values (real numbers), as compared to classification, that can predict categorical or discrete values. Before we begin, if you are a beginner, I highly recommend this article. Least Angle Regression (LARS) is an algorithm u
    3 min read

    Linear Model Classification

    Logistic Regression in Machine Learning
    Logistic Regression is a supervised machine learning algorithm used for classification problems. Unlike linear regression which predicts continuous values it predicts the probability that an input belongs to a specific class. It is used for binary classification where the output can be one of two po
    11 min read
    Understanding Activation Functions in Depth
    In artificial neural networks, the activation function of a neuron determines its output for a given input. This output serves as the input for subsequent neurons in the network, continuing the process until the network solves the original problem. Consider a binary classification problem, where the
    6 min read

    Regularization

    Implementation of Lasso Regression From Scratch using Python
    Lasso Regression (Least Absolute Shrinkage and Selection Operator) is a linear regression technique that combines prediction with feature selection. It does this by adding a penalty term to the cost function shrinking less relevant feature's coefficients to zero. This makes it effective for high-dim
    7 min read
    Implementation of Ridge Regression from Scratch using Python
    Prerequisites: Linear Regression Gradient Descent Introduction: Ridge Regression ( or L2 Regularization ) is a variation of Linear Regression. In Linear Regression, it minimizes the Residual Sum of Squares ( or RSS or cost function ) to fit the training examples perfectly as possible. The cost funct
    4 min read
    Implementation of Elastic Net Regression From Scratch
    Prerequisites: Linear RegressionGradient DescentLasso & Ridge RegressionIntroduction: Elastic-Net Regression is a modification of Linear Regression which shares the same hypothetical function for prediction. The cost function of Linear Regression is represented by J. \frac{1}{m} \sum_{i=1}^{m}\l
    5 min read

    K-Nearest Neighbors (KNN)

    Implementation of Elastic Net Regression From Scratch
    Prerequisites: Linear RegressionGradient DescentLasso & Ridge RegressionIntroduction: Elastic-Net Regression is a modification of Linear Regression which shares the same hypothetical function for prediction. The cost function of Linear Regression is represented by J. \frac{1}{m} \sum_{i=1}^{m}\l
    5 min read
    Brute Force Approach and its pros and cons
    In this article, we will discuss the Brute Force Algorithm and what are its pros and cons. What is the Brute Force Algorithm?A brute force algorithm is a simple, comprehensive search strategy that systematically explores every option until a problem's answer is discovered. It's a generic approach to
    3 min read
    Implementation of KNN classifier using Scikit - learn - Python
    K-Nearest Neighbors is a most simple but fundamental classifier algorithm in Machine Learning. It is under the supervised learning category and used with great intensity for pattern recognition, data mining and analysis of intrusion. It is widely disposable in real-life scenarios since it is non-par
    3 min read
    Regression using k-Nearest Neighbors in R Programming
    Machine learning is a subset of Artificial Intelligence that provides a machine with the ability to learn automatically without being explicitly programmed. The machine in such cases improves from the experience without human intervention and adjusts actions accordingly. It is primarily of 3 types:
    5 min read

    Support Vector Machines

    Support Vector Machine (SVM) Algorithm
    Support Vector Machine (SVM) is a supervised machine learning algorithm used for classification and regression tasks. It tries to find the best boundary known as hyperplane that separates different classes in the data. It is useful when you want to do binary classification like spam vs. not spam or
    9 min read
    Classifying data using Support Vector Machines(SVMs) in Python
    Introduction to SVMs: In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. A Support Vector Machine (SVM) is a discriminative classifier
    4 min read
    Support Vector Regression (SVR) using Linear and Non-Linear Kernels in Scikit Learn
    Support vector regression (SVR) is a type of support vector machine (SVM) that is used for regression tasks. It tries to find a function that best predicts the continuous output value for a given input value. SVR can use both linear and non-linear kernels. A linear kernel is a simple dot product bet
    5 min read
    Major Kernel Functions in Support Vector Machine (SVM)
    In previous article we have discussed about SVM(Support Vector Machine) in Machine Learning. Now we are going to learn  in detail about SVM Kernel and Different Kernel Functions and its examples.Types of SVM Kernel FunctionsSVM algorithm use the mathematical function defined by the kernel. Kernel Fu
    4 min read
    ML - Stochastic Gradient Descent (SGD)
    Stochastic Gradient Descent (SGD) is an optimization algorithm in machine learning, particularly when dealing with large datasets. It is a variant of the traditional gradient descent algorithm but offers several advantages in terms of efficiency and scalability, making it the go-to method for many d
    8 min read

    Decision Tree

    Major Kernel Functions in Support Vector Machine (SVM)
    In previous article we have discussed about SVM(Support Vector Machine) in Machine Learning. Now we are going to learn  in detail about SVM Kernel and Different Kernel Functions and its examples.Types of SVM Kernel FunctionsSVM algorithm use the mathematical function defined by the kernel. Kernel Fu
    4 min read
    CART (Classification And Regression Tree) in Machine Learning
    CART( Classification And Regression Trees) is a variation of the decision tree algorithm. It can handle both classification and regression tasks. Scikit-Learn uses the Classification And Regression Tree (CART) algorithm to train Decision Trees (also called “growing” trees). CART was first produced b
    11 min read
    Decision Tree Classifiers in R Programming
    Classification is the task in which objects of several categories are categorized into their respective classes using the properties of classes. A classification model is typically used to, Predict the class label for a new unlabeled data objectProvide a descriptive model explaining what features ch
    4 min read
    Decision Tree Regression using sklearn - Python
    Decision Tree Regression is a method used to predict continuous values like prices or scores by using a tree-like structure. It works by splitting the data into smaller parts based on simple rules taken from the input features. These splits help reduce errors in prediction. At the end of each branch
    4 min read

    Ensemble Learning

    Ensemble Methods in Python
    Ensemble means a group of elements viewed as a whole rather than individually. An Ensemble method creates multiple models and combines them to solve it. Ensemble methods help to improve the robustness/generalizability of the model. In this article, we will discuss some methods with their implementat
    11 min read
    Random Forest Regression in Python
    A random forest is an ensemble learning method that combines the predictions from multiple decision trees to produce a more accurate and stable prediction. It is a type of supervised learning algorithm that can be used for both classification and regression tasks.In regression task we can use Random
    7 min read
    ML | Extra Tree Classifier for Feature Selection
    Prerequisites: Decision Tree Classifier Extremely Randomized Trees Classifier(Extra Trees Classifier) is a type of ensemble learning technique which aggregates the results of multiple de-correlated decision trees collected in a "forest" to output it's classification result. In concept, it is very si
    6 min read
    Implementing the AdaBoost Algorithm From Scratch
    AdaBoost means Adaptive Boosting which is a ensemble learning technique that combines multiple weak classifiers to create a strong classifier. It works by sequentially adding classifiers to correct the errors made by previous models giving more weight to the misclassified data points. In this articl
    4 min read
    XGBoost
    Traditional machine learning models like decision trees and random forests are easy to interpret but often struggle with accuracy on complex datasets. XGBoost short form for eXtreme Gradient Boosting is an advanced machine learning algorithm designed for efficiency, speed and high performance.It is
    6 min read
    CatBoost in Machine Learning
    When working with machine learning we often deal with datasets that include categorical data. We use techniques like One-Hot Encoding or Label Encoding to convert these categorical features into numerical values. However One-Hot Encoding can lead to sparse matrix and cause overfitting. This is where
    5 min read
    LightGBM (Light Gradient Boosting Machine)
    LightGBM is an open-source high-performance framework developed by Microsoft. It is an ensemble learning framework that uses gradient boosting method which constructs a strong learner by sequentially adding weak learners in a gradient descent manner.It's designed for efficiency, scalability and high
    7 min read
    Stacking in Machine Learning
    Stacking is a ensemble learning technique where the final model known as the “stacked model" combines the predictions from multiple base models. The goal is to create a stronger model by using different models and combining them.Architecture of StackingStacking architecture is like a team of models
    3 min read
geeksforgeeks-footer-logo
Corporate & Communications Address:
A-143, 7th Floor, Sovereign Corporate Tower, Sector- 136, Noida, Uttar Pradesh (201305)
Registered Address:
K 061, Tower K, Gulshan Vivante Apartment, Sector 137, Noida, Gautam Buddh Nagar, Uttar Pradesh, 201305
GFG App on Play Store GFG App on App Store
Advertise with us
  • Company
  • About Us
  • Legal
  • Privacy Policy
  • In Media
  • Contact Us
  • Advertise with us
  • GFG Corporate Solution
  • Placement Training Program
  • Languages
  • Python
  • Java
  • C++
  • PHP
  • GoLang
  • SQL
  • R Language
  • Android Tutorial
  • Tutorials Archive
  • DSA
  • Data Structures
  • Algorithms
  • DSA for Beginners
  • Basic DSA Problems
  • DSA Roadmap
  • Top 100 DSA Interview Problems
  • DSA Roadmap by Sandeep Jain
  • All Cheat Sheets
  • Data Science & ML
  • Data Science With Python
  • Data Science For Beginner
  • Machine Learning
  • ML Maths
  • Data Visualisation
  • Pandas
  • NumPy
  • NLP
  • Deep Learning
  • Web Technologies
  • HTML
  • CSS
  • JavaScript
  • TypeScript
  • ReactJS
  • NextJS
  • Bootstrap
  • Web Design
  • Python Tutorial
  • Python Programming Examples
  • Python Projects
  • Python Tkinter
  • Python Web Scraping
  • OpenCV Tutorial
  • Python Interview Question
  • Django
  • Computer Science
  • Operating Systems
  • Computer Network
  • Database Management System
  • Software Engineering
  • Digital Logic Design
  • Engineering Maths
  • Software Development
  • Software Testing
  • DevOps
  • Git
  • Linux
  • AWS
  • Docker
  • Kubernetes
  • Azure
  • GCP
  • DevOps Roadmap
  • System Design
  • High Level Design
  • Low Level Design
  • UML Diagrams
  • Interview Guide
  • Design Patterns
  • OOAD
  • System Design Bootcamp
  • Interview Questions
  • Inteview Preparation
  • Competitive Programming
  • Top DS or Algo for CP
  • Company-Wise Recruitment Process
  • Company-Wise Preparation
  • Aptitude Preparation
  • Puzzles
  • School Subjects
  • Mathematics
  • Physics
  • Chemistry
  • Biology
  • Social Science
  • English Grammar
  • Commerce
  • World GK
  • GeeksforGeeks Videos
  • DSA
  • Python
  • Java
  • C++
  • Web Development
  • Data Science
  • CS Subjects
@GeeksforGeeks, Sanchhaya Education Private Limited, All rights reserved
We use cookies to ensure you have the best browsing experience on our website. By using our site, you acknowledge that you have read and understood our Cookie Policy & Privacy Policy
Lightbox
Improvement
Suggest Changes
Help us improve. Share your suggestions to enhance the article. Contribute your expertise and make a difference in the GeeksforGeeks portal.
geeksforgeeks-suggest-icon
Create Improvement
Enhance the article with your expertise. Contribute to the GeeksforGeeks community and help create better learning resources for all.
geeksforgeeks-improvement-icon
Suggest Changes
min 4 words, max Words Limit:1000

Thank You!

Your suggestions are valuable to us.

What kind of Experience do you want to share?

Interview Experiences
Admission Experiences
Career Journeys
Work Experiences
Campus Experiences
Competitive Exam Experiences