Skip to content
geeksforgeeks
  • Courses
    • DSA to Development
    • Get IBM Certification
    • Newly Launched!
      • Master Django Framework
      • Become AWS Certified
    • For Working Professionals
      • Interview 101: DSA & System Design
      • Data Science Training Program
      • JAVA Backend Development (Live)
      • DevOps Engineering (LIVE)
      • Data Structures & Algorithms in Python
    • For Students
      • Placement Preparation Course
      • Data Science (Live)
      • Data Structure & Algorithm-Self Paced (C++/JAVA)
      • Master Competitive Programming (Live)
      • Full Stack Development with React & Node JS (Live)
    • Full Stack Development
    • Data Science Program
    • All Courses
  • Tutorials
    • Data Structures & Algorithms
    • ML & Data Science
    • Interview Corner
    • Programming Languages
    • Web Development
    • CS Subjects
    • DevOps And Linux
    • School Learning
  • Practice
    • Build your AI Agent
    • GfG 160
    • Problem of the Day
    • Practice Coding Problems
    • GfG SDE Sheet
  • Contests
    • Accenture Hackathon (Ending Soon!)
    • GfG Weekly [Rated Contest]
    • Job-A-Thon Hiring Challenge
    • All Contests and Events
  • Data preprocessing
  • Data Manipulation
  • Data Analysis using Pandas
  • EDA
  • Pandas Exercise
  • Pandas AI
  • Numpy
  • Matplotlib
  • Plotly
  • Data Analysis
  • Machine Learning
  • Data science
Open In App
Next Article:
Python | Pandas Index.duplicated()
Next article icon

Pandas Find Duplicate Rows

Last Updated : 17 Jan, 2025
Comments
Improve
Suggest changes
Like Article
Like
Report

Most simple way to find duplicate rows in DataFrame is by using the duplicated() method. This method returns a boolean Series indicating whether each row is a duplicate of a previous row.

Python 
import pandas as pd  data = {'Name': ['John', 'Alice', 'Bob', 'Eve', 'John', 'Charlie'],          'Age': [25, 30, 22, 35, 25, 28],          'Gender': ['Male', 'Female', 'Male', 'Female', 'Male', 'Male'],          'Salary': [50000, 55000, 40000, 70000, 50000, 48000]}  df = pd.DataFrame(data) # Find duplicate rows duplicates = df[df.duplicated()] print(duplicates)

Output
   Name  Age Gender  Salary 4  John   25   Male   50000

In addition to the this method, there are several other approaches to find out the Duplicate rows in Pandas.

1. Identifying Duplicates Based on Specific Columns

Sometimes, you may only want to check for duplicates in specific columns, rather than the entire row. we can use duplicated() with the subset parameter to check for duplicates in specific columns.

The subset parameter allows you to specify which columns to consider when looking for duplicates. In this case, we're checking for duplicates based on the Name and Age columns.

Python 
# Find duplicates based on 'Name' and 'Age' columns duplicates_by_columns = df[df.duplicated(subset=['Name', 'Age'])] print(duplicates_by_columns)

Output
   Name  Age Gender  Salary 4  John   25   Male   50000

2. Removing Duplicate Rows Using drop duplicates

Once duplicates are identified, you can remove them using the drop_duplicates() method. This method returns a new DataFrame with the duplicate rows removed.

The drop_duplicates() method removes all rows that are identical to a previous row. By default, it keeps the first occurrence of each row and drops subsequent duplicates.

Python 
# Remove duplicate rows df_no_duplicates = df.drop_duplicates() print(df_no_duplicates)

Output
      Name  Age  Gender  Salary 0     John   25    Male   50000 1    Alice   30  Female   55000 2      Bob   22    Male   40000 3      Eve   35  Female   70000 5  Charlie   28    Male   48000

Here are some key takeaways:

  • Identify duplicates using duplicated()
  • Find duplicates based on specific columns
  • Remove duplicate rows with drop_duplicates()

Next Article
Python | Pandas Index.duplicated()

A
abhirajksingh
Improve
Article Tags :
  • Pandas
  • AI-ML-DS
  • Python-pandas
  • Python pandas-basics

Similar Reads

  • Python | Pandas Index.duplicated()
    The Index.duplicated() method in Pandas is a powerful tool for identifying duplicate values within an index. It returns a boolean array where duplicates are marked as True based on the specified criteria and False denotes unique values or the first occurrence of duplicates. This method is especially
    5 min read
  • Python | Pandas Index.get_duplicates()
    Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric python packages. Pandas is one of those packages and makes importing and analyzing data much easier. Pandas Index.get_duplicates() function extract duplicated index elements. This functio
    2 min read
  • Python | Pandas Index.drop_duplicates()
    Pandas Index.drop_duplicates() function return Index with duplicate values removed in Python. Syntax of Pandas Index.drop_duplicates() Syntax: Index.drop_duplicates(labels, errors='raise') Parameters : keep : {‘first’, ‘last’, False} ‘first’ : Drop duplicates except for the first occurrence.(default
    2 min read
  • Python | Pandas Series.duplicated()
    Pandas series is a One-dimensional ndarray with axis labels. The labels need not be unique but must be a hashable type. The object supports both integer and label-based indexing and provides a host of methods for performing operations involving the index. Pandas Series.duplicated() function indicate
    2 min read
  • Data Structures in Pandas
    Pandas is an open-source library that uses for working with relational or labeled data both easily and intuitively. It provides various data structures and operations for manipulating numerical data and time series. It offers a tool for cleaning and processes your data. It is the most popular Python
    4 min read
  • Python | Pandas Index.identical()
    Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric python packages. Pandas is one of those packages and makes importing and analyzing data much easier. Pandas Index.identical() function determine if two Index objects contains the same ele
    2 min read
  • How to count duplicates in Pandas Dataframe?
    Let us see how to count duplicates in a Pandas DataFrame. Our task is to count the number of duplicate entries in a single column and multiple columns. Under a single column : We will be using the pivot_table() function to count the duplicates in a single column. The column in which the duplicates a
    2 min read
  • Python | Pandas Series.drop_duplicates()
    Pandas Series.drop_duplicates() function returns a series object with duplicate values removed from the given series object. Syntax: Series.drop_duplicates(keep='first', inplace=False) Parameter : keep : {‘first’, ‘last’, False}, default ‘first’ inplace : If True, performs operation inplace and retu
    2 min read
  • Python | Pandas DataFrame.isin()
    In this article, we will explore the Pandas DataFrame.isin() method provided by the Pandas library in Python. Python is widely recognized for its proficiency in data analysis, largely attributed to its exceptional ecosystem of data-centric packages. Among these, Pandas stands out as an essential too
    2 min read
  • Python | Pandas TimedeltaIndex.duplicated
    Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric python packages. Pandas is one of those packages and makes importing and analyzing data much easier. Pandas TimedeltaIndex.duplicated() function detects duplicate values in the given Time
    2 min read
geeksforgeeks-footer-logo
Corporate & Communications Address:
A-143, 7th Floor, Sovereign Corporate Tower, Sector- 136, Noida, Uttar Pradesh (201305)
Registered Address:
K 061, Tower K, Gulshan Vivante Apartment, Sector 137, Noida, Gautam Buddh Nagar, Uttar Pradesh, 201305
GFG App on Play Store GFG App on App Store
Advertise with us
  • Company
  • About Us
  • Legal
  • Privacy Policy
  • In Media
  • Contact Us
  • Advertise with us
  • GFG Corporate Solution
  • Placement Training Program
  • Languages
  • Python
  • Java
  • C++
  • PHP
  • GoLang
  • SQL
  • R Language
  • Android Tutorial
  • Tutorials Archive
  • DSA
  • Data Structures
  • Algorithms
  • DSA for Beginners
  • Basic DSA Problems
  • DSA Roadmap
  • Top 100 DSA Interview Problems
  • DSA Roadmap by Sandeep Jain
  • All Cheat Sheets
  • Data Science & ML
  • Data Science With Python
  • Data Science For Beginner
  • Machine Learning
  • ML Maths
  • Data Visualisation
  • Pandas
  • NumPy
  • NLP
  • Deep Learning
  • Web Technologies
  • HTML
  • CSS
  • JavaScript
  • TypeScript
  • ReactJS
  • NextJS
  • Bootstrap
  • Web Design
  • Python Tutorial
  • Python Programming Examples
  • Python Projects
  • Python Tkinter
  • Python Web Scraping
  • OpenCV Tutorial
  • Python Interview Question
  • Django
  • Computer Science
  • Operating Systems
  • Computer Network
  • Database Management System
  • Software Engineering
  • Digital Logic Design
  • Engineering Maths
  • Software Development
  • Software Testing
  • DevOps
  • Git
  • Linux
  • AWS
  • Docker
  • Kubernetes
  • Azure
  • GCP
  • DevOps Roadmap
  • System Design
  • High Level Design
  • Low Level Design
  • UML Diagrams
  • Interview Guide
  • Design Patterns
  • OOAD
  • System Design Bootcamp
  • Interview Questions
  • Inteview Preparation
  • Competitive Programming
  • Top DS or Algo for CP
  • Company-Wise Recruitment Process
  • Company-Wise Preparation
  • Aptitude Preparation
  • Puzzles
  • School Subjects
  • Mathematics
  • Physics
  • Chemistry
  • Biology
  • Social Science
  • English Grammar
  • Commerce
  • World GK
  • GeeksforGeeks Videos
  • DSA
  • Python
  • Java
  • C++
  • Web Development
  • Data Science
  • CS Subjects
@GeeksforGeeks, Sanchhaya Education Private Limited, All rights reserved
We use cookies to ensure you have the best browsing experience on our website. By using our site, you acknowledge that you have read and understood our Cookie Policy & Privacy Policy
Lightbox
Improvement
Suggest Changes
Help us improve. Share your suggestions to enhance the article. Contribute your expertise and make a difference in the GeeksforGeeks portal.
geeksforgeeks-suggest-icon
Create Improvement
Enhance the article with your expertise. Contribute to the GeeksforGeeks community and help create better learning resources for all.
geeksforgeeks-improvement-icon
Suggest Changes
min 4 words, max Words Limit:1000

Thank You!

Your suggestions are valuable to us.

What kind of Experience do you want to share?

Interview Experiences
Admission Experiences
Career Journeys
Work Experiences
Campus Experiences
Competitive Exam Experiences