Skip to content

Pandas dataframe.drop_duplicates()

Pandas DataFrame duplicated() Method | Pandas Method

Last Updated : 02 Feb, 2024

Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages and makes importing and analyzing data much easier.

Pandas duplicated() method identifies duplicated rows in a DataFrame. It returns a boolean series which is True only for unique rows.

Example:

Python3

   import pandas as pd 
df = pd.DataFrame({ 
    'Name': ['Alice', 'Bob', 'Alice', 'Charlie'], 
    'Age': [25, 32, 25, 37] 
}) 
duplicates = df[df.duplicated()] 
print(duplicates)
 
  

Output:

Name  Age  2  Alice   25

Syntax

Syntax: DataFrame.duplicated(subset=None, keep=’first’)

Parameters:

subset: Takes a column or list of column label. It’s default value is none. After passing columns, it will consider them only for duplicates.

keep: Controls how to consider duplicate value. It has only three distinct values and the default is ‘first’.
–> If ‘first‘, it considers the first value as unique and the rest of the same values as duplicate.
–> If ‘last‘, it considers the last value as unique and the rest of the same values as duplicate.
–> If False, it considers all of the same values as duplicates.

Returns: A series with boolean values for each row in the DataFrame

To download the CSV file used, Click Here.

Examples

Let’s look at some examples of the duplicated method in Pandas library used to identify duplicated rows in a DataFrame.

Example 1: Returning a boolean series

In the following example, a boolean series is returned based on duplicate values in the First Name column.

Python

   # importing pandas package 
import pandas as pd 
  
# making data frame from csv file 
data = pd.read_csv("employees.csv") 
  
# sorting by first name 
data.sort_values("First Name", inplace = True) 
  
# making a bool series 
bool_series = data["First Name"].duplicated() 
  
# displaying data 
data.head() 
  
# display data 
data[bool_series] 
 
  

Output:
As shown in the output image, since the keep parameter was a default that is ‘first‘, hence whenever the name occurs, the first one is considered Unique, and the rest Duplicate.

Example 2: Removing duplicates

In this example, the keep parameter is set to False, so that only Unique values are taken and the duplicate values are removed from DataFrame.

Python

   # importing pandas package 
import pandas as pd 
  
# making data frame from csv file 
data = pd.read_csv("employees.csv") 
  
# sorting by first name 
data.sort_values("First Name", inplace = True) 
  
# making a bool series 
bool_series = data["First Name"].duplicated(keep = False) 
  
# bool series 
bool_series 
  
# passing NOT of bool series to see unique values only 
data = data[~bool_series] 
  
# displaying data 
data.info() 
data 
 
  

Output:
Since the duplicated() method returns False for duplicates, the NOT of the series is taken to see the unique values in the DataFrame.

removing duplicate value with duplicated output

Pandas dataframe.drop_duplicates()

K

Kartikaybhutani

Improve

Article Tags :

Practice Tags :

python

Similar Reads

Apply function to every row in a Pandas DataFrame

Python is a great language for performing data analysis tasks. It provides a huge amount of Classes and functions which help in analyzing and manipulating data more easily. In this article, we will see how we can apply a function to every row in a Pandas Dataframe. Apply Function to Every Row in a P

Joining two Pandas DataFrames using merge()

The merge() function is designed to merge two DataFrames based on one or more columns with matching values. The basic idea is to identify columns that contain common data between the DataFrames and use them to align rows. Let's understand the process of joining two pandas DataFrames usingÂ merge(), e

Python | Pandas DataFrame.astype()

Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages and makes importing and analyzing data much easier. DataFrame.astype() method is used to cast a pandas object to a specified dtype.astype(

Python | Pandas DataFrame.set_index()

Pandas DataFrame.set_index() method sets one or more columns as the index of a DataFrame. It can accept single or multiple column names and is useful for modifying or adding new indices to your DataFrame. By doing so, you can enhance data retrieval, indexing, and merging tasks. Syntax: DataFrame.set

Pandas DataFrame.reset_index()

In Pandas, reset_index() method is used to reset the index of a DataFrame. By default, it creates a new integer-based index starting from 0, making the DataFrame easier to work with in various scenarios, especially after performing operations like filtering, grouping or multi-level indexing. Example

Python | Pandas Dataframe.at[ ]

Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages and makes importing and analyzing data much easier. Pandas at[] is used to return data in a dataframe at the passed location. The passed l

Pandas DataFrame iterrows() Method

iterrows() method in Pandas is a simple way to iterate over rows of a DataFrame. It returns an iterator that yields each row as a tuple containing the index and the row data (as a Pandas Series). This method is often used in scenarios where row-wise operations or transformations are required. Exampl

Python | Pandas Series.iteritems()

Pandas series is a One-dimensional ndarray with axis labels. The labels need not be unique but must be a hashable type. The object supports both integer- and label-based indexing and provides a host of methods for performing operations involving the index. Pandas Series.iteritems() function iterates

Python | Pandas.to_datetime()

When a CSV file is imported and a Data Frame is made, the Date time objects in the file are read as a string object rather than a Date Time object Hence itâ€™s very tough to perform operations like Time difference on a string rather than a Date Time object. Pandas to_datetime() method helps to convert

Python | pandas.to_numeric method

Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric python packages. Pandas is one of those packages and makes importing and analyzing data much easier. pandas.to_numeric() is one of the general functions in Pandas which is used to convert