Get Unique Rows in Pandas DataFrame

You can use the drop_duplicates() function to remove duplicate rows and get unique rows from a Pandas DataFrame. This method duplicates rows based on column values and returns unique rows. If you want to get duplicate rows from Pandas DataFrame you can use DataFrame.duplicated() function.

In this article, I will explain how to get unique rows from DataFrame using the drop duplicates() method and how you can also get unique rows based on specified columns.

Key Points –

• The drop_duplicates() method can be used to retrieve unique rows in a DataFrame by removing duplicates.
• By default, drop_duplicates() retains the first occurrence of each unique row in the DataFrame.
• The subset parameter specifies particular columns to identify unique rows, allowing the uniqueness check to focus on a subset of columns.
• Setting keep=False with drop_duplicates() removes all duplicate rows, leaving only completely unique rows in the result.
• When working with large DataFrames, specifying columns in the subset parameter and using keep=False can improve performance in retrieving unique rows.
• By default, retrieving unique rows does not reset the index, which can be reset manually if necessary.

<a id="h-quick-examples-of-get-unique-rows-in-pandas"></a>

Quick Examples of Get Unique Rows in Pandas

If you are in a hurry, below are some quick examples of getting unique rows in Pandas.

# Quick examples of get unique rows

# Example 1: Use drop_duplicates()
# Get unique row values 
df1 = df.drop_duplicates()

# Example 2: Set default param Keep = first
# Get the unique rows
df1 = df.drop_duplicates(keep='first')

# Example 3: Set keep = last duplicate row & 
# Get unique row
df1 = df.drop_duplicates(keep='last')

# Example 4: Set keep param as False & 
# Get unique rows 
df1 = df.drop_duplicates(keep=False)

# Example 5: Get unique rows based on specified columns 
df1 = df.drop_duplicates(subset=["Courses", "Fee"], keep=False)

<a id="h-usage-of-get-unique-rows-in-pandas-dataframe"></a>

Usage of Get Unique Rows in Pandas DataFrame

To get unique rows in a Pandas DataFrame, you can use the drop_duplicates() method. This method removes duplicate rows and keeps only the first occurrence (by default).

Now, let's create a DataFrame with duplicate values, execute these examples, and validate the results. Our DataFrame contains column names Courses, Fee, Duration, and Discount.

# Create DataFrame
import pandas as pd
import numpy as np
technologies = {
    'Courses':["Spark","PySpark","Python","pandas","Python","Spark","pandas"],
    'Fee' :[20000,25000,22000,30000,22000,20000,30000],
    'Duration':['30days','40days','35days','50days','40days','30days','50days'],
    'Discount':[1000,2300,1200,2000,2300,1000,2000]
              }
df = pd.DataFrame(technologies)
print("Create DataFrame:\n", df)

Yields below output.

<a id="h-get-unique-rows-based-on-all-columns"></a>

Get Unique Rows based on All Columns

Use DataFrame.drop_duplicates() without any arguments todrop rowswith the same values matching on all columns. It takes default values subset=None and keep=‘first’. By running this function on the above DataFrame, it returns four unique rows after removing duplicate rows.

# Use drop_duplicates() to get
# Unique row values 
df1 = df.drop_duplicates()
print("Get unique rows from the DataFrame:/n", df1)

# Set default param Keep = first
# Get the unique rows
df1 = df.drop_duplicates(keep='first')
print("Get unique rows from the DataFrame:\n", df1)

Yields below output.

<a id="h-set-keep-param-as-last-amp-get-the-unique-rows"></a>

Set Keep Param as Last & Get the Unique Rows

As we know from the above this function by default keeps the first duplicate. However, we can also keep the last duplicate by specifying the keep param as last.

# Set keep param last & get unique rows
df1 = df.drop_duplicates( keep='last')
print("Get unique rows from the DataFrame:\n", df1)

Yields below output.

# Output:
# Get unique rows from the DataFrame:
  Courses    Fee Duration  Discount
1  PySpark  25000   40days      2300
2   Python  22000   35days      1200
4   Python  22000   40days      2300
5    Spark  20000   30days      1000
6   pandas  30000   50days      2000

<a id="h-set-keep-param-as-false-amp-get-the-pandas-unique-rows"></a>

Set Keep Param as False & Get the Pandas Unique Rows

When we pass 'keep=False' to the drop_duplicates() function it, will remove all the duplicate rows from the DataFrame and return unique rows. Let's use this df.drop_duplicates(keep=False) syntax and get the unique rows of the given DataFrame.

# Set keep param as False & get unique rows 
df1 = df.drop_duplicates(keep=False)
print("Get unique rows from the DataFrame:\n", df1)

# Output:
# Get unique rows from the DataFrame:   
#    Courses    Fee Duration  Discount
# 1  PySpark  25000   40days      2300
# 2   Python  22000   35days      1200
# 4   Python  22000   40days      2300

<a id="h-get-pandas-unique-rows-based-on-specified-columns"></a>

Get Pandas Unique Rows based on Specified Columns

We can also get unique rows based on specified columns by setting 'keep=False' and specifying the columns in the drop_duplicates() function, it will return the unique rows of specified columns.

# Get unique rows based on specified columns 
df1 = df.drop_duplicates(subset=["Courses", "Fee"], keep=False)
print("Get unique rows from the DataFrame:\n", df1)

# Output:
# Get unique rows from the DataFrame:
#   Courses    Fee Duration  Discount
# 1  PySpark  25000   40days      2300

<a id="h-faq-on-get-unique-rows-in-pandas-dataframe"></a>

FAQ on Get Unique Rows in Pandas DataFrame

Use the drop_duplicates() method to get unique rows in a DataFrame. This method removes duplicate rows and returns only the unique ones.

You can pass a list of column names to the subset parameter in drop_duplicates() to consider only those columns when identifying unique rows.

The keep parameter in drop_duplicates() allows you to specify whether to keep the first (keep='first') or last (keep='last') occurrence of a duplicate row. The default is keep='first'.

Without parameters, drop_duplicates() considers all columns in the DataFrame and removes any rows that are duplicates across all columns.

If you want to remove duplicate rows based on the index, use the drop_duplicates() method while setting the subset parameter to the index, or by using the df.index directly.

Use the duplicated() method, which returns a boolean Series indicating whether a row is a duplicate of a previous row.

<a id="h-conclusion"></a>

Conclusion

In this article, you have learned to get unique rows from Pandas DataFrame using the drop_duplicates() function with multiple examples. Also, I explained the usage of drop_duplicates() and how to use various parameters.

Happy learning!!

<a id="h-related-articles"></a>

Related Articles

• Pandas Get List of All Duplicate Rows
• Drop DataFrame rows by index
• Get row number in Pandas DataFrame
• How to get first N rows of DataFrame?
• How to get first row of DataFrame Pandas?
• How to get last row of Pandas DataFrame?
• How to append DataFrames using for loop?
• How to add/insert row to Pandas DataFrame?
• You can drop duplicate columns from DataFrame
• How to Drop Multiple Columns by Index in Pandas

<a id="h-reference"></a>

Reference

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html