Pandas - Drop Infinite Values From DataFrame

By using replace() & dropna() methods you can remove infinite values from rows & columns in pandas DataFrame. Infinite values are represented in NumPy as np.inf & -np.inf for negative values. you get np with the statement import numpy as np.

In this article, I will explain how to drop/remove infinite values from Pandas DataFrame. In order to remove infinite values, you can either first replace infinite values with NaN and remove NaN from DataFrame or use pd.set_option('use_inf_as_na',True) to consider all infinite values as Nan.

Key Points –

• Infinite values (inf or -inf) can arise due to calculations like division by zero or overflow.
• Presence of infinite values can affect data analysis, visualizations, and calculations, leading to misleading results.
• Converting infinite values to NaN allows for easier handling and enables using standard dropna() functions.
• The pd.set_option('mode.use_inf_as_na', True) command can globally set infinite values to be treated as NaN.
• Infinite values can be dropped from rows or columns using filtering techniques or by setting them to NaN and using dropna().
• Infinite values can be handled in a single, chainable operation for efficient data cleaning.
• Handling infinite values by replacing them with NaN or dropping rows can impact the memory usage of the DataFrame, especially with large datasets.

<a id="h-1-create-a-pandas-dataframe-with-sample-data"></a>

1. Create a Pandas DataFrame With Sample Data

Let's create a DataFrame with a few rows and columns, execute some examples and validate the results. Our DataFrame contains column names Courses, Fee, Duration, and Discount with infinite values on all columns.

# Create DataFrame
import pandas as pd
import numpy as np
technologies = {
    'Courses':["Spark","PySpark","Hadoop","Python","pandas",np.inf,"Python",-np.inf],
    'Fee' :[22000,25000,23000,np.inf,26000,25000,-np.inf,24000],
    'Duration':['30days','50days','55days', '40days','60days',-np.inf,'55days',np.inf],
    'Discount':[1000,2300,1200,np.inf,2500,-np.inf,2000,1500]
                }
df = pd.DataFrame(technologies)
print("Create DataFrame:\n",df)

Yields below output.

<a id="h-2-pandas-drop-infinite-values"></a>

2. Pandas Drop Infinite Values

By using df.replace() function is used to replace infinite values with NaN, and then use the pandas.DataFrame.dropna() method to remove the rows with NaN, Null/None values. This eventually drops infinite values from pandas DataFrame. The inplace=True parameter modifies the original DataFrame in place.

# Replace infinite updated data with nan
df.replace([np.inf, -np.inf], np.nan, inplace=True)

# Drop rows with NaN
df.dropna(inplace=True)
print("DataFrame without infinite values:\n",df)

Yields below output. df.replace([np.inf, -np.inf], np.nan, inplace=True) replaces all np.inf & -np,inf values with NaN on the current DataFrame.

<a id="h-3-using-pandas-option-context-to-consider-infinite-as-nan"></a>

3. Using pandas.option.context() to Consider Infinite as NaN

You can use with pd.option_context('mode.use_inf_as_na',True): to consider all inf as Nan within a block of code. In python with is used to specify the scope of the block. IN case if you wanted to consider all inf as Nan in a complete program the use pd.set_option('use_inf_as_na',True).

In this example, option_context is used as a context manager to set the 'mode.use_inf_as_na' option to True temporarily, which means treating infinite values as NaN. The subsequent code within the with block will operate with this option setting.

Note: For older versions, replace use_inf_as_na with use_inf_as_null.

# Using option_context 
# To consider infinite as NaN temporarily
with pd.option_context('mode.use_inf_as_na', True):
    df = df.dropna()
print("DataFrame without infinite values:\n",df)

# Drop the rows with nan or infinite values
with pd.option_context('mode.use_inf_as_na', True):
  df.dropna(inplace=True)
print(df)

Yields the same output as above.

<a id="h-4-using-pandas-replace-amp-dropna-to-drop-infinite-values"></a>

4. Using pandas replace() & dropna() To Drop Infinite Values

Use df.replace() to replace entire infinite values with np.nan and use pd.DataFrame.dropna(axis=0). to drop rows. This ideally drops all infinite values from pandas DataFrame.

For instance, first uses replace() to replace infinite values with NaN, and then dropna() is applied to remove rows containing NaN. The resulting DataFrame (df) will not have rows containing infinite values.

# Replace to drop rows or columns infinite values
df = df.replace([np.inf, -np.inf], np.nan).dropna(axis=0)
print("DataFrame without infinite values:\n",df)

Yields the same output as above.

<a id="h-5-pandas-changing-option-to-consider-infinite-as-nan"></a>

5. Pandas Changing Option to Consider Infinite as NaN

You can do using pd.set_option() to pandas provided the option to use consider infinite as NaN. It makes the entire pandas module consider the infinite values as NaN. Use the pandas.DataFrame.dropna() method to drop the rows with infinite values.

# Changing option to consider infinite as nan
pd.set_option('mode.use_inf_as_na', True)
df.dropna(inplace=True)
print("DataFrame without infinite values:\n",df)

Yields the same output as above.

<a id="h-6-using-dataframe-isin-to-create-filter"></a>

6. Using DataFrame.isin() to Create Filter

Use filter df=df[~df_filter] to mask the infinite values. This version combines the filtering step (df[~df_filter]) with dropping NaN values (.dropna()), making the code more concise while achieving the same result. Adjust the code as needed for your specific use case.

# Using DataFrame.isin() to Create Filter
df_filter = df.isin([np.nan, np.inf, -np.inf])

# Mask df with the filter
df = df[~df_filter]
df.dropna(inplace=True)
print("DataFrame without infinite values:\n",df)

# Mask df with the filter and drop NaN values
df = df[~df_filter].dropna()
print("DataFrame without infinite values:\n", df)

Yields the same output as above.

<a id="h-7-select-non-null-rows-using-dataframe-replace"></a>

7. Select Non-Null Rows Using DataFrame.replace()

You can use df[df.replace([np.inf,-np.inf],np.nan).notnull().all(axis=1)] to replace infinite and -infinite with NaN, and then select non-null rows. axis set 1 to drop columns.

# Using replace method to select non-null rows
df = df[df.replace([np.inf, -np.inf], np.nan).notnull().all(axis=1)] 
print("DataFrame without infinite values:\n", df)

Yields the same output as above.

<a id="h-complete-example-of-pandas-drop-infinite-values"></a>

Complete Example of pandas Drop Infinite Values

import pandas as pd
import numpy as np
technologies = {
    'Courses':["Spark","PySpark","Hadoop","Python","pandas",np.inf,"Python",-np.inf],
    'Fee' :[22000,25000,23000,np.inf,26000,25000,-np.inf,24000],
    'Duration':['30day','50days','55days', '40days','60days',-np.inf,'55days',np.inf],
    'Discount':[1000,2300,1200,np.inf,2500,-np.inf,2000,1500]
                }
df = pd.DataFrame(technologies)
print(df)

# Replace infinite display updated data with nan
df.replace([np.inf, -np.inf], np.nan)
print(df)

# Replace infinite updated data with nan
df.replace([np.inf, -np.inf], np.nan, inplace=True)
# Drop rows with NaN
df.dropna(inplace=True)
print(df)

# Changing option context to use infinite as nan
with pd.option_context('mode.use_inf_as_na', True):
# Drop the rows with nan or infinite values
  df.dropna(inplace=True)
print(df)

# Replace to drop rows or columns infinite values
df = df.replace([np.inf, -np.inf], np.nan).dropna(axis=0)
print(df)

# Changing option to consider infinite as nan
pd.set_option('mode.use_inf_as_na', True)
df.dropna(inplace=True)
print(df)

# Using DataFrame.isin() to Create Filter
df_filter = df.isin([np.nan, np.inf, -np.inf])
# Mask df with the filter
df = df[~df_filter]
df.dropna(inplace=True)
print(df)

# Using replace method to select non-null rows
df = df[df.replace([np.inf, -np.inf], np.nan).notnull().all(axis=1)] 
print(df)

<a id="h-frequently-asked-questions-on-drop-infinite-values-from-dataframe"></a>

Frequently Asked Questions on Drop Infinite Values From DataFrame

To drop rows with infinite values from a pandas DataFrame, you can use a combination of replace() and dropna() functions.

You can use pd.option_context() to set the option 'mode.use_inf_as_na' to True temporarily. This will make operations treat infinite values as NaN within the specified context.

You can use DataFrame.isin() to create a filter for infinite values in a pandas DataFrame.

You can use DataFrame.isin() to create a filter for infinite values and then apply this filter to drop rows with infinite values.

<a id="h-conclusion"></a>

Conclusion

In this article, you have learned how to drop infinite values from pandas DataFrame using DataFrame.replace(), DataFrame.dropna(), and DataFrame.isin() method. Also, you have learned how to replace all infinite values with Nan or any specific values.

Happy Learning !!

<a id="h-related-articles"></a>

Related Articles

• Pandas Insert List into Cell of DataFrame
• Replace NaN with Blank/Empty String on Pandas
• How to Change Position of a Column in Pandas
• Install Python Pandas on Windows, Linux & Mac OS
• Pandas Correlation of Columns
• Pandas Join Explained With Examples
• How to Rename Columns With List in Pandas
• Pandas set_index() – Set Index to DataFrame
• Pandas Create DataFrame From Dict (Dictionary)
• Select Rows From List of Values in Pandas DataFrame
• Drop a Level from a Multi-Level Column Index in pandas
• How to Apply a Function to Two Columns on pandas DataFrame

<a id="h-references"></a>

References

• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html?