Built with Tutify.io

    AIStudyLabs

    Polars DataFrame fill_nan() Usage & Examples

    Polars DataFrame fill_nan() Usage & Examples

    In Polars, the fill_nan() function is used to replace NaN (Not a Number) values in a DataFrame with a specified value. It is specifically designed for handling missing or undefined numerical data in floating-point columns (f32 or f64). This function is particularly useful for managing incomplete or undefined numerical values in a dataset.

    In this article, I will explain the Polars DataFrame fill_nan() function, including its syntax, parameters, and usage, to demonstrate how we can create a new DataFrame with NaN values replaced by a specified value.

    Key Points –

    <a id="h-polars-dataframe-fill-nan-introduction"></a>

    Polars DataFrame fill_nan() Introduction

    Let’s know the syntax of the fill_nan() function.

    
# Syntax of fill_nan()
DataFrame.fill_nan(value: Expr | int | float | None) → DataFrame

    <a id="h-parameters-of-the-polars-dataframe-fill-nan"></a>

    Parameters of the Polars DataFrame fill_nan()

    Following are the parameters of the fill_nan() method.

    <a id="h-return-value"></a>

    Return Value

    This function returns a new DataFrame with NaN values replaced by the specified value.

    <a id="h-usage-of-polars-dataframe-fill-nan-function"></a>

    Usage of Polars DataFrame fill_nan() Function

    The fill_nan() function in Polars replaces all NaN (Not a Number) values in a DataFrame with a specified value. It is useful when dealing with missing numerical data, ensuring that computations and analyses are not affected by NaN values.

    To run some examples of the Polars DataFrame fill_nan() function, let’s create a Polars DataFrame.

    
import polars as pl

df = pl.DataFrame({
    "A": [1.5, 2, float("nan"), 4],
    "B": [0.6, float("nan"), 5, 12]
})

print("Original DataFrame:\n", df)

    Yields below output.

    If you want to replace NaN (Not a Number) values with a fixed value, you can use the fill_nan() method. This method is specifically designed to handle NaN values in floating-point columns.

    
# Replace NaN values with 0
fixed_value = 0
df2 = df.fill_nan(fixed_value)
print("DataFrame after replacing NaN:\n", df2)

# Replace NaN values with 0
df2 = df.fill_nan(0)
print("DataFrame after replacing NaN:\n", df2)

    Here,

    <a id="h-replace-nan-with-forward-fill-previous-value"></a>

    Replace NaN with Forward Fill (Previous Value)

    You can replace NaN values with the forward fill method, which propagates the last valid observation forward to fill NaN values. This is particularly useful for time series or sequential data where you want to carry the previous value forward to handle missing data.

    To achieve this, you can use the fill_null() method in combination with the forward_fill() strategy. Note that forward_fill() works with null values, so you'll first need to convert NaN values to null if your data contains NaN.

    
# Replace NaN with null
df = df.fill_nan(None)

# Apply forward fill
df2 = df.fill_null(strategy="forward")
print("DataFrame after Forward Fill:\n", df2)

# Output:
# DataFrame after Forward Fill:
# shape: (4, 2)
┌─────┬──────┐
│ A   ┆ B    │
│ --- ┆ ---  │
│ f64 ┆ f64  │
╞═════╪══════╡
│ 1.5 ┆ 0.6  │
│ 2.0 ┆ 0.6  │
│ 2.0 ┆ 5.0  │
│ 4.0 ┆ 12.0 │
└─────┴──────┘

    Here,

    <a id="h-replace-nan-with-backward-fill-next-value"></a>

    Replace NaN with Backward Fill (Next Value)

    You can replace NaN values with the backward fill method, which propagates the next valid observation backward to fill NaN values. This is useful when you want to handle missing data by carrying the next available value backward.

    
# Replace NaN with null
df = df.fill_nan(None)

# Apply backward fill
df2 = df.fill_null(strategy="backward")
print("DataFrame after backward fill:\n", df2)

# Output:
# DataFrame after backward fill:
# shape: (4, 2)
┌─────┬──────┐
│ A   ┆ B    │
│ --- ┆ ---  │
│ f64 ┆ f64  │
╞═════╪══════╡
│ 1.5 ┆ 0.6  │
│ 2.0 ┆ 5.0  │
│ 4.0 ┆ 5.0  │
│ 4.0 ┆ 12.0 │
└─────┴──────┘

    Here,

    <a id="h-replace-nan-with-a-custom-value"></a>

    Replace NaN with a Custom Value

    To replace NaN values with a custom value in a Polars DataFrame, you can use the fill_nan() function. This function allows you to specify the value that will replace all NaN values in the DataFrame.

    
# Custom value to replace NaN
custom_value = 99.99

# Replace NaN with the custom value
df2= df.fill_nan(custom_value)
print("DataFrame after replacing NaN:\n", df2)

# Output:
# DataFrame after replacing NaN:
# shape: (4, 2)
┌───────┬───────┐
│ A     ┆ B     │
│ ---   ┆ ---   │
│ f64   ┆ f64   │
╞═══════╪═══════╡
│ 1.5   ┆ 0.6   │
│ 2.0   ┆ 99.99 │
│ 99.99 ┆ 5.0   │
│ 4.0   ┆ 12.0  │
└───────┴───────┘

    Here,

    <a id="h-replace-nan-with-different-values-for-each-column"></a>

    Replace NaN with Different Values for Each Column

    You can replace NaN values with different values for each column by using the with_columns() method along with the fill_nan() function. This allows you to specify a custom replacement value for each column individually.

    
# Replace NaN with different values per column
df2 = df.with_columns([
    pl.col("A").fill_nan(10),  
    pl.col("B").fill_nan(20)   
])
print("DataFrame after replacing NaN with different values:\n", df2)

# Output:
# DataFrame after replacing NaN with different values:
# shape: (4, 2)
┌──────┬──────┐
│ A    ┆ B    │
│ ---  ┆ ---  │
│ f64  ┆ f64  │
╞══════╪══════╡
│ 1.5  ┆ 0.6  │
│ 2.0  ┆ 20.0 │
│ 10.0 ┆ 5.0  │
│ 4.0  ┆ 12.0 │
└──────┴──────┘

    Here,

    <a id="h-conclusion"></a>

    Conclusion

    In summary, the fill_nan() function in Polars is an essential tool for handling missing (NaN) values efficiently. By offering multiple replacement strategies, it simplifies data cleaning and ensures data integrity, making it a valuable feature when working with floating-point data.

    Happy Learning!!

    <a id="h-related-articles"></a>

    Related Articles

    <a id="h-references"></a>

    References