Polars DataFrame fill_null() Usage & Examples

In Polars, the fill_null() method is used to replace None (null) values in a DataFrame or Series with a specified value or strategy. Properly handling missing values leads to cleaner and more reliable data for analysis. This function is similar to fillna() in pandas.

In this article, I will discuss the fill_null() function in Polars DataFrame, covering its syntax, parameters, and usage. I will also demonstrate how to replace None values in a DataFrame using different filling strategies.

Key Points –

• The fill_null() method is used to replace null (missing) values in a Polars DataFrame with a specified value or a computed strategy.
• It can replace null values with a fixed scalar, a dictionary for column-specific replacements, or an expression.
• Supports multiple strategies like "forward", "backward", "min", "max", "mean", "zero", and "one" to compute replacement values dynamically.
• A dictionary can be passed to fill different columns with different values.
• fill_null() returns a new DataFrame and does not modify the original DataFrame in place.
• Polars also provides fill_nan() for handling NaN values specifically and fill_null(strategy="forward") for propagating values.
• The limit argument restricts the number of consecutive null values that can be replaced.
• By default, matches_supertype=True allows filling nulls with values that may not exactly match the column's type.

<a id="h-polars-dataframe-fill-null-introduction"></a>

Polars DataFrame fill_null() Introduction

Let’s know the syntax of the fill_null() method.

# Syntax of fill_null()
DataFrame.fill_null(
value: Any | Expr | None = None,
strategy: FillNullStrategy | None = None,
limit: int | None = None,
*,
matches_supertype: bool = True,
) → DataFrame

<a id="h-parameters-of-the-dataframe-fill-null"></a>

Parameters of the DataFrame fill_null()

Following are the parameters of the fill_null() method.

• value (Any | Expr | None, default=None)
  • A single scalar value, dictionary (for column-specific replacements), or an expression to replace null values.
  • Example: df.fill_null(0) replaces all null values with 0.
• strategy (FillNullStrategy | None, default=None)
  • Alternative to value, this allows filling null values using predefined strategies:
    • "forward" - Fill with the previous non-null value.
    • "backward" - Fill with the next non-null value.
    • "min", "max", "mean", "zero", "one" - Fill with the respective computed values.
• limit (int | None, default=None)
  • Limits how many consecutive null values should be filled.
• matches_supertype (bool, default=True)
  • If True, allows filling values even when they don't exactly match the column type.

<a id="h-return-value"></a>

Return Value

This function returns a DataFrame where None values are replaced based on the specified filling strategy.

<a id="h-usage-of-polars-dataframe-fill-null-method"></a>

Usage of Polars DataFrame fill_null() Method

The fill_null() method in Polars is used to replace null (missing) values in a DataFrame or Series with a specified value or a computed strategy.

First, let’s create a Polars DataFrame.

import polars as pl

df = pl.DataFrame(({
     'Courses':["Spark",'Java',"Scala",'Python'],
     'Fee' :[20000,None,26000,24000],
     'Duration':['30days','40days', None,'40days'],
     'Discount':[1000,None,2500,None]
               }))
print("Original DataFrame:\n", df)

Yields below output.

You can use the fill_null() method in Polars to replace null (missing) values with a fixed value.

# Replace nulls with a fixed value
df2 = df.fill_null(99)
print("DataFrame after replacing nulls:\n", df2)

Here,

• fill_null(99) replaces all None values in the entire DataFrame with 99.
• You can replace 99 with any other value as needed.

You can also replace nulls in specific columns using the fill_null() method on individual columns.

# Replace nulls in the 'Fee' column with 0 and in 'Discount' column with 500
df2 = df.with_columns(
    df["Fee"].fill_null(0),
    df["Discount"].fill_null(500)
)
print("DataFrame after replacing nulls in specific columns:\n", df2)

# Output:
# DataFrame after replacing nulls in specific columns:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fee   ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 20000 ┆ 30days   ┆ 1000     │
│ Java    ┆ 0     ┆ 40days   ┆ 500      │
│ Scala   ┆ 26000 ┆ null     ┆ 2500     │
│ Python  ┆ 24000 ┆ 40days   ┆ 500      │
└─────────┴───────┴──────────┴──────────┘

Here,

• Only the Fee and Discount columns had null values replaced.
• The Duration column remains unchanged since it wasn’t specified.

<a id="h-forward-fill-forward-propagate-previous-values"></a>

Forward Fill ("forward") - Propagate Previous Values

You can use the fill_null(strategy="forward") method in Polars to fill None values by propagating the last non-null value down the column, also known as forward filling.

# Forward fill missing values
df2 = df.fill_null(strategy="forward")
print("DataFrame after forward fill:\n", df2)

# Output:
# DataFrame after forward fill:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fee   ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 20000 ┆ 30days   ┆ 1000     │
│ Java    ┆ 20000 ┆ 40days   ┆ 1000     │
│ Scala   ┆ 26000 ┆ 40days   ┆ 2500     │
│ Python  ┆ 24000 ┆ 40days   ┆ 2500     │
└─────────┴───────┴──────────┴──────────┘

Here,

• fill_null(strategy="forward") replaces None values with the previous non-null value in the same column.
• If a column starts with None, it remains unchanged since there's no previous value to propagate.

<a id="h-backward-fill-backward-use-next-available-value"></a>

Backward Fill ("backward") - Use Next Available Value

You can use the fill_null(strategy="backward") method in Polars to replace None values with the next available value in the same column (also known as backward filling).

# Backward fill missing values
df2 = df.fill_null(strategy="backward")
print("DataFrame after backward fill:\n", df2)

# Output:
# DataFrame after backward fill:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fee   ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 20000 ┆ 30days   ┆ 1000     │
│ Java    ┆ 26000 ┆ 40days   ┆ 2500     │
│ Scala   ┆ 26000 ┆ 40days   ┆ 2500     │
│ Python  ┆ 24000 ┆ 40days   ┆ null     │
└─────────┴───────┴──────────┴──────────┘

Here,

• fill_null(strategy="backward") replaces None values with the next non-null value in the same column.
• If a column ends with None, it remains unchanged since there's no next value to propagate.

<a id="h-replace-null-values-using-min-or-max-strategy"></a>

Replace Null Values Using "min" or "max" Strategy

You can use the fill_null(strategy="min") or fill_null(strategy="max") method in Polars to replace None values with the minimum or maximum value of the respective column. Alternatively, the fill_null() method allows you to handle null values by applying the "min" or "max" strategy in a Polars DataFrame.

# Replace null values with column-wise minimum values
df2 = df.fill_null(strategy="min")
print("DataFrame after filling nulls with min values:\n", df2)

# Output:
# DataFrame after filling nulls with min values:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fee   ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 20000 ┆ 30days   ┆ 1000     │
│ Java    ┆ 20000 ┆ 40days   ┆ 1000     │
│ Scala   ┆ 26000 ┆ 30days   ┆ 2500     │
│ Python  ┆ 24000 ┆ 40days   ┆ 1000     │
└─────────┴───────┴──────────┴──────────┘

# Replace null values with column-wise maximum values
df2 = df.fill_null(strategy="max")
print("\nDataFrame after filling nulls with max values:\n", df2)

# Output:
# DataFrame after filling nulls with max values:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fee   ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 20000 ┆ 30days   ┆ 1000     │
│ Java    ┆ 26000 ┆ 40days   ┆ 2500     │
│ Scala   ┆ 26000 ┆ 40days   ┆ 2500     │
│ Python  ┆ 24000 ┆ 40days   ┆ 2500     │
└─────────┴───────┴──────────┴──────────┘

Here,

• fill_null(strategy="min") replaces None values with the minimum non-null value in the column.
• fill_null(strategy="max") replaces None values with the maximum non-null value in the column.

<a id="h-column-wise-replacement-using-a-dictionary"></a>

Column-Wise Replacement Using a Dictionary

You can use the fill_null() method with a dictionary to replace None values with specific values per column in Polars.

# Define column-wise replacement values
fill_values = {
    "Fee": 15000,        # Replace nulls in 'Fee' with 15000
    "Duration": "Unknown", # Replace nulls in 'Duration' with 'Unknown'
    "Discount": 700      # Replace nulls in 'Discount' with 700
}

# Replace null values using a dictionary
df_filled = df.with_columns([
    df[col].fill_null(value) for col, value in fill_values.items()
])
print("DataFrame after column-wise filling:\n", df_filled)

# Output:
# DataFrame after column-wise filling:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fee   ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 20000 ┆ 30days   ┆ 1000     │
│ Java    ┆ 15000 ┆ 40days   ┆ 700      │
│ Scala   ┆ 26000 ┆ Unknown  ┆ 2500     │
│ Python  ┆ 24000 ┆ 40days   ┆ 700      │
└─────────┴───────┴──────────┴──────────┘

Here,

• The fill_values dictionary specifies different replacement values for each column.
• The with_columns([…]) method applies fill_null(value) to each column in the dictionary.

<a id="h-conclusion"></a>

Conclusion

In summary, the fill_null() method in Polars is a versatile solution for managing missing values in a DataFrame. It enables you to replace nulls with a fixed value, propagate values using forward or backward fill, or apply strategies such as "min", "max", or "mean".

Happy Learning!!

<a id="h-related-articles"></a>

Related Articles

• Polars DataFrame head() Function
• Polars DataFrame sample() Method
• Polars DataFrame quantile() Method
• Polars DataFrame max() Method
• Polars Cast String to Integer
• Convert Polars Cast Int to String
• Convert Polars Cast String to Float
• Convert Polars Cast Float to String
• Polars DataFrame shift() Usage & Examples
• Polars DataFrame schema() Usage & Examples

<a id="h-references"></a>

References

• https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.fill_nan.html