Replace Value by Null in Polars

In Polars, you can replace specific values with null by using the when-then-otherwise expression within a with_columns() or select() operation. replacing a value with null means detecting specific values in a DataFrame column, such as "N/A", -1, or 0, and substituting them with missing values, represented as null (or None in Python). This is similar to using NaN for missing data in pandas. In this article, I will explain how the replace a value by null in polars.

Key Points –

Replacing values with null means marking specific data entries as missing or invalid.
Nulls represent missing or undefined data in Polars DataFrames.
Common use cases include replacing placeholders like "unknown", "N/A", or invalid numeric values.
Polars does not automatically interpret placeholders as null; explicit replacement is required.
The when().then().otherwise() expression is the most idiomatic way to replace values with null.
The with_columns() method is used to apply these conditional replacements to a DataFrame.

Usage of Replace Value by Null in Polars

To replace a specific value with null in a Polars column, you can use the pl.when().then().otherwise() expression inside the with_columns() method. This allows you to conditionally set values to null (i.e., None in Python).

To run some examples of replacing values by null in polars, let’s create a Polars DataFrame.


import polars as pl

# Creating a dictionary with course information
technologies = {
    'Courses': ["Spark", "Pandas", "unknown", "Python"],
    'Fees': [30000, -40000, 35000, -60000]
}

# Creating a Polars DataFrame from the dictionary
df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To replace a specific number with null in Polars, you can use a conditional expression with when().then().otherwise() or the apply() method, though the most idiomatic and recommended approach is using when().


# Replace a specific number with null
df = df.with_columns(
    pl.when(pl.col("Fees") == -40000)
      .then(None)
      .otherwise(pl.col("Fees"))
      .alias("Fees")
)
print(df)

In the above example, this replaces all occurrences of -40000 with null in the Fees column.

Replace a Specific String with Null

To replace a specific string with null in a Polars DataFrame column, use the with_columns() method along with a conditional expression using when().then().otherwise().


# Replace a specific string with null
df2 = df.with_columns(
    pl.when(pl.col("Courses") == "unknown")
      .then(None)
      .otherwise(pl.col("Courses"))
      .alias("Courses")
)
print(df2)

# Output:
# shape: (4, 2)
┌─────────┬────────┐
│ Courses ┆ Fees   │
│ ---     ┆ ---    │
│ str     ┆ i64    │
╞═════════╪════════╡
│ Spark   ┆ 30000  │
│ Pandas  ┆ -40000 │
│ null    ┆ 35000  │
│ Python  ┆ -60000 │
└─────────┴────────┘

In the above example, this replaces "unknown" with null in the Courses column while keeping other values intact.

Replace Multiple Values with Null Using a List

You can replace multiple values in a Polars DataFrame with null (i.e., None in Python) by using the with_columns() method combined with either apply() or when().then().otherwise() expressions. For example, you can replace all occurrences of "unknown" in the "Courses" column and all negative values in the "Fees" column with null.


# List of values to replace with null in Courses
values_to_nullify = ["unknown"]

df2 = df.with_columns([
    # Replace listed strings in 'Courses' with None
    pl.when(pl.col("Courses").is_in(values_to_nullify))
      .then(None)
      .otherwise(pl.col("Courses"))
      .alias("Courses"),

    # Replace negative 'Fees' with None
    pl.when(pl.col("Fees") < 0)
      .then(None)
      .otherwise(pl.col("Fees"))
      .alias("Fees"),
])
print(df2)

# Output:
# shape: (4, 2)
┌─────────┬───────┐
│ Courses ┆ Fees  │
│ ---     ┆ ---   │
│ str     ┆ i64   │
╞═════════╪═══════╡
│ Spark   ┆ 30000 │
│ Pandas  ┆ null  │
│ null    ┆ 35000 │
│ Python  ┆ null  │
└─────────┴───────┘

In the above example, this replaces "unknown" in Courses with null and all negative values in Fees with null.

Replace Null-Looking Placeholders Like "N/A"

To replace placeholder strings like "N/A" with null in Polars, you can use the when().then().otherwise() approach, or is_in() if you're dealing with multiple placeholder values.


import polars as pl

technologies = {
    'Courses': ["Spark", "Pandas", "N/A", "Python"],
    'Fees': [30000, -40000, 35000, -60000]
}

df = pl.DataFrame(technologies)

# Replace null-looking placeholders like "N/A"
df2 = df.with_columns(
    pl.when(pl.col("Courses") == "N/A")
      .then(None)
      .otherwise(pl.col("Courses"))
      .alias("Courses")
)
print(df2)

# Output:
# shape: (4, 2)
┌────────┬────────┐
│ Courses│ Fees   │
│ ---    │ ---    │
│ str    │ i64    │
╞════════╪════════╡
│ Spark  │ 30000  │
│ Pandas │ -40000 │
│ null   │ 35000  │
│ Python │ -60000 │
└────────┴────────┘

Replace 0 Values in a Column with Null

To replace all 0 values in a specific column with null in Polars, you can use a conditional expression with when().then().otherwise().


import polars as pl

technologies = {
    'Courses': ["Spark", "Pandas", "unknown", "Python"],
    'Fees': [30000, 0, 35000, -60000]
}

df = pl.DataFrame(technologies)

# Replace 0 values in a column with null
df2 = df.with_columns(
    pl.when(pl.col("Fees") == 0)
      .then(None)
      .otherwise(pl.col("Fees"))
      .alias("Fees")
)
print(df2)

# Output:
# shape: (4, 2)
┌─────────┬────────┐
│ Courses ┆ Fees   │
│ ---     ┆ ---    │
│ str     ┆ i64    │
╞═════════╪════════╡
│ Spark   ┆ 30000  │
│ Pandas  ┆ null   │
│ unknown ┆ 35000  │
│ Python  ┆ -60000 │
└─────────┴────────┘

Conclusion

In conclusion, replacing values with null in Polars is a common and powerful data cleaning technique. Whether you're handling placeholder strings, invalid numbers, or custom conditions, the most idiomatic way to do this is using the when().then().otherwise() expression inside the with_columns() method.

Happy Learning!!

References

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.filter.html