Removing Null Values on Selected Columns only in Polars DataFrame

In Polars, removing null values from selected columns means you're filtering the DataFrame by removing rows that have null (missing) values in specific columns, while leaving other columns intact, even if they contain nulls. This approach allows you to drop rows only if certain columns have null values, without affecting rows where other columns might have nulls. In this article, I will explain how to remove null values from specific columns in a Polars DataFrame.

Key Points–

Use filter() with is_not_null() to retain rows with non-null values in a specific column.
Chain multiple is_not_null() conditions using the & (AND) operator to filter rows where multiple selected columns are non-null.
Logical operators like | (OR) can be used if you want to retain rows where at least one column is non-null.
is_not_null() works on all data types and is the standard way to check for missing values in Polars.
Filtering nulls in specific columns improves data quality without over-cleaning.
It's a flexible way to handle missing values when preparing data for analysis or modeling.

Usage of Removing Null Values on Selected Columns

To remove rows with null values from a specific column in Polars, you can use the filter() function combined with the is_not_null() condition for that column. This will filter out rows where the specified column contains null values.

To run some examples of removing null values on selected columns only in Polars DataFrame, let’s create a Polars DataFrame.


import polars as pl

technologies= {
    'Courses':["Spark","PySpark",None,"Python","Pandas"],
    'Fees' :[22000,None,23000,24000,None],
    'Duration':['30days','50days','35days', '40days',None],
    'Discount':[None,2300,1000,None,2500]
          }
df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To drop rows with null values in a single column in Polars, you can use the is_not_null() function to filter out the rows where that specific column contains None or null values.


# Drop rows where 'Fees' is null
df2 = df.filter(pl.col("Fees").is_not_null())
print("DataFrame after dropping rows with nulls in 'Fees':\n", df2)

Here,

pl.col("Fees").is_not_null() filters the rows where the "Fees" column does not have null values.
This means only rows where "Fees" has a value will be kept, and the rest (with null in "Fees") will be dropped.

Drop Rows with Nulls in Multiple Columns

To drop rows with nulls in multiple specific columns in Polars, apply is_not_null() to each column and combine the conditions using the logical AND operator (&).


# Drop rows where 'Fees' or 'Discount' is null
df2 = df.filter(
    pl.col("Fees").is_not_null() & pl.col("Discount").is_not_null()
)
print("DataFrame after dropping rows with nulls in 'Fees' and 'Discount':\n", df2)

# Output:
# DataFrame after dropping rows with nulls in 'Fees' and 'Discount':
# shape: (1, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ null    ┆ 23000 ┆ 35days   ┆ 1000     │
└─────────┴───────┴──────────┴──────────┘

Here,

pl.col("Fees").is_not_null() keeps rows where "Fees" is not null
pl.col("Discount").is_not_null() keeps rows where "Discount" is not null
Combined with &, this drops rows where either is null

Remove Rows where "Courses" is Null

To remove rows where "Courses" is null in a Polars DataFrame, you can simply filter the DataFrame using the is_not_null() condition on the "Courses" column.


# Remove rows where "Courses" is null
df2 = df.filter(pl.col("Courses").is_not_null())
print(df2)

# Output:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ null     │
│ PySpark ┆ null  ┆ 50days   ┆ 2300     │
│ Python  ┆ 24000 ┆ 40days   ┆ null     │
│ Pandas  ┆ null  ┆ null     ┆ 2500     │
└─────────┴───────┴──────────┴──────────┘

Now, only rows where "Courses" is not null are kept in the DataFrame.

Remove Rows where "Duration" or "Discount" is Null

To remove rows where either "Duration" or "Discount" is null in a Polars DataFrame, you can use the filter() function combined with the is_not_null() condition for both columns and the logical AND operator (&).


# Remove rows where "Duration" or "Discount" is null
df2 = df.filter(
    pl.col("Duration").is_not_null() & pl.col("Discount").is_not_null()
)
print(df2)

# Output:
# shape: (2, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ PySpark ┆ null  ┆ 50days   ┆ 2300     │
│ null    ┆ 23000 ┆ 35days   ┆ 1000     │
└─────────┴───────┴──────────┴──────────┘

In this case, only rows where both "Duration" and "Discount" are non-null will remain.

Use Reduce to Dynamically Apply AND Condition

To dynamically apply an AND condition for dropping rows with null values across multiple columns using reduce, you can combine the logical conditions for each column using the reduce function from the functools module.


import polars as pl
from functools import reduce

# Sample data
technologies = {
    'Courses': ["Spark", "PySpark", None, "Python", "Pandas"],
    'Fees': [22000, None, 23000, 24000, None],
    'Duration': ['30days', '50days', '35days', '40days', None],
    'Discount': [None, 2300, 1000, None, 2500]
}

df = pl.DataFrame(technologies)

# Selected columns to check
selected_columns = ["Fees", "Discount", "Duration"]

# Using reduce to dynamically apply AND condition
df2 = df.filter(
    reduce(
        lambda acc, col: acc & pl.col(col).is_not_null(), 
        selected_columns, 
        pl.lit(True)
    )
)
print("DataFrame after dropping rows where any of the selected columns has nulls:\n", df2)

# Output:
# DataFrame after dropping rows where any of the selected columns has nulls:
# shape: (1, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ null    ┆ 23000 ┆ 35days   ┆ 1000     │
└─────────┴───────┴──────────┴──────────┘

Here,

reduce iterates over the selected columns list.
lambda acc, col: acc & pl.col(col).is_not_null() applies the AND condition across all columns dynamically.
- acc is the accumulator (starting as pl.lit(True), meaning "True" for every row).
- pl.col(col).is_not_null() checks if the current column’s values are not null.
The final result filters rows where none of the selected columns have null values.

Conclusion

In conclusion, managing null values in specific columns is simple and efficient with the filter() function and the is_not_null() condition. This approach lets you selectively eliminate rows with nulls in one or more columns, offering flexibility for data cleaning tasks.

Happy Learning!!

References

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.explode.html

Removing Null Values on Selected Columns only in Polars DataFrame

Usage of Removing Null Values on Selected Columns

Drop Rows with Nulls in Multiple Columns

Remove Rows where "Courses" is Null

Remove Rows where "Duration" or "Discount" is Null

Use Reduce to Dynamically Apply AND Condition

Conclusion

Related Articles

References