Polars DataFrame Drop Nans

In polars, the drop_nans() method is used to remove rows from a DataFrame that contain NaN (Not a Number) values. More precisely, dropping NaNs, or nulls, refers to eliminating rows with missing data to ensure a cleaner and more consistent dataset.

In this article, I will explain the Polars DataFrame drop_nans() function, detailing its syntax, parameters, and how to use it to create a new DataFrame by removing rows that contain NaN values in specified columns.

Key Points –

The drop_nulls() method removes rows containing null (NaN) values from a Polars DataFrame.
By default, it drops rows where any column has a null value.
You can specify columns using the subset parameter to drop rows based on nulls in specific columns only.
drop_nans() does not remove null values use drop_nulls() for that.
It returns a new DataFrame with rows containing NaNs removed.
drop_nulls() handles mixed data types (strings, integers, floats) seamlessly.
drop_nulls() returns a new DataFrame; it does not modify the original in place.
Boolean filtering with filter() and is_null() / is_not_null() gives precise control over null removal.

Polars DataFrame drop_nans() Introduction

Let’s know the syntax of the drop_nans() function


# Syntax of polars DataFrame drop_nans()
DataFrame.drop_nans(
subset: ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None = None,
) → DataFrame

Parameters of the Polars DataFrame drop_nans()

Following are the parameters of the DataFrame drop_nans() method.

subset (optional) - Can be a single column name, a collection of column names, or a column selector. Specifies which column(s) to check for NaN values. If None (default), all columns are checked.

Return Value

This function returns a new DataFrame with rows containing NaNs dropped based on the subset criteria.

Usage of Polars DataFrame drop_nans() Method

The drop_nulls() method in Polars removes rows from a DataFrame that contain null (missing) values. By default, it drops rows with nulls in any column, but you can also specify specific columns to check for nulls.

First, let’s create a Polars DataFrame.


import polars as pl

technologies = {
    'Courses': ["Spark", "PySpark", None, "Python", "Pandas"],
    'Fees': [20000, None, 25000, 30000, None],
}

df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To remove all rows containing null values in any column, simply use the drop_nulls() method without any arguments. This will return a DataFrame with only the rows that have complete (non-null) data.


# Drop rows with any nulls in any column
df2 = df.drop_nulls()
print("DataFrame after dropping rows with any nulls:\n", df2)

Here,

Rows 2, 3, and 5 were dropped because they contained null values.
drop_nulls() drops rows where any column is null.

Drop Rows with NaNs only in Specific Columns

To drop rows that have null (NaN) values in specific columns of a Polars DataFrame, you can use the subset parameter of the drop_nulls() method. Simply provide a list of the column names you want to check, and only rows with nulls in those columns will be removed.


# Drop rows with nulls in specific columns, e.g., 'Courses' only
df2 = df.drop_nulls(subset=['Courses'])
print("DataFrame after dropping rows with nulls in 'Courses' column:\n", df2)

# Drop rows with nans only in specific columns
df2 = df.drop_nulls(["Courses"])
print("DataFrame after dropping rows with nulls in 'Courses' column:\n", df2)

# Output:
#  DataFrame after dropping rows with nulls in 'Courses' column:
# shape: (4, 2)
┌─────────┬───────┐
│ Courses ┆ Fees  │
│ ---     ┆ ---   │
│ str     ┆ i64   │
╞═════════╪═══════╡
│ Spark   ┆ 20000 │
│ PySpark ┆ null  │
│ Python  ┆ 30000 │
│ Pandas  ┆ null  │
└─────────┴───────┘

Here,

subset=['Courses'] tells Polars to look for nulls only in the Courses column.
Rows with nulls in other columns (like Fees) will not be dropped.

Drop Rows with NaNs in a Single Column

To remove rows containing null values in a specific column, pass that column name inside a single-item list to the drop_nulls() method.


# Drop rows with nans in a single column
df2 = df.drop_nulls(["Fees"])
print("DataFrame after dropping rows with nulls in 'Fees':\n", df2)

# Output:
# DataFrame after dropping rows with nulls in 'Fees':
# shape: (3, 2)
┌─────────┬───────┐
│ Courses ┆ Fees  │
│ ---     ┆ ---   │
│ str     ┆ i64   │
╞═════════╪═══════╡
│ Spark   ┆ 20000 │
│ null    ┆ 25000 │
│ Python  ┆ 30000 │
└─────────┴───────┘

Here,

Drops only rows where the Fees column has a null value.
Keeps rows where other columns may have nulls, as long as Fees is not null.

Drop Rows with NaNs Using a Condition

To drop rows with NaNs based on a condition in Polars, use boolean filtering with the filter() method alongside is_not_null() method. This allows you to keep only the rows where the specified values are present.


# Drop rows with nans using a condition
df2 = df.filter(df["Fees"].is_not_null())
print("DataFrame after dropping rows where 'Fees' is null:\n", df2)

# Output:
# DataFrame after dropping rows where 'Fees' is null:
# shape: (3, 2)
┌─────────┬───────┐
│ Courses ┆ Fees  │
│ ---     ┆ ---   │
│ str     ┆ i64   │
╞═════════╪═══════╡
│ Spark   ┆ 20000 │
│ null    ┆ 25000 │
│ Python  ┆ 30000 │
└─────────┴───────┘

Similarly, to drop rows with NaNs (nulls) based on a condition in Polars, you can use the filter() method along with logical expressions like is_null() or is_not_null().


# Custom condition: Drop rows where 'Courses' is null and 'Fees' > 20000
df2 = df.filter(~(df["Courses"].is_null() & (df["Fees"] > 20000)))
print("DataFrame after conditional null filtering:\n", df2)

# Output:
# DataFrame after conditional null filtering:
# shape: (4, 2)
┌────────┬───────┐
│ Courses│ Fees  │
│ ---    │ ---   │
│ str    │ i64   │
╞════════╪═══════╡
│ Spark  │ 20000 │
│ PySpark│ null  │
│ Python │ 30000 │
│ Pandas │ null  │
└────────┴───────┘

Here,

The row with Courses = None and Fees = 25000 was dropped.
Other rows with just one null do remain, because the condition was specific.

Drop Rows with NaNs from a DataFrame with Multiple Data Types

To drop rows with null values from a Polars DataFrame containing multiple data types, you can use methods like drop_nulls() or conditional filtering just as you would with any DataFrame.


import polars as pl

data = {
    "name": ["Alice", None, "Charlie", "David"],
    "age": [25, 30, None, 40],
    "score": [88.5, 92.0, 85.0, None],
    "active": [True, None, True, True],
}
df = pl.DataFrame(data)

df2 = df.drop_nulls()
print("DataFrame after dropping rows with any nulls:\n", df2)

# Output:
# DataFrame after dropping rows with any nulls:
# shape: (1, 4)
┌───────┬─────┬───────┬────────┐
│ name  ┆ age ┆ score ┆ active │
│ ---   ┆ --- ┆ ---   ┆ ---    │
│ str   ┆ i64 ┆ f64   ┆ bool   │
╞═══════╪═════╪═══════╪════════╡
│ Alice ┆ 25  ┆ 88.5  ┆ true   │
└───────┴─────┴───────┴────────┘

Here,

drop_nulls() will scan all columns, regardless of their data type (e.g., strings, integers, floats, booleans).
It removes rows where any column contains a null (missing) value.
The output DataFrame will only include rows with complete data across all columns.

Conclusion

In conclusion, Polars provides efficient and flexible ways to handle and drop null (NaN) values from DataFrames, regardless of data types. Whether you want to remove rows with nulls across all columns or target specific columns, Polars’ methods like drop_nulls() and conditional filtering with filter() make data cleaning simple and effective.

Happy Learning!!

References

https://docs.pola.rs/api/python/dev/reference/dataframe/api/polars.DataFrame.explode.html

Polars DataFrame Drop Nans

Polars DataFrame drop_nans() Introduction

Parameters of the Polars DataFrame drop_nans()

Return Value

Usage of Polars DataFrame drop_nans() Method

Drop Rows with NaNs only in Specific Columns

Drop Rows with NaNs in a Single Column

Drop Rows with NaNs Using a Condition

Drop Rows with NaNs from a DataFrame with Multiple Data Types

Conclusion

Related Articles

References