Polars Filter by Column Value

In Polars, you can filter a DataFrame by column value using the filter() method, which operates similarly to Pandas but is optimized for speed and efficiency. Filtering can be performed using filter(), is_in(), str.contains(), and is_null() functions. Additionally, you can apply logical operators like | (OR), & (AND), and == (equal) for more advanced filtering. In this article, I will explain how to filter a Polars DataFrame by column values.

Key Points –

Use filter() for efficient row selection based on column conditions.
Use == to select rows where a column matches a specific value.
Use != to exclude rows with a specific column value.
Comparison operators (==, !=, <, >, <=, >=) can be used to filter numeric, string, and date values.
Multiple conditions can be combined using & (AND) and | (OR) inside parentheses.
Use str.contains("pattern") to filter rows that contain specific substrings.
Use is_in(["value1", "value2"]) to filter rows where a column matches any value in a list.
Use is_null() and is_not_null() to filter rows based on missing values.

Usage of Polars Filter by Column Value

The filter() method in Polars is used to select specific rows from a DataFrame based on column values. It is highly efficient and supports multiple conditions for filtering data.

To run some examples of polars filter by column value, let’s create a Polars DataFrame.


import polars as pl

# Creating a new Polars DataFrame
technologies= {
    'Courses':["Spark","PySpark","Hadoop","Python","Pandas"],
    'Fees' :[22000,25000,20000,24000,26000],
    'Duration':['30days','50days','40days','50days','40days'],
    'Discount':[1000,2300,1500,1200,2500]
}

df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To filter rows in a Polars DataFrame where a column equals a specific value, you can use the filter() method.


# Filtering rows where 'Courses' is 'Python'
df2 = df.filter(df["Courses"] == "Python")
print("Filtered DataFrame:\n", df2)

Here,

filter(df["Courses"] == "Python") filters rows where the "Courses" column is equal to "Python".
The result is a new filtered Polars DataFrame containing only rows that match the condition.

Filter Rows Where a Column is Greater than a Value

To filter rows in a Polars DataFrame where a column is greater than a specific value, use the filter() method with a comparison operator (>).


# Filtering rows where 'Fees' is greater than 22000
df2 = df.filter(df["Fees"] > 22000)
print("Filtered DataFrame:\n", df2)

# Output:
# Filtered DataFrame:
# shape: (3, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ PySpark ┆ 25000 ┆ 50days   ┆ 2300     │
│ Python  ┆ 24000 ┆ 50days   ┆ 1200     │
│ Pandas  ┆ 26000 ┆ 40days   ┆ 2500     │
└─────────┴───────┴──────────┴──────────┘

Here,

filter(df["Fees"] > 22000) filters rows where the "Fees" column value is greater than 22000.
The result is a new Polars DataFrame with only the matching rows.

Filter Rows Where a Column is Within a List of Values (is_in)

To filter rows in a Polars DataFrame where a column's values are within a specific list, use the is_in() method inside filter() function.


# Filtering rows where 'Courses' is in ["Python", "Spark"]
df2= df.filter(df["Courses"].is_in(["Python", "Spark"]))
print("Filtered DataFrame:\n", df2)

# Output:
# Filtered DataFrame:
# shape: (2, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     │
│ Python  ┆ 24000 ┆ 50days   ┆ 1200     │
└─────────┴───────┴──────────┴──────────┘

Here,

is_in(["Python", "Spark"]) checks if the "Courses" column values exist in the given list (["Python", "Spark"]).
filter() filters the DataFrame based on this condition.
The result is a new Polars DataFrame containing only the matching rows.

Filter Rows Where a Column is Not Equal to a Value

To filter rows where a column is not equal to a specific value in Polars, use the != operator or the filter() method.


# Filter rows where Duration is not '50days'
df2 = df.filter(df["Duration"] != "50days")
print("Filtered DataFrame:\n", df2)

# Output:
# Filtered DataFrame:
# shape: (3, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     │
│ Hadoop  ┆ 20000 ┆ 40days   ┆ 1500     │
│ Pandas  ┆ 26000 ┆ 40days   ┆ 2500     │
└─────────┴───────┴──────────┴──────────┘

Here,

df["Duration"] != "50days" selects all rows except those where "Duration" is "50days".
filter(): filters the DataFrame based on this condition.

Filter Rows Using Multiple Conditions (AND &)

To filter rows in Polars using multiple conditions (logical AND &), you can use the filter() method with multiple conditions inside parentheses.


# Filtering rows where 'Fees' > 22000 AND 'Discount' < 2000
df2= df.filter((df["Fees"] > 22000) & (df["Discount"] < 2000))
print("Filtered DataFrame:\n", df2)

# Output:
# Filtered DataFrame:
# shape: (1, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Python  ┆ 24000 ┆ 50days   ┆ 1200     │
└─────────┴───────┴──────────┴──────────┘

Here,

(df["Fees"] > 22000) selects rows where "Fees" is greater than 22000.
(df["Discount"] < 2000) selects rows where "Discount" is less than 2000.
&: the AND operator ensures that both conditions must be met.
filter(…): filters the DataFrame based on the condition.

Filter Rows Where a String Column Contains a Substring

To filter rows in a Polars DataFrame where a string column contains a specific substring, use the str.contains() function inside filter() function.


# Filtering rows where 'Courses' contains "Spark"
df2 = df.filter(df["Courses"].str.contains("Spark"))
print("Filtered DataFrame:\n", df2)

# Output:
# Filtered DataFrame:
# shape: (2, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     │
│ PySpark ┆ 25000 ┆ 50days   ┆ 2300     │
└─────────┴───────┴──────────┴──────────┘

Here,

str.contains("Spark") checks if the "Courses" column contains "Spark" anywhere in the string.
filter(…) filters the DataFrame based on this condition.
It returns a boolean mask, which filter() uses to keep matching rows.

Filter Rows Where a Column Value is Missing (null)

You can filter rows where a column value is missing (null) using the is_null() method inside filter() function.


import polars as pl

# Creating a Polars DataFrame with some null values
technologies = {
    'Courses': ["Spark", "PySpark", "Hadoop", "Python", "Pandas"],
    'Fees': [22000, 25000, 20000, None, 26000],  
    'Duration': ['30days', '50days', None, '50days', '40days'],  
    'Discount': [1000, 2300, 1500, None, 2500] 
}

df = pl.DataFrame(technologies)

# Filtering rows where 'Discount' is null
df2 = df.filter(df["Discount"].is_null())
print("Filtered DataFrame:\n", df2)

# Output:
# Filtered DataFrame:
# shape: (1, 4)
┌─────────┬──────┬──────────┬──────────┐
│ Courses ┆ Fees ┆ Duration ┆ Discount │
│ ---     ┆ ---  ┆ ---      ┆ ---      │
│ str     ┆ i64  ┆ str      ┆ i64      │
╞═════════╪══════╪══════════╪══════════╡
│ Python  ┆ null ┆ 50days   ┆ null     │
└─────────┴──────┴──────────┴──────────┘

Here,

is_null() checks if the "Discount" column has null values.
filter(…) selects only rows where "Discount" is null.

Conclusion

In summary, filtering by column values in Polars is a highly efficient and powerful method for extracting relevant data from a DataFrame. The filter() method enables users to apply various conditions, including equality, inequality, numerical comparisons, list-based filtering, and logical operations. By utilizing functions like filter(), is_in(), and str.contains(), you can efficiently process large datasets with minimal performance impact.

Happy Learning!!

Reference

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.filter.html

Polars Filter by Column Value

Usage of Polars Filter by Column Value

Filter Rows Where a Column is Greater than a Value

Filter Rows Where a Column is Within a List of Values (is_in)

Filter Rows Where a Column is Not Equal to a Value

Filter Rows Using Multiple Conditions (AND &)

Filter Rows Where a String Column Contains a Substring

Filter Rows Where a Column Value is Missing (null)

Conclusion

Related Articles

Reference