Polars Counting Elements in List Column

In Polars, counting elements in a list column refers to the process of determining how many individual items are present within each list inside a column that holds lists of values. This is important when working with data where each row in the column contains multiple items in the form of lists (like a list of numbers, strings, or mixed data types).

To do this in Polars, you can use the list.len() method, which computes the number of elements in each list. You can also use other functions to count specific occurrences of an item in each list or across the entire list column. In this article, I will explain how to count elements in a list column.

Key Points –

• A list column in Polars contains multiple values (lists) in each row, allowing for more complex data structures within a single column.
• To count the total number of elements in each list, the list.len() function is commonly used.
• Count unique elements in a list column by using list.unique() followed by list.len() to get the number of distinct items.
• You can count non-null elements in a list by using list.eval() to check for non-null entries followed by list.sum().
• Count elements that meet a condition (e.g., values greater than a threshold) by applying list.eval() with a conditional check.
• Counting elements satisfying multiple conditions can be done by chaining multiple conditions inside list.eval().
• Handling missing values, Null elements can be filtered out using list.drop_nulls() before counting.
• Polars is optimized for performance and can count list elements efficiently, even with large datasets.

<a id="h-usage-of-polars-counting-elements-in-list-column"></a>

Usage of Polars Counting Elements in List Column

To count the number of elements in each list within a list column in Polars, you can use the list.len() function. This function computes the length of each list in the column, effectively giving you the count of elements in each list.

Now, let’s create a Polars DataFrame.

import polars as pl

# Create a DataFrame with a list column
df = pl.DataFrame({
    "id": [1, 2, 3],
    "values": [[5, 10, 12], [15, 25], [20, 30, 40, 50]]
})
print("Original DataFrame:\n", df)

Yields below output.

To count the total number of elements in each list within the values column, you can use the list.len() method in Polars. This method will return the length of each list in the values column.

# Count the total elements in each list
df2 = df.with_columns(
    pl.col("values").list.len().alias("num_elements")
)
print("DataFrame with Element Counts:\n", df2)

Here,

• The list.len() function counts the number of elements in each list within the values column.
• The new column num_elements contains the count of elements in each corresponding list.

<a id="h-count-even-numbers-in-each-list"></a>

Count Even Numbers in Each List

To count the even numbers in each list within the 'values' column, you can combine Polars list functions such as list.eval() with a condition to filter even numbers (element % 2 == 0), and then apply list.sum() to get the count.

# Count even numbers in each list
df2 = df.with_columns(
    pl.col("values")
    .list.eval(pl.element() % 2 == 0)
    .list.sum()
    .alias("even_count")
)
print("DataFrame with Even Number Counts:\n", df2)

# Output:
# DataFrame with Even Number Counts:
# shape: (3, 3)
┌─────┬────────────────┬────────────┐
│ id  ┆ values         ┆ even_count │
│ --- ┆ ---            ┆ ---        │
│ i64 ┆ list[i64]      ┆ u32        │
╞═════╪════════════════╪════════════╡
│ 1   ┆ [5, 10, 12]    ┆ 2          │
│ 2   ┆ [15, 25]       ┆ 0          │
│ 3   ┆ [20, 30, … 50] ┆ 4          │
└─────┴────────────────┴────────────┘

Here,

• list.eval(pl.element() % 2 == 0) returns a list of booleans like [False, True, True]
• list.sum() sums the True values, effectively counting how many even numbers are present in each list.

<a id="h-count-elements-greater-than-a-value-e-g-gt-25"></a>

Count Elements Greater than a Value (e.g., > 25)

To count the elements greater than a specific value (e.g., > 25) in a list column in Polars, you can apply the following method: Use list.eval() to evaluate the condition for each element in the list, and then use list.sum() to count the True values, which correspond to the elements greater than 25.

# Count elements greater than 25
df2 = df.with_columns(
    pl.col("values")
    .list.eval(pl.element() > 25)  # condition for elements greater than 25
    .list.sum()
    .alias("count_greater_than_25")
)
print("DataFrame with Count of Elements Greater Than 25:\n", df2)

# Output:
# DataFrame with Count of Elements Greater Than 25
# shape: (3, 3)
┌─────┬────────────────┬───────────────────────┐
│ id  ┆ values         ┆ count_greater_than_25 │
│ --- ┆ ---            ┆ ---                   │
│ i64 ┆ list[i64]      ┆ u32                   │
╞═════╪════════════════╪═══════════════════════╡
│ 1   ┆ [5, 10, 12]    ┆ 0                     │
│ 2   ┆ [15, 25]       ┆ 0                     │
│ 3   ┆ [20, 30, … 50] ┆ 3                     │
└─────┴────────────────┴───────────────────────┘

Here,

• pl.element() > 25 checks whether each element in the list is greater than 25.
• list.eval() evaluates this condition for every element in the list.
• list.sum() counts the number of True values (those greater than 25).

<a id="h-count-unique-elements-in-each-list"></a>

Count Unique Elements in Each List

To count unique elements in each list within a column in Polars, you can use the list.unique() method to first extract the unique elements from each list and then use list.len() to count the length of the resulting list of unique elements.

import polars as pl

# Create a DataFrame with list column
df = pl.DataFrame({
    "id": [1, 2, 3],
    "values": [[5, 10, 12, 12], [15, 25, 25], [20, 30, 30, 50]]
})

# Count unique elements in each list
df2 = df.with_columns(
    pl.col("values")
    .list.unique()  # Get unique elements in each list
    .list.len()  # Count the unique elements
    .alias("unique_count")
)

print("DataFrame with Unique Element Counts:\n", df2)

# Output:
# DataFrame with Unique Element Counts:
# shape: (3, 3)
┌─────┬────────────────┬──────────────┐
│ id  ┆ values         ┆ unique_count │
│ --- ┆ ---            ┆ ---          │
│ i64 ┆ list[i64]      ┆ u32          │
╞═════╪════════════════╪══════════════╡
│ 1   ┆ [5, 10, … 12]  ┆ 3            │
│ 2   ┆ [15, 25, 25]   ┆ 2            │
│ 3   ┆ [20, 30, … 50] ┆ 3            │
└─────┴────────────────┴──────────────┘

Here,

• list.unique() removes duplicate values from each list.
• list.len() counts how many unique elements are left in each list.
• The result is stored in the new column unique_count.

<a id="h-count-non-null-elements-in-each-list"></a>

Count Non-null Elements in Each List

To count the non-null elements in each list within the values column in Polars, you can use list.eval() to identify non-null entries, followed by list.sum() to calculate their count.

import polars as pl

# DataFrame with some null values inside list elements
df = pl.DataFrame({
    "id": [1, 2, 3],
    "values": [[5, None, 12], [None, None], [20, 30, None, 50]]
})

# Count non-null elements in each list
df2 = df.with_columns(
    pl.col("values")
    .list.eval(pl.element().is_not_null())
    .list.sum()
    .alias("non_null_count")
)
print("DataFrame with Non-Null Element Counts:\n", df2)

# Output:
# DataFrame with Non-Null Element Counts:
# shape: (3, 3)
┌─────┬────────────────┬────────────────┐
│ id  ┆ values         ┆ non_null_count │
│ --- ┆ ---            ┆ ---            │
│ i64 ┆ list[i64]      ┆ u32            │
╞═════╪════════════════╪════════════════╡
│ 1   ┆ [5, null, 12]  ┆ 2              │
│ 2   ┆ [null, null]   ┆ 0              │
│ 3   ┆ [20, 30, … 50] ┆ 3              │
└─────┴────────────────┴────────────────┘

Here,

• pl.element().is_not_null() checks if each element is not null.
• list.eval(…) evaluates this condition within each list.
• list.sum() counts how many True (non-null) values are present.

Similarly, if you want to first remove the nulls, then count the remaining elements directly, you can use list.drop_nulls() followed by list.len() function.

# Count elements after dropping nulls
df2 = df.with_columns(
    pl.col("values")
    .list.drop_nulls()
    .list.len()
    .alias("count_after_dropping_nulls")
)

print("DataFrame with Filtered Element Counts:\n", df2)

# Output:
#  shape: (3, 3)
 shape: (3, 3)
┌─────┬────────────────┬────────────────────────────┐
│ id  ┆ values         ┆ count_after_dropping_nulls │
│ --- ┆ ---            ┆ ---                        │
│ i64 ┆ list[i64]      ┆ u32                        │
╞═════╪════════════════╪════════════════════════════╡
│ 1   ┆ [5, null, 12]  ┆ 2                          │
│ 2   ┆ [null, null]   ┆ 0                          │
│ 3   ┆ [20, 30, … 50] ┆ 3                          │
└─────┴────────────────┴────────────────────────────┘

<a id="h-conclusion"></a>

Conclusion

In summary, counting elements in list columns using Polars is powerful, efficient, and expressive. Whether you're counting total elements, filtering based on conditions, handling nulls, or identifying unique values, Polars provides clear, vectorized methods to handle it all seamlessly.

Happy Learning!!

<a id="h-related-articles"></a>

Related Articles

• Convert Polars Casting a Column to Decimal
• Polars Looping Through the Rows in a Dataset
• Make a Constant Column in Polars
• Extract Value of Polars Literal
• Convert Polars String to Integer
• Polars Rename Columns to Lowercase
• How to Select Last Column of Polars DataFrame
• How to Change Position of a Column in Polars
• How to Add a Column with Numerical Value in Polars
• Get First N Characters from a String Column in Polars
• Removing Null Values on Selected Columns only in Polars DataFrame

<a id="h-references"></a>

References

• https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.max.html