Polars Looping Through the Rows in a Dataset

In Polars, looping through the rows of a dataset involves iterating over each row in a DataFrame and accessing its data for specific tasks, such as transformations, condition checks, or aggregations. This technique is commonly used in data processing when operations need to be applied individually to each row. Polars provides methods like iter_rows() to facilitate this, allowing you to perform custom operations on each row, whether it's modifying data, applying conditions, or extracting particular values. In this article, I will demonstrate how to Polars loop through the rows in a Dataset.

Key Points–

iter_rows() is the primary method for iterating over rows in Polars, returning each row as a tuple or dictionary.
Using named=True in iter_rows() allows accessing column values by name, improving code readability.
Looping through rows is useful when specific row-wise operations, like conditional logic, need to be applied.
enumerate() can be used with iter_rows() to keep track of the row index during iteration.
The output of iter_rows() can be collected into new lists, dictionaries, or DataFrames for further processing.
You can keep track of row numbers (or indices) while looping through rows using enumerate() or by manually adding counters.
You can select specific columns using select() before looping to optimize performance and reduce memory usage.
Apply custom functions or logic to each row to perform transformations not easily handled by vectorized operations.

Usage of Polars Looping Through the Rows in a Dataset

To loop through the rows of a Polars DataFrame, you can use the iter_rows() method. This method allows you to iterate through each row of the DataFrame, either as a tuple or as a named tuple (dictionary-like structure) for better readability.

To run some examples of how to Polars looping through the rows in a dataset, let’s create a Polars DataFrame.


import polars as pl

technologies= {
    'Courses':["Spark","PySpark","Hadoop","Python","Pandas"],
    'Fees' :[22000,25000,23000,24000,26000],
    'Duration':['30days','50days','35days', '40days','55days'],
    'Discount':[1000,2300,1000,1200,2500]
          }
df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To iterate over the rows of a Polars DataFrame, you can use the iter_rows() method. Below is an example of how to loop through the rows of the technologies DataFrame using iter_rows() in Polars.


# Looping through the rows using iter_rows()
for row in df.iter_rows():
    print(row)

Each row here is a tuple of the values for the corresponding columns: "Courses", "Fees", "Duration", and "Discount".

Loop with named=True for Readability

When you set named=True in the iter_rows() method, each row is returned as a dictionary where the column names are used as keys. This makes the code more readable by accessing column values by name rather than by index.


# Looping through the rows 
# Using iter_rows() with named=True
for row in df.iter_rows(named=True):
    print(row)

# Output:
# {'Courses': 'Spark', 'Fees': 22000, 'Duration': '30days', 'Discount': 1000}
# {'Courses': 'PySpark', 'Fees': 25000, 'Duration': '50days', 'Discount': 2300}
# {'Courses': 'Hadoop', 'Fees': 23000, 'Duration': '35days', 'Discount': 1000}
# {'Courses': 'Python', 'Fees': 24000, 'Duration': '40days', 'Discount': 1200}
# {'Courses': 'Pandas', 'Fees': 26000, 'Duration': '55days', 'Discount': 2500}

By using named=True, each row is printed with clear column names, making it much easier to understand the data while iterating.

Conditional Logic Inside the Loop

You can incorporate conditional logic when iterating through the rows of a Polars DataFrame using iter_rows(named=True). This allows you to filter or act on data based on specific conditions. For example, the following code prints a message when the discount exceeds 2000.


# Loop through rows with named=True for readability
for row in df.iter_rows(named=True):
    # Apply conditional logic
    if row['Discount'] > 2000:
        print(f"Course {row['Courses']} has a discount greater than 2000! Discount: {row['Discount']}")
    else:
        print(f"Course {row['Courses']} has a discount of {row['Discount']}")
        
# Output:
# Course Spark has a discount of 1000
# Course PySpark has a discount greater than 2000! Discount: 2300
# Course Hadoop has a discount of 1000
# Course Python has a discount of 1200
# Course Pandas has a discount greater than 2000! Discount: 2500

In this example, the code checks whether the Discount for each course is greater than 2000. If it is, it prints a message indicating the discount is high, otherwise, it just prints the discount value.

Select Specific Columns to Loop Over

If you want to loop through only certain columns in a Polars DataFrame instead of all columns, you can first select the relevant columns and then iterate over the resulting DataFrame using iter_rows().


# Select specific columns
selected_df = df.select(['Courses', 'Fees'])

# Loop through selected rows
for row in selected_df.iter_rows(named=True):
    print(f"Course: {row['Courses']}, Fee: {row['Fees']}")
    
# Output:
# Course: Spark, Fee: 22000
# Course: PySpark, Fee: 25000
# Course: Hadoop, Fee: 23000
# Course: Python, Fee: 24000
# Course: Pandas, Fee: 26000

In the above example, use select(['col1', 'col2']) to narrow down the DataFrame to just the columns you're interested in. Then, loop through the rows with iter_rows(named=True) to access values by column name for improved readability.

Loop with Row Index using enumerate()

If you want to loop through the rows of a Polars DataFrame with the row index, you can use Python's built-in enumerate() function. This will allow you to keep track of the index while iterating through the rows.

Here’s how you can loop through the rows with enumerate() while using named=True for better readability.


# Loop through rows with enumerate to get index and row data
for idx, row in enumerate(df.iter_rows(named=True)):
    print(f"Row {idx}: Course = {row['Courses']}, Fees = ₹{row['Fees']}, Duration = {row['Duration']}, Discount = ₹{row['Discount']}")
    
# Output:
# runfile('C:/Users/chaga/.spyder-py3/temp.py', wdir='C:/Users/chaga/.spyder-py3')
#Row 0: Course = Spark, Fees = ₹22000, Duration = 30days, Discount = ₹1000
# Row 1: Course = PySpark, Fees = ₹25000, Duration = 50days, Discount = ₹2300
# Row 2: Course = Hadoop, Fees = ₹23000, Duration = 35days, Discount = ₹1000
# Row 3: Course = Python, Fees = ₹24000, Duration = 40days, Discount = ₹1200
# Row 4: Course = Pandas, Fees = ₹26000, Duration = 55days, Discount = ₹2500

Here,

enumerate() gives you the index (idx) and the row (row).
Inside the loop, row['Courses'], row['Fees'], etc., are accessed using column names, thanks to named=True.
You can use idx to get the row number, which starts from 0.

Modify Rows and Collect New List

If you want to loop through the rows, modify the data, and collect the modified rows into a new list, you can do that easily using iter_rows(named=True).


# Modify and collect new list
modified_rows = []
for row in df.iter_rows(named=True):
    new_row = row.copy()
    new_row['Discount'] = int(row['Discount'] * 1.10)  # increase discount by 10%
    modified_rows.append(new_row)

# Print modified data
for row in modified_rows:
    print(row)
    
# Output:
# {'Courses': 'Spark', 'Fees': 22000, 'Duration': '30days', 'Discount': 1100}
# {'Courses': 'PySpark', 'Fees': 25000, 'Duration': '50days', 'Discount': 2530}
# {'Courses': 'Hadoop', 'Fees': 23000, 'Duration': '35days', 'Discount': 1100}
# {'Courses': 'Python', 'Fees': 24000, 'Duration': '40days', 'Discount': 1320}
# {'Courses': 'Pandas', 'Fees': 26000, 'Duration': '55days', 'Discount': 2750}

Conclusion

In conclusion, looping through the rows in a Polars DataFrame is a powerful technique for applying custom logic on a row-by-row basis. While Polars is optimized for fast, vectorized operations, row iteration with iter_rows() (especially with named=True for clarity) is useful when your task requires accessing or modifying individual rows. Use it thoughtfully, lean on it when necessary, but prioritize vectorized solutions for better performance on large datasets.

Happy Learning!!

References

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.iter_slices.html