Polars DataFrame pipe() Usage & Examples

In Polars, the pipe() method follows a functional programming approach, enabling you to apply custom transformations to a DataFrame in a clear and modular manner. The key concept is to "pipe" the DataFrame into a function, perform the transformation, and return the updated DataFrame (or any other result). By using pipe(), you can seamlessly chain multiple transformations or operations into one expression, making your code more concise, readable, and maintainable.

In this article, I will explain the pipe() method of the Polars DataFrame, covering its syntax, parameters, and practical use cases. This method returns the result of the function you provide, typically a new Polars DataFrame with the transformation applied.

Key Points –

pipe() allows for chaining custom transformations on a DataFrame in a clean, readable, and reusable way.
The function passed to pipe() must accept the DataFrame as its first argument.
Polars DataFrames are immutable, and pipe() returns a new DataFrame with each transformation.
The function used with pipe() can return anything, including another DataFrame, Series, scalar, or any other object.
You can chain multiple pipe() calls together to apply several transformations sequentially.
Using pipe() improves readability by avoiding deeply nested function calls and complex method chains.
pipe() does not modify the DataFrame in place. Instead, it returns a new modified DataFrame.
Additional arguments and keyword arguments can be passed to the function within pipe().

Polars DataFrame pipe() Introduction

Let’s know the syntax of the pipe() function.


# Syntax of Dataframe pipe()
DataFrame.pipe(
function: Callable[Concatenate[DataFrame, P], T],
*args: P.args,
**kwargs: P.kwargs,
) → T

Parameters of the Polars DataFrame pipe()

Following are the parameters of the pipe() method.

function - This is the function that you want to apply to the DataFrame. It must accept the DataFrame as its first argument (followed by other arguments that can be passed).
*args - These are additional positional arguments that will be passed to the function after the DataFrame.
**kwargs - These are additional keyword arguments passed to the function.
T - The return value of the function. This can be any type: another DataFrame, a Series, a scalar, or even other objects.

Usage of Polars DataFrame pipe() Method

The pipe() method in Polars allows you to chain custom transformations on a DataFrame in a clean, readable way, especially when you want to apply custom functions or multiple steps in a pipeline style.

To run some examples of the Polars DataFrame pipe() function, let’s create a Polars DataFrame.


import polars as pl

technologies= {
    'Courses':["Spark","PySpark","Hadoop","Python","Pandas"],
    'Fees' :[22000,25000,23000,24000,26000],
    'Duration':['30days','50days','70days', '40days','55days'],
    'Discount':[1000,2300,2000,1200,2500]
          }
df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To use the pipe() method to add a derived column in Polars, you can chain a transformation that computes a new column based on existing ones. For instance, you might want to calculate a new column based on the Fees and Discount columns.


# Using pipe() to add a derived column (Final_Fee)
df2 = df.pipe(
    lambda df: df.with_columns(
        (pl.col("Fees") - pl.col("Discount")).alias("Final_Fee")
    )
)
print("DataFrame with Derived Column:\n", df2)

Here,

We use the pipe() method to apply the transformation.
Inside the lambda function, we use the with_columns() method to create a new derived column Final_Fee.
- The new column is calculated by subtracting the Discount from the Fees column.
The derived column Final_Fee is added to the DataFrame.

Using pipe() Method to Apply Multiple Transformations

If you want to apply multiple transformations on a Polars DataFrame using the pipe() method, it's a clean and elegant way to chain operations, especially useful when you want to modularize your transformations.


# Transformation 1: Convert 'Duration' to integer
def convert_duration(df):
    return df.with_columns(
        pl.col("Duration").str.extract(r"(\d+)", 1).cast(pl.Int32).alias("Duration_days")
    )

# Transformation 2: Add a new column for final fee
def add_final_fee(df):
    return df.with_columns(
        (pl.col("Fees") - pl.col("Discount")).alias("Final_Fee")
    )

# Apply using pipe
df_transformed = (df.pipe(convert_duration).pipe(add_final_fee))
print("Transformed DataFrame:\n", df_transformed)

# Output:
# Transformed DataFrame:
# shape: (5, 6)
┌─────────┬───────┬──────────┬──────────┬───────────────┬───────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount ┆ Duration_days ┆ Final_Fee │
│ ---     ┆ ---   ┆ ---      ┆ ---      ┆ ---           ┆ ---       │
│ str     ┆ i64   ┆ str      ┆ i64      ┆ i32           ┆ i64       │
╞═════════╪═══════╪══════════╪══════════╪═══════════════╪═══════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     ┆ 30            ┆ 21000     │
│ PySpark ┆ 25000 ┆ 50days   ┆ 2300     ┆ 50            ┆ 22700     │
│ Hadoop  ┆ 23000 ┆ 70days   ┆ 2000     ┆ 70            ┆ 21000     │
│ Python  ┆ 24000 ┆ 40days   ┆ 1200     ┆ 40            ┆ 22800     │
│ Pandas  ┆ 26000 ┆ 55days   ┆ 2500     ┆ 55            ┆ 23500     │
└─────────┴───────┴──────────┴──────────┴───────────────┴───────────┘

Rename Columns Using pipe() Method

To rename the columns of a Polars DataFrame using the pipe() method, you can define a function that handles the renaming and then chain it through the pipe() function.


# Renaming columns using pipe() method
df2= df.pipe(lambda df: df.rename({"Courses": "Course_Name", "Fees": "Course_Fee", "Duration": "Course_Duration", "Discount": "Course_Discount"}))
print("Renamed DataFrame:\n", df2)

# Output:
# Renamed DataFrame:
# shape: (5, 4)
┌─────────────┬────────────┬─────────────────┬─────────────────┐
│ Course_Name ┆ Course_Fee ┆ Course_Duration ┆ Course_Discount │
│ ---         ┆ ---        ┆ ---             ┆ ---             │
│ str         ┆ i64        ┆ str             ┆ i64             │
╞═════════════╪════════════╪═════════════════╪═════════════════╡
│ Spark       ┆ 22000      ┆ 30days          ┆ 1000            │
│ PySpark     ┆ 25000      ┆ 50days          ┆ 2300            │
│ Hadoop      ┆ 23000      ┆ 70days          ┆ 2000            │
│ Python      ┆ 24000      ┆ 40days          ┆ 1200            │
│ Pandas      ┆ 26000      ┆ 55days          ┆ 2500            │
└─────────────┴────────────┴─────────────────┴─────────────────┘

Here,

The pipe() method is used to apply a lambda function that renames the columns.
The rename() function maps the original column names to the new names.

Filter Rows Using a Function with the pipe()

To filter rows using a function with the pipe() method in Polars, you can pass a lambda function inside pipe() that applies a condition to filter the rows based on one or more columns.

Here’s an example where we use the pipe() method to filter the rows based on the Fees column (e.g., only courses where the fee is greater than 24000).


# Using pipe() to filter rows where Fees > 24000
df2 = df.pipe(
    lambda df: df.filter(pl.col("Fees") > 24000)
)
print("Filtered DataFrame:\n", df2)

# Output:
# Filtered DataFrame:
# shape: (2, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ PySpark ┆ 25000 ┆ 50days   ┆ 2300     │
│ Pandas  ┆ 26000 ┆ 55days   ┆ 2500     │
└─────────┴───────┴──────────┴──────────┘

Here,

The pipe() method is used to apply a transformation to the DataFrame.
Inside the lambda function, we use the filter() method to filter rows where the Fees column is greater than 24000.
The result is a new DataFrame with only the rows that meet the filter condition.

Convert Column Data Types Using pipe() Method

You can convert column data types using the pipe() method in Polars by combining it with the with_columns() method and cast() function.

Here’s a clear example of how to do it using pipe() to convert the Fees column from Int64 to Float64.


# Convert 'Fees' column from Int64 to Float64 using pipe()
df2 = df.pipe(
    lambda df: df.with_columns(
        pl.col("Fees").cast(pl.Float64)
    )
)
print("DataFrame with Converted Data Type:\n", df2)

# Output:
# DataFrame with Converted Data Type:
# shape: (5, 4)
┌─────────┬─────────┬──────────┬──────────┐
│ Courses ┆ Fees    ┆ Duration ┆ Discount │
│ ---     ┆ ---     ┆ ---      ┆ ---      │
│ str     ┆ f64     ┆ str      ┆ i64      │
╞═════════╪═════════╪══════════╪══════════╡
│ Spark   ┆ 22000.0 ┆ 30days   ┆ 1000     │
│ PySpark ┆ 25000.0 ┆ 50days   ┆ 2300     │
│ Hadoop  ┆ 23000.0 ┆ 70days   ┆ 2000     │
│ Python  ┆ 24000.0 ┆ 40days   ┆ 1200     │
│ Pandas  ┆ 26000.0 ┆ 55days   ┆ 2500     │
└─────────┴─────────┴──────────┴──────────┘

Here,

pipe() allows you to apply a transformation function to the DataFrame.
Inside the lambda, pl.col("Fees").cast(pl.Float64) converts the type of Fees from integer to float.
You can apply this to multiple columns too by passing a list of expressions to with_columns()

Add a Flag Based on Condition Using pipe() Method

To add a flag column based on a condition using the pipe() method in Polars.


# Add a flag based on condition using pipi() method
def add_long_duration_flag(df: pl.DataFrame) -> pl.DataFrame:
    return df.with_columns(
        (pl.col("Duration").str.strip_chars("days").cast(pl.Int32) > 50)
        .alias("Long_Duration")
    )

df2 = df.pipe(add_long_duration_flag)
print("DataFrame with Duration Flag:\n", df2)

# Output:
# DataFrame with Duration Flag:
# shape: (5, 5)
┌─────────┬───────┬──────────┬──────────┬───────────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount ┆ Long_Duration │
│ ---     ┆ ---   ┆ ---      ┆ ---      ┆ ---           │
│ str     ┆ i64   ┆ str      ┆ i64      ┆ bool          │
╞═════════╪═══════╪══════════╪══════════╪═══════════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     ┆ false         │
│ PySpark ┆ 25000 ┆ 50days   ┆ 2300     ┆ false         │
│ Hadoop  ┆ 23000 ┆ 70days   ┆ 2000     ┆ true          │
│ Python  ┆ 24000 ┆ 40days   ┆ 1200     ┆ false         │
│ Pandas  ┆ 26000 ┆ 55days   ┆ 2500     ┆ true          │
└─────────┴───────┴──────────┴──────────┴───────────────┘

Conclusion

In summary, the pipe() method in Polars provides a powerful and readable way to apply multiple transformations. By encapsulating each transformation in a function and chaining them with pipe(), you keep your code modular, clean, and easy to maintain.

Happy Learning!!

References

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.pivot.html

Polars DataFrame pipe() Usage & Examples

Polars DataFrame pipe() Introduction

Parameters of the Polars DataFrame pipe()

Usage of Polars DataFrame pipe() Method

Using pipe() Method to Apply Multiple Transformations

Rename Columns Using pipe() Method

Filter Rows Using a Function with the pipe()

Convert Column Data Types Using pipe() Method

Add a Flag Based on Condition Using pipe() Method

Conclusion

Related Articles

References