How to Add a Column to a Polars DataFrame Using .with_columns()

In Polars, the with_columns() method is used to add one or more columns to a DataFrame. This method returns a new DataFrame, leaving the original DataFrame unchanged, ensuring immutability. It is highly versatile and can be used to perform complex transformations, add calculated or derived columns, and even introduce constant values.

In this article, I will explain the Polars DataFrame.with_columns() method, covering its syntax, parameters, and usage to demonstrate how it returns a new Polars DataFrame with added or modified columns, while maintaining the immutability of the original DataFrame.

Key Points –

with_columns() is used to add, modify, or update columns in a Polars DataFrame.
It allows adding or modifying multiple columns simultaneously in a single operation.
Columns are defined using Polars expressions, enabling complex transformations and calculations.
You can add multiple columns at once, which is more efficient than adding columns individually.
The method supports conditional column creation using expressions like pl.when(), pl.col(), and pl.lit().
It does not modify the original DataFrame, providing a functional approach to DataFrame manipulation.
New columns can be named using .alias() within the expression.
You can use complex expressions, such as mathematical operations or column-wise transformations, to create new columns.

Syntax of with_columns() Function

Following is the syntax of the with_columns() function.


# Syntax of with_columns()
DataFrame.with_columns(
    *exprs: IntoExpr | Iterable[IntoExpr],
    **named_exprs: IntoExpr,
) → DataFrame

Parameters of the Polars DataFrame.with_columns()

Following are the parameters of the polars DataFrame.with_columns() method.

*exprs (Positional Arguments):
One or more expressions (IntoExpr) or an iterable of expressions. (e.g., column operations or transformations).
These expressions specify how to compute the columns to be added or modified.
You can use Polars' expression functions like pl.col, pl.lit, or mathematical operations.
**named_exprs (Keyword Arguments):
This is a keyword argument where the key is the column name, and the value is the expression that defines the column values.
Allows you to specify new or modified columns by name directly in the method call.

Return Value

This method returns a new DataFrame with the added columns, without modifying the original DataFrame.

Usage of polars.DataFrame.with_columns()

The polars.DataFrame.with_columns() method is used to add or modify one or more columns in a Polars DataFrame. This method is versatile and allows you to apply transformations, arithmetic operations, conditional logic, or custom functions to create new columns or update existing ones.

First, let’s create a polars DataFrame.


import polars as pl

# Creating a new Polars DataFrame
technologies = {
    'Courses': ["Spark", "Hadoop", "Python", "Pandas"],
    'Fees': [22000, 25000, 20000, 26000],
    'Duration': ['30days', '50days', '40days', '40days'],
    'Discount': [1000, 1500, 1200, 2500]
}

df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To add a new column with arithmetic operations, you can use the with_columns() method to perform the calculation and add the result as a new column. For instance, let's add a new column called "Net_Fees", which will be calculated by subtracting the Discount from the Fees.


# Adding a new column "Net_Fees" by subtracting Discount from Fees
df2 = df.with_columns(
    (pl.col("Fees") - pl.col("Discount")).alias("Net_Fees")
)
print("DataFrame with Net_Fees Column:\n", df2)

Here,

We are subtracting the Discount column from the Fees column to calculate the Net_Fees for each course.
The new column Net_Fees is added to the DataFrame using the with_columns() method.

Adding a Constant Value Column

You can use the with_columns() method to add a column with a constant value by assigning the same value to each row. For instance, you can create a new column named "Country" and set its value to "India" for every row.


# Adding a constant column "Country" with the value "India"
df2 = df.with_columns(
    pl.lit("India").alias("Country"))
print("DataFrame with Country Column:\n", df2)

# Output:
# DataFrame with Country Column:
# shape: (4, 5)
┌─────────┬───────┬──────────┬──────────┬─────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount ┆ Country │
│ ---     ┆ ---   ┆ ---      ┆ ---      ┆ ---     │
│ str     ┆ i64   ┆ str      ┆ i64      ┆ str     │
╞═════════╪═══════╪══════════╪══════════╪═════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     ┆ India   │
│ Hadoop  ┆ 25000 ┆ 50days   ┆ 1500     ┆ India   │
│ Python  ┆ 20000 ┆ 40days   ┆ 1200     ┆ India   │
│ Pandas  ┆ 26000 ┆ 40days   ┆ 2500     ┆ India   │
└─────────┴───────┴──────────┴──────────┴─────────┘

Here,

The pl.lit("India") function creates a column with the constant value "India" for every row.
We then use the .alias("Country") to name this column "Country".
The new column is added to the DataFrame using with_columns().

Creating a Column Based on Multiple Existing Columns

To create a new column based on multiple existing columns, you can use the with_columns() method to perform operations across those columns.


# Creating a new column "Final_Fee" based on multiple existing columns
df2= df.with_columns(
    ((pl.col("Fees") - pl.col("Discount")) + (pl.col("Fees") * 0.10)).alias("Final_Fee"))
print("DataFrame with Final_Fee Column:\n", df2)

# Output:
# DataFrame with Final_Fee Column:
# shape: (4, 5)
┌─────────┬───────┬──────────┬──────────┬───────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount ┆ Final_Fee │
│ ---     ┆ ---   ┆ ---      ┆ ---      ┆ ---       │
│ str     ┆ i64   ┆ str      ┆ i64      ┆ f64       │
╞═════════╪═══════╪══════════╪══════════╪═══════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     ┆ 23200.0   │
│ Hadoop  ┆ 25000 ┆ 50days   ┆ 1500     ┆ 26000.0   │
│ Python  ┆ 20000 ┆ 40days   ┆ 1200     ┆ 20800.0   │
│ Pandas  ┆ 26000 ┆ 40days   ┆ 2500     ┆ 26100.0   │
└─────────┴───────┴──────────┴──────────┴───────────┘

Here,

The Final_Fee is computed by first subtracting the Discount from the Fees and then adding 10% of Fees (calculated as pl.col("Fees") * 0.10).
The new column is added using with_columns() with the alias "Final_Fee".

Conditional Column Creation

You can create columns conditionally based on certain criteria using expressions like pl.when(), pl.col(), and pl.lit(). These expressions are used inside the with_columns() method to add new columns based on conditions.

Suppose we have created a new column called "Discount" which will be labeled as 'High' if the "Discount_Status," is greater than or equal to 1500, and 'Low' otherwise.


# Creating a new column "Discount_Status" based on a condition
df2 = df.with_columns(
    pl.when(pl.col("Discount") >= 1500)
    .then(pl.lit("High"))
    .otherwise(pl.lit("Low"))
    .alias("Discount_Status"))
print("DataFrame with Discount_Status Column:\n", df2)

# Output:
# DataFrame with Discount_Status Column:
# shape: (4, 5)
┌─────────┬───────┬──────────┬──────────┬─────────────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount ┆ Discount_Status │
│ ---     ┆ ---   ┆ ---      ┆ ---      ┆ ---             │
│ str     ┆ i64   ┆ str      ┆ i64      ┆ str             │
╞═════════╪═══════╪══════════╪══════════╪═════════════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     ┆ Low             │
│ Hadoop  ┆ 25000 ┆ 50days   ┆ 1500     ┆ High            │
│ Python  ┆ 20000 ┆ 40days   ┆ 1200     ┆ Low             │
│ Pandas  ┆ 26000 ┆ 40days   ┆ 2500     ┆ High            │
└─────────┴───────┴──────────┴──────────┴─────────────────┘

Here,

The pl.when(pl.col("Discount") >= 1500) checks if the Discount is greater than or equal to 1500.
The .then(pl.lit("High")) sets the value to "High" when the condition is met.
The .otherwise(pl.lit("Low")) sets the value to "Low" when the condition is not met.
The .alias("Discount_Status") assigns the name "Discount_Status" to the new column.

Adding Multiple Columns Simultaneously

To add multiple columns simultaneously in Polars, you can pass a list of expressions to the with_columns() method. Each expression can either perform a transformation or computation, just like when adding a single column.


# Adding multiple columns simultaneously: "Net_Fee" and "Discount_Percentage"
df2 = df.with_columns([
    (pl.col("Fees") + pl.col("Discount")).alias("Net_Fee"),
    ((pl.col("Discount") / pl.col("Fees")) * 100).alias("Discount_Percentage")
])
print("DataFrame with Net_Fee and Discount_Percentage Columns:\n", df2)

# Output:
# DataFrame with Net_Fee and Discount_Percentage Columns:
# shape: (4, 6)
┌─────────┬───────┬──────────┬──────────┬─────────┬─────────────────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount ┆ Net_Fee ┆ Discount_Percentage │
│ ---     ┆ ---   ┆ ---      ┆ ---      ┆ ---     ┆ ---                 │
│ str     ┆ i64   ┆ str      ┆ i64      ┆ i64     ┆ f64                 │
╞═════════╪═══════╪══════════╪══════════╪═════════╪═════════════════════╡
│ Spark   ┆ 22000 ┆ 30days   ┆ 1000     ┆ 23000   ┆ 4.545455            │
│ Hadoop  ┆ 25000 ┆ 50days   ┆ 1500     ┆ 26500   ┆ 6.0                 │
│ Python  ┆ 20000 ┆ 40days   ┆ 1200     ┆ 21200   ┆ 6.0                 │
│ Pandas  ┆ 26000 ┆ 40days   ┆ 2500     ┆ 28500   ┆ 9.615385            │
└─────────┴───────┴──────────┴──────────┴─────────┴─────────────────────┘

Here,

We pass a list of expressions to with_columns(), where each expression creates a new column.
The first expression calculates "Net_Fee" by addition the Discount from the Fees.
The second expression calculates the "Discount_Percentage" by dividing the Discount by the Fees and multiplying by 100 to get the percentage.
The columns are added to the DataFrame all at once, making it more efficient than adding columns individually.

Conclusion

In this article, I have explained the Polars DataFrame.with_columns() function, including its syntax, parameters, and usage, to understand how it returns a new DataFrame with added or modified columns. Also whether adding constant values, performing calculations, creating conditional columns, or applying custom functions, with_columns() provides a simple and efficient method for DataFrame manipulation.

Happy Learning!!

References

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.vstack.html

How to Add a Column to a Polars DataFrame Using .with_columns()

Syntax of with_columns() Function

Parameters of the Polars DataFrame.with_columns()

Return Value

Usage of polars.DataFrame.with_columns()

Adding a Constant Value Column

Creating a Column Based on Multiple Existing Columns

Conditional Column Creation

Adding Multiple Columns Simultaneously

Conclusion

Related Articles

References