Polars DataFrame extend() Explained by Examples

In Polars, the DataFrame.extend() method is used to append rows from one DataFrame to another in-place, directly modifying the original DataFrame. This operation requires both DataFrames to have the same schema, that is, identical column names, data types, and column order.

In this article, I will explain the Polars DataFrame extend() function, including its syntax, parameters, and usage, and show how it returns the modified DataFrame (self) with both the original and appended rows.

Key Points –

extend() is used to append rows from one DataFrame to another, modifying the original DataFrame in-place.
Both DataFrames must have identical column names and data types for extend() to work.
The function updates the target DataFrame directly and returns None.
It's ideal for combining batches of data, especially in streaming or iterative contexts.
Unlike some other methods, extend() does not automatically cast data types between mismatched schemas.
extend() is optimized for speed and is faster than using row-wise concatenation techniques.
The column order in both DataFrames must match, in addition to names and types.
Works well with dynamically created DataFrames as long as the schema is consistent.

Polars DataFrame extend() Introduction

Let’s know the syntax of the extend() function.


# Syntax of DataFrame.extend()
DataFrame.extend(other: DataFrame) → DataFrame

Parameters of the Polars DataFrame extend()

It allows only one parameter.

other (DataFrame) - The DataFrame whose rows will be appended to the current DataFrame.

Return Value

This method returns the original DataFrame, now extended with rows from other.

Usage of Polars DataFrame extend() Function

The extend() method in Polars adds rows from one DataFrame to another in-place, directly modifying the original DataFrame. This means the new rows are appended without creating a copy.

To run some examples of the Polars DataFrame extend() function, let’s create a Polars DataFrame.


import polars as pl

technologies = {
    'Courses':["Spark","PySpark","Pandas"],
    'Fees' :[20000,25000,30000]
              }
df1 = pl.DataFrame(technologies)
print("First DataFrame:\n", df1)

technologies2 = {
    'Courses':["Polars","Java","Python"],
    'Fees' :[15000,35000,40000] 
}
        
df2 = pl.DataFrame(technologies2)
print("Second DataFRame:\n", df2)

Yields below output.

To use the extend() method in Polars to combine two DataFrames by appending rows from one to the other. It operates in-place, modifying the original DataFrame, and returns None.


# Extend df1 with df2
df1.extend(df2)
print("After extend:\n", df1)

Here,

df1.extend(df2) adds all rows from df2 to df1.
It modifies df1 in place.
Both DataFrames must have the same schema.

Extend with Multiple Columns

When working with Polars DataFrames that have multiple columns, you can use the extend() method to append rows from one DataFrame to another. This method modifies the original DataFrame in-place, provided both DataFrames share the same schema, meaning identical column names and data types.


import polars as pl

# First DataFrame with multiple columns
df1 = pl.DataFrame({
    'Courses': ["Spark", "PySpark", "Pandas"],
    'Fees': [20000, 25000, 30000],
    'Duration': ["30 Days", "40 Days", "35 Days"],
    'Discount': [1000, 1500, 1200]
})

# Second DataFrame with the same schema
df2 = pl.DataFrame({
    'Courses': ["Polars", "Java", "Python"],
    'Fees': [15000, 35000, 40000],
    'Duration': ["25 Days", "50 Days", "45 Days"],
    'Discount': [800, 2000, 1800]
})

# Use extend to add df2 rows to df1
df1.extend(df2)
print("After Extending df1 with df2:\n", df1)

# Output:
# After Extending df1 with df2:
# shape: (6, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Duration ┆ Discount │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ str      ┆ i64      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 20000 ┆ 30 Days  ┆ 1000     │
│ PySpark ┆ 25000 ┆ 40 Days  ┆ 1500     │
│ Pandas  ┆ 30000 ┆ 35 Days  ┆ 1200     │
│ Polars  ┆ 15000 ┆ 25 Days  ┆ 800      │
│ Java    ┆ 35000 ┆ 50 Days  ┆ 2000     │
│ Python  ┆ 40000 ┆ 45 Days  ┆ 1800     │
└─────────┴───────┴──────────┴──────────┘

Here,

The extend() method modifies the originalDataFrame**(df1) in-place** and returns None.
The column names and data types must match exactly between df1 and df2.

Extend an Empty DataFrame

You can extend an empty Polars DataFrame by appending rows from another DataFrame using the extend() method, as long as the schemas match.


import polars as pl

# Create an empty DataFrame with a defined schema
df_empty = pl.DataFrame(
    schema=[
        ("Courses", pl.Utf8),
        ("Fees", pl.Int64)
    ]
)

# Create a DataFrame with actual data
df_data = pl.DataFrame({
    'Courses': ["Spark", "PySpark", "Pandas"],
    'Fees': [20000, 25000, 30000]
})

# Extend the empty DataFrame with df_data
df_empty.extend(df_data)
print("After Extending Empty DataFrame:\n", df_empty)

# Output:
# After Extending Empty DataFrame:
# shape: (3, 2)
┌─────────┬───────┐
│ Courses ┆ Fees  │
│ ---     ┆ ---   │
│ str     ┆ i64   │
╞═════════╪═══════╡
│ Spark   ┆ 20000 │
│ PySpark ┆ 25000 │
│ Pandas  ┆ 30000 │
└─────────┴───────┘

Here,

The empty DataFrame must define the schema upfront, otherwise Polars doesn’t know the expected column types.
The extend() method appends rows in-place and works even when starting from an empty DataFrame.

Extend Using Dynamic Data

You can extend a Polars DataFrame with dynamically generated data, such as data from user input, APIs, or runtime computations, as long as the schema matches the original DataFrame.


import polars as pl

# Existing DataFrame
df1 = pl.DataFrame({
    'Courses': ["Spark", "PySpark"],
    'Fees': [20000, 25000]
})

# Dynamic data (e.g., fetched or generated at runtime)
df2 = {
    'Courses': ["Pandas", "Polars"],
    'Fees': [30000, 15000]
}

# Convert dynamic data to Polars DataFrame
df_dynamic = pl.DataFrame(df2)

# Extend main DataFrame with dynamic data
df1.extend(df_dynamic)
print("After Extending with Dynamic Data:\n", df1)

# Output:
# After Extending with Dynamic Data:
# shape: (4, 2)
┌─────────┬───────┐
│ Courses ┆ Fees  │
│ ---     ┆ ---   │
│ str     ┆ i64   │
╞═════════╪═══════╡
│ Spark   ┆ 20000 │
│ PySpark ┆ 25000 │
│ Pandas  ┆ 30000 │
│ Polars  ┆ 15000 │
└─────────┴───────┘

Conclusion

In summary, the extend() method in Polars provides an efficient way to append rows from one DataFrame to another in-place, as long as their schemas match. It's especially useful when working with dynamic or incremental data, making it a practical tool for building and updating DataFrames efficiently.

Happy Learning!!

References

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.explode.html

Polars DataFrame extend() Explained by Examples

Polars DataFrame extend() Introduction

Parameters of the Polars DataFrame extend()

Return Value

Usage of Polars DataFrame extend() Function

Extend with Multiple Columns

Extend an Empty DataFrame

Extend Using Dynamic Data

Conclusion

Related Articles

References