How to Effectively Create Duplicate Rows in Polars?

In Polars, creating duplicate rows involves repeating rows within a DataFrame, causing each row to appear multiple times. This can be done for all rows or selectively based on specific conditions or criteria. To effectively create duplicate rows, you can utilize Polars' optimized methods to replicate rows either by a fixed number or according to dynamic conditions. In this article, I will explain how to effectively create duplicate rows in polars.

Key Points –

• Use methods like repeat_by() to duplicate every row in a DataFrame a fixed number of times.
• Use pl.concat() to concatenate a DataFrame with itself for straightforward duplication of all rows.
• Apply conditional logic using map_elements() to repeat rows based on specific criteria (e.g., value in a column).
• Concatenate two or more identical DataFrames to stack them vertically, duplicating the entire dataset.
• When using map_elements() to create lists, use explode() to flatten these lists into separate rows.
• Use repeat_by() on a specific column to repeat individual column values a fixed number of times.
• Utilize map_elements() to apply complex logic for row duplication, such as creating lists of repeated values.
• Create a column with lists, and then use explode() to expand the lists into individual rows.

<a id="h-usage-of-effectively-create-duplicate-rows-in-polars"></a>

Usage of Effectively Create Duplicate Rows in Polars

Creating duplicate rows in Polars refers to adding copies of existing rows within a DataFrame, causing one or more rows to appear multiple times. This technique can be highly beneficial in various situations, including data augmentation, testing, and maintaining consistency within your dataset.

Now, let’s create a Polars DataFrame.

import polars as pl

technologies = {
    'Courses': ["Spark", "PySpark", "Pandas"],
    'Fees': [22000, 25000, 30000],
    'Discount': [1000, 2500, 2000],
    'Duration': ['30days', '40days', '50days']
}

df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To join two identical DataFrames in Polars, you can use concat() to stack them vertically. This allows you to duplicate all rows in a DataFrame by concatenating it with itself.

# Duplicate all rows once
df_duplicated = pl.concat([df, df])
print("Duplicated DataFrame:\n", df_duplicated)

Here,

• pl.concat([df, df]): This joins two identical DataFrames by stacking them vertically (i.e., adding the rows of the second DataFrame below the first).
• The resulting DataFrame has double the number of rows as the original DataFrame, keeping the same columns.

<a id="h-repeat-each-row-a-fixed-number-of-times-using-repeat-by"></a>

Repeat Each Row a Fixed Number of Times Using repeat_by()

To repeat each row a specific number of times in Polars, you can use the repeat_by() method. This function allows you to repeat the values of each column according to a given condition or list, offering more flexibility than simple concatenation.

# Repeat each row twice using repeat_by
repeated_df = df.select(pl.all().repeat_by(2).flatten())
print("Repeated Rows (each row duplicated twice):\n", repeated_df)

# Output:
# Repeated Rows (each row duplicated twice):
# shape: (6, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Discount ┆ Duration │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ i64      ┆ str      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 1000     ┆ 30days   │
│ Spark   ┆ 22000 ┆ 1000     ┆ 30days   │
│ PySpark ┆ 25000 ┆ 2500     ┆ 40days   │
│ PySpark ┆ 25000 ┆ 2500     ┆ 40days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
└─────────┴───────┴──────────┴──────────┘

Here,

• repeat_by(2): This repeats each row (or value in the column) twice.
• The resulting DataFrame now contains 6 rows instead of 3, as each row from the original DataFrame is repeated twice.

<a id="h-repeat-data-using-repeat-by-on-a-specific-column"></a>

Repeat Data Using repeat_by() on a Specific Column

To repeat data on a specific column using the repeat_by() function in Polars, you can apply repeat_by() only to the selected column, while leaving other columns unchanged.

Here’s an example where we repeat the Courses column a fixed number of times (e.g., 2 times), but leave the other columns (e.g., Fees, Discount, Duration) unchanged

# Repeat only the "Courses" column twice
df2 = df.with_columns(
    pl.col("Courses").repeat_by(2).alias("Courses"))

print("DataFrame with 'Courses' Repeated Twice:\n", df2)

# Output:
# DataFrame with 'Courses' Repeated Twice:
# shape: (3, 4)
┌────────────────────────┬───────┬──────────┬──────────┐
│ Courses                ┆ Fees  ┆ Discount ┆ Duration │
│ ---                    ┆ ---   ┆ ---      ┆ ---      │
│ list[str]              ┆ i64   ┆ i64      ┆ str      │
╞════════════════════════╪═══════╪══════════╪══════════╡
│ ["Spark", "Spark"]     ┆ 22000 ┆ 1000     ┆ 30days   │
│ ["PySpark", "PySpark"] ┆ 25000 ┆ 2500     ┆ 40days   │
│ ["Pandas", "Pandas"]   ┆ 30000 ┆ 2000     ┆ 50days   │
└────────────────────────┴───────┴──────────┴──────────┘

Here,

• repeat_by(2): This repeats the values in the Courses column twice, while the other columns (Fees, Discount, Duration) are untouched.
• The number of rows doubles, with the Courses column repeating values but other columns retaining their original values.

Alternatively, to repeat values in the "Courses" column using repeat_by(2) and then explode the repeated values. However, the explode method is used to flatten lists or arrays into individual rows.

# Repeat the "Courses" column twice, then explode
df2 = df.with_columns(pl.col("Courses").repeat_by(2)).explode("Courses")
print("DataFrame with 'Courses' Repeated and Exploded:\n", df2)

# Output:
# DataFrame with 'Courses' Repeated and Exploded:
# shape: (6, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Discount ┆ Duration │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ i64      ┆ str      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 1000     ┆ 30days   │
│ Spark   ┆ 22000 ┆ 1000     ┆ 30days   │
│ PySpark ┆ 25000 ┆ 2500     ┆ 40days   │
│ PySpark ┆ 25000 ┆ 2500     ┆ 40days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
└─────────┴───────┴──────────┴──────────┘

Here,

• repeat_by(2): Repeats each value in the Courses column twice.
• explode("Courses"): Flattens the repeated values so that they are displayed in separate rows, while keeping other columns intact.

<a id="h-repeat-values-in-a-column-using-map-elements-and-explode"></a>

Repeat Values in a Column Using map_elements() and explode()

To repeat values in a column using map_elements() and explode() in Polars, you can first map each value in the column to a list of repeated values, then apply the explode() method to flatten these lists into separate rows.

# Repeat values in the 'Courses' column twice and then explode the lists
df2 = df.with_columns(
    pl.col("Courses")
    .map_elements(lambda x: [x, x])  # Repeat each value twice
    .alias("Courses")  # Create the new 'Courses' column
).explode("Courses")  # Flatten the lists into individual rows
print("DataFrame with 'Courses' Repeated and Exploded:\n", df2)

# Output:
# DataFrame with 'Courses' Repeated and Exploded:
# shape: (6, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Discount ┆ Duration │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ i64      ┆ str      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 1000     ┆ 30days   │
│ Spark   ┆ 22000 ┆ 1000     ┆ 30days   │
│ PySpark ┆ 25000 ┆ 2500     ┆ 40days   │
│ PySpark ┆ 25000 ┆ 2500     ┆ 40days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
└─────────┴───────┴──────────┴──────────┘

Here,

• map_elements(lambda x: [x, x]): This function is applied to each element in the Courses column, repeating each value twice, creating a list with two elements.
• explode("Courses"): This flattens the lists into individual rows, duplicating the values from the Courses column while keeping the other columns the same.

<a id="h-create-duplicate-rows-based-on-conditional-logic"></a>

Create Duplicate Rows Based on Conditional Logic

To create duplicate rows based on conditional logic in Polars, you can use map_elements() to conditionally repeat a row’s value into a list (based on your logic), and then use explode() to create the duplicate rows.

# Apply conditional duplication
df_cond_dup = df.with_columns(
    pl.col("Courses")
    .map_elements(lambda x: [x, x] if x == "Pandas" else [x])
    .alias("Courses")
).explode("Courses")
print("\nDataFrame After Conditional Duplication:\n", df_cond_dup)

# Output:
# DataFrame After Conditional Duplication:
# shape: (4, 4)
┌─────────┬───────┬──────────┬──────────┐
│ Courses ┆ Fees  ┆ Discount ┆ Duration │
│ ---     ┆ ---   ┆ ---      ┆ ---      │
│ str     ┆ i64   ┆ i64      ┆ str      │
╞═════════╪═══════╪══════════╪══════════╡
│ Spark   ┆ 22000 ┆ 1000     ┆ 30days   │
│ PySpark ┆ 25000 ┆ 2500     ┆ 40days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
│ Pandas  ┆ 30000 ┆ 2000     ┆ 50days   │
└─────────┴───────┴──────────┴──────────┘

Here,

• map_elements(): Applies logic per row. If "Pandas", returns [x, x]; else [x].
• explode(): Turns lists in the column into separate rows, duplicating others accordingly.

<a id="h-conclusion"></a>

Conclusion

In conclusion, Polars provides efficient methods for creating duplicate rows in a DataFrame using functions like repeat_by() and concat(). This technique is powerful and can be applied for tasks such as data augmentation, performing specific data manipulations, or replicating data for further processing.

Happy Learning!!

<a id="h-related-articles"></a>

Related Articles

• Strip Entire Polars DataFrame
• How to Update the Polars DataFrame
• Mapping a Python Dict to a Polars Series
• Polars Counting Elements in List Column
• Convert Polars Casting a Column to Decimal
• How to Convert Struct to Series in Polars?
• How to Remove Duplicate Columns in Polars?
• Conditional Assignment in Polars DataFrame
• Check if any Value in a Polars DataFrame is True
• How to Transform a Series of a Polars DataFrame?
• Add a New Column into an Existing Polars DataFrame

<a id="h-reference"></a>

Reference

• https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.is_empty.html