Built with Tutify.io

    AIStudyLabs

    Polars DataFrame shift() Usage & Examples

    Polars DataFrame shift() Usage & Examples

    In Polars, the shift() function is used to shift the values in a column up or down by a specified number of positions, filling the empty positions with null or a given value. This is particularly useful for computing row-wise differences or aligning time series data.

    In this article, I will explain the Polars DataFrame shift() function by using its syntax, parameters, and usage to generate a new DataFrame with all columns shifted by n positions.

    Key Points –

    <a id="h-polars-dataframe-shift-introduction"></a>

    Polars DataFrame shift() Introduction

    Let’s know the syntax of the shift() method.

    
# Syntax of shift()
DataFrame.shift(n: int = 1, *, fill_value: IntoExpr | None = None) → DataFrame

    <a id="h-parameters-of-the-polars-dataframe-shift"></a>

    Parameters of the Polars DataFrame shift()

    Following are the parameters of the DataFrame shift() method.

    <a id="h-return-value"></a>

    Return Value

    This function returns a new DataFrame with all columns shifted by n positions.

    <a id="h-usage-of-polars-dataframe-shift-method"></a>

    Usage of Polars DataFrame shift() Method

    The shift() method in Polars is used to move values within a DataFrame up or down by a specified number of positions while filling the empty spaces with null or a custom value.

    Now, Let’s create a Polars DataFrame.

    
import polars as pl

data = {'Column1': [2, 4, 6, 8, 10],
        'Column2': [3, 5, 7, 9, 11]}
df = pl.DataFrame(data)
print("Original DataFrame:\n",df)

    Yields below output.

    You can use the shift() method to move all column values down by 1 position (default behavior).

    
# Shift all columns down by 1 (default)
df2 = df.shift(1)
print("Shifted DataFrame:\n", df2)

    Here,

    <a id="h-shifting-all-columns-up-n-1"></a>

    Shifting All Columns Up (n=-1)

    Alternatively, to shift all columns up by 1 position in a Polars DataFrame, use the shift() function with n=-1.

    
# Shift all columns up by 1 position
df2 = df.shift(-1)
print("Shifted DataFrame (Up by 1):\n", df2)

# Output:
# Shifted DataFrame (Up by 1):
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ 4       ┆ 5       │
│ 6       ┆ 7       │
│ 8       ┆ 9       │
│ 10      ┆ 11      │
│ null    ┆ null    │
└─────────┴─────────┘

    Here,

    <a id="h-shift-with-a-custom-fill-value"></a>

    Shift with a Custom Fill Value

    To shift all columns in a Polars DataFrame while replacing null values with a custom fill value, use the fill_value parameter in the shift() function.

    
# Shift all columns down by 1 and fill nulls with 0
df2 = df.shift(1, fill_value=0)
print("Shifted DataFrame with Custom Fill Value:\n", df2)

# Output:
# Shifted DataFrame with Custom Fill Value:
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ 0       ┆ 0       │
│ 2       ┆ 3       │
│ 4       ┆ 5       │
│ 6       ┆ 7       │
│ 8       ┆ 9       │
└─────────┴─────────┘

    Here,

    <a id="h-shift-only-a-specific-column"></a>

    Shift Only a Specific Column

    To shift only a specific column in a Polars DataFrame, use the with_columns() method along with shift() function.

    
# Shift only 'Column1' down by 2
df2 = df.with_columns(df["Column1"].shift(2))
print("Shifted DataFrame (Only 'Column1' shifted):\n", df2)

# Output:
# Shifted DataFrame (Only 'Column1' shifted):
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ null    ┆ 3       │
│ null    ┆ 5       │
│ 2       ┆ 7       │
│ 4       ┆ 9       │
│ 6       ┆ 11      │
└─────────┴─────────┘

    Here,

    <a id="h-shift-with-negative-n-and-fill-value"></a>

    Shift with Negative n and Fill Value

    Similarly, to shift a column up (negative n) while replacing null values with a custom fill value, use the shift() function with fill_value.

    
# Shift all columns up by 1 and fill missing values with -1
df2 = df.shift(-1, fill_value=-1)
print("Shifted DataFrame with Negative Shift and Custom Fill Value:\n", df2)

# Output:
# Shifted DataFrame with Negative Shift and Custom Fill Value:
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ 4       ┆ 5       │
│ 6       ┆ 7       │
│ 8       ┆ 9       │
│ 10      ┆ 11      │
│ -1      ┆ -1      │
└─────────┴─────────┘

    Here,

    <a id="h-compute-differences-using-shift"></a>

    Compute Differences Using shift()

    Finally, to compute differences between consecutive rows in a Polars DataFrame, use shift() to align values and subtract.

    
# Compute differences by subtracting the shifted DataFrame
df2 = df.with_columns((df - df.shift(2)))
print("DataFrame with Differences:\n", df2)

# Output:
# DataFrame with Differences:
# shape: (5, 2)
┌─────────┬─────────┐
│ Column1 ┆ Column2 │
│ ---     ┆ ---     │
│ i64     ┆ i64     │
╞═════════╪═════════╡
│ null    ┆ null    │
│ null    ┆ null    │
│ 4       ┆ 4       │
│ 4       ┆ 4       │
│ 4       ┆ 4       │
└─────────┴─────────┘

    Here,

    <a id="h-conclusion"></a>

    Conclusion

    In conclusion, the shift() function in Polars is a powerful tool for manipulating DataFrame values by shifting them up or down. It is commonly used in time-series analysis, calculating differences, creating lagged values, and aligning data efficiently.

    Happy Learning!!

    <a id="h-related-articles"></a>

    Related Articles

    <a id="h-reference"></a>

    Reference