Built with Tutify.io

    AIStudyLabs

    Pandas DataFrame diff() Method

    Pandas DataFrame diff() Method

    In Pandas, the diff() method is used to calculate the difference between two successive rows or columns in a DataFrame. It is useful for finding changes or variations between data points over a specified axis.

    In this article, I will explain the pandas DataFrame diff() method by using its syntax, parameters, and usage, we can generate a DataFrame or Series of the same shape with the calculated differences. The first row or column (depending on the axis) will have NaN values due to the absence of preceding elements for comparison.

    Key Points –

    <a id="h-syntax-of-pandas-nbsp-dataframe-diff-method"></a>

    Syntax of Pandas DataFrame diff() Method

    Following is the syntax of the pandas DataFrame diff() method.

    
# Syntax of the DataFrame diff() method
DataFrame.diff(periods=1, axis=0)

    <a id="h-parameters-of-the-dataframe-diff"></a>

    Parameters of the DataFrame diff()

    Following are the parameters of the DataFrame diff() method.

    <a id="h-return-value"></a>

    Return Value

    It returns a DataFrame of the same shape as the input, containing the calculated differences.

    <a id="h-usage-of-pandas-dataframe-diff-method"></a>

    Usage of Pandas DataFrame diff() Method

    The diff() method in Pandas is used to compute the difference between successive rows or columns in a DataFrame.

    To run some examples of the Pandas DataFrame diff() method, let’s create a Pandas DataFrame using data from a dictionary.

    
import pandas as pd

# Create a DataFrame
df = pd.DataFrame({
    'A': [5, 8, 13, 15, 19],
    'B': [2, 6, 9, 12, 16]
})

print("Original DataFrame:\n",df)

    Yields below output.

    For default usage of the diff() method, which computes the row-wise difference in a DataFrame. By default, diff() calculates the difference between the current row and the previous row within each column. This is done along axis=0, which means the differences are computed row-wise.

    
# Compute row-wise differences
df2 = df.diff()
print("Row-wise difference (default):\n", df2)

    Here,

    <a id="h-specifying-the-number-of-periods"></a>

    Specifying the Number of Periods

    Alternatively, to specify the number of periods for calculating differences in a DataFrame, you can use the periods parameter in the diff() method. This allows you to calculate the difference between the current row and the row that is a specified number of periods before it.

    
# Compute differences with periods=2
df2 = df.diff(periods=2)
print("Row-wise difference with periods=2:\n", df2)

# Output:
# Row-wise difference with periods=2:
#      A    B
# 0  NaN  NaN
# 1  NaN  NaN
# 2  8.0  7.0
# 3  7.0  6.0
# 4  6.0  7.0

    Here,

    <a id="h-column-wise-difference"></a>

    Column-wise Difference

    To calculate the column-wise difference in a DataFrame, you can use the diff() method with the axis=1 parameter. This will compute the difference between each element and the previous element in the same row across columns.

    
# Compute column-wise differences
df2 = df.diff(axis=1)
print("Column-wise difference:\n", df2)

# Output:
# Column-wise difference:
#     A    B
# 0 NaN -3.0
# 1 NaN -2.0
# 2 NaN -4.0
# 3 NaN -3.0
# 4 NaN -3.0

    Here,

    <a id="h-handling-nan-values"></a>

    Handling NaN Values

    Similarly, when working with the diff() method, NaN (Not a Number) values can appear in several scenarios, such as at the beginning of the DataFrame where there are no preceding values to compute the difference, or if the DataFrame already contains NaN values.

    
# Compute row-wise differences
df2 = df_with_nan.diff()
print("Row-wise difference (default handling of NaN values):\n", df2)

# Output:
# Row-wise difference (default handling of NaN values):
#      A    B
# 0  NaN  NaN
# 1  3.0  4.0
# 2  NaN  3.0
# 3  NaN  NaN
# 4  4.0  NaN

    Here,

    <a id="h-filling-nan-values-before-applying-diff"></a>

    Filling NaN Values Before Applying diff()

    You can fill NaN values with a specific value before computing differences using the fillna() method.

    
# Fill NaN values with 0 and then compute differences
df2 = df_with_nan.fillna(0).diff()
print("Row-wise difference after filling NaN values with 0:\n", df2)

# Output:
# Row-wise difference after filling NaN values with 0:
#        A     B
# 0   NaN   NaN
# 1   3.0   4.0
# 2  -8.0   3.0
# 3  15.0  -9.0
# 4   4.0  16.0

    Here,

    <a id="h-dropping-nan-values-before-applying-diff"></a>

    Dropping NaN Values Before Applying diff()

    You can drop rows with NaN values before computing differences using the dropna() method.

    
# Drop NaN values and then compute differences
df2 = df_with_nan.dropna().diff()
print("Row-wise difference after dropping rows with NaN values:\n", df2)

# Output:
# Row-wise difference after dropping rows with NaN values:
#       A     B
# 0   NaN   NaN
# 1   3.0   4.0
# 4  11.0  10.0

    Here,

    <a id="h-forward-filling-nan-values"></a>

    Forward Filling NaN Values

    You can forward-fill NaN values using the ffill() method before applying diff().

    
# Forward fill NaN values and then compute differences
df2 = df_with_nan.ffill().diff()
print("Row-wise difference after forward filling NaN values:\n", df2)

# Output:
# Row-wise difference after forward filling NaN values:
#      A    B
# 0  NaN  NaN
# 1  3.0  4.0
# 2  0.0  3.0
# 3  7.0  0.0
# 4  4.0  7.0

    Here,

    <a id="h-frequently-asked-questions-on-pandas-dataframe-diff-method"></a>

    Frequently Asked Questions on Pandas DataFrame diff() Method

    The diff() method in Pandas is used to compute the difference between consecutive elements in a DataFrame or Series. By default, it calculates the difference between the current and the previous row.

    By default, diff() calculates the difference between each element and the previous element in the same column (row-wise difference) with a period of 1.

    You can specify the number of periods by using the periods parameter. For example, df.diff(periods=2) will compute the difference between the current row and the row two periods before.

    You can handle NaN values before or after applying diff() using methods like fillna(), dropna(), ffill() (forward fill), and bfill() (backward fill).

    By setting the axis parameter to 1 (i.e., df.diff(axis=1)), you can compute the difference between each element and the previous element in the same row (column-wise difference).

    <a id="h-conclusion"></a>

    Conclusion

    In conclusion, the pandas.DataFrame.diff() method is a powerful tool for calculating differences between consecutive elements in a DataFrame or Series. This functionality is particularly useful in time series analysis, identifying trends, and detecting changes over specified periods. By understanding its syntax, parameters, and various applications, you can effectively leverage this method to gain deeper insights into your data.

    Happy Learning!!

    <a id="h-related-articles"></a>

    Related Articles

    <a id="h-references"></a>

    References