Explain PySpark last() Function with Examples

The last() function in PySpark is an aggregate function that returns the last value from a column or expression. Unlike first(), which returns the beginning value, last() focuses on retrieving the most recent or last occurrence within a dataset or group.

This function is especially helpful in use cases such as:

Getting the last available record in a dataset.
Retrieving the latest value from each group (e.g., last salary per department).
Simplifying aggregated results when only the most recent entry is needed.

In this article, we’ll explore

What is the PySpark last() function.
Syntax and parameters.
Examples with groupBy().
Using ignorenulls.
Difference between last() and last_value().
Frequently Asked Questions (FAQs).

Key Points-

The last() function is an aggregate function in PySpark used to return the last element of a column.
It belongs to the pyspark.sql.functions module.
By default, if the last element of a column is NULL, it will return NULL.
The ignorenulls parameter can be set to True to skip null values and return the last non-null element.
It is commonly used with groupBy() to fetch the last value from each group.
The function returns a value with the same data type as the input column.
If no non-null values exist in a column, the result will still be NULL even with ignorenulls=True.
last() does not guarantee order unless combined with .orderBy().
Useful for summarizing datasets where the latest or final value from a group is required.
When applied to an empty DataFrame, last() will return None.

PySpark last() Function

The last() function is a part of pyspark.sql.functions module, returns the last element of a column. If the last value is NULL, it will return that NULL by default. By avoiding this, you can set the ignorenulls parameter to True it will return the latest non-Null value. If all values are null, then null is returned.

Syntax of PySpark last()

Following is the syntax of the last() function.


# Syntax of the last()
pyspark.sql.functions.last(col, ignorenulls=False)

Parameters

col: The column name or expression on which the function will operate.
ignorenulls (optional): A boolean flag
- False (default): Considers null values. If the last value is null, it returns null.
- True: Skips null values and returns the last non-null value.

Return Value

Returns the last value in the specified column, preserving the same data type as the input. By default, it returns NULL if the last position contains a null value. However, when ignorenulls=True is set, it skips nulls and returns the most recent non-null value.

Create Sample DataFrame

Let’s create a sample DataFrame with null values to demonstrate how last() works.


# Create dataFrame
from pyspark.sql import SparkSession
from pyspark.sql.functions import last

# Initialize Spark session
spark = SparkSession.builder.appName("LastFunctionExample").getOrCreate()

# Sample data
data = [
    ("James", "Sales", 3000),
    ("Maria", "Finance", 2000),
    ("Robert", "Sales", 4100),
    ("Michael", "Sales", 4600),
    ("Jen", "Finance", None),
    ("Jeff", "Marketing", 3000)
]

columns = ["employee_name", "department", "salary"]

df = spark.createDataFrame(data, columns)
df.show()

Yields below output.

Using last() with groupBy()

You can apply the last() function in combination with groupBy() to retrieve the last value of a column within each group.


# Use last() with groupBy()
df.groupBy("department").agg(last("salary")).show()

Yields below the output.

Using last() Without groupBy()

You can also use last() directly on a DataFrame column (without groupBy()), and it behaves as a global aggregate function.


# Apply last() directly on the DataFrame
df.select(last("salary")).show()

Yields below the output.


# Output:
+------------+
|last(salary)|
+------------+
|        3000|
+------------+

Using last() with ignorenulls=True

You can handle null values by using the last() function with the ignorenulls=True option, which skips nulls when retrieving the last element. By applying last() on grouped data, Spark returns the last available non-null value from the chosen column.


# Handle null values with ignorenulls
df.groupBy("department").agg(last("salary", ignorenulls=True)).show()

Yields below output.


# Output:
+----------+------------+
|department|last(salary)|
+----------+------------+
|     Sales|        4600|
|   Finance|        2000|
| Marketing|        3000|
+----------+------------+

Difference Between last() and last_value()

The following table shows the difference between last() and last_value().

| Feature | last() (Aggregate Function) | last_value() (Window Function) | | --- | --- | --- | | Type | Aggregate function | Window function | | Module | pyspark.sql.functions | pyspark.sql.functions | | Usage | Used with groupBy() or as an aggregate to fetch the last element of a column or group | Used with Window specifications to get the last value within an ordered partition | | Ordering | Does not guarantee order unless combined with .orderBy() | Requires .orderBy() for deterministic results | | Null Handling | Supports ignorenulls parameter | Supports ignorenulls parameter | | Scope | Operates on entire DataFrame or groups | Operates row-by-row within partitions or globally with Window | | Return Type | Same data type as input column | Same data type as input column |

Frequently Asked Questions of PySpark last() Function

PySpark does not have a direct tail() function like Pandas. To get the last record, you typically use .orderBy() and .collect(). The last() aggregate is different—it works on groups, not row positions.

You cannot apply last() with conditions directly. Instead, filter the DataFrame first and then apply last().

By itself, last() does not guarantee order since DataFrames are unordered. To get consistent results, you must apply .orderBy() before calling last().

If the DataFrame is empty, last() will return None. Handle this case to avoid runtime errors.

You can use last() inside agg() with groupBy() to fetch the last value per group, which is helpful when summarizing grouped data.

Conclusion

In this article, we explored the PySpark last() function with syntax, parameters, and practical examples. We saw how it works with groupBy() and how the ignorenulls parameter helps to skip null values.

By combining last() with grouping and ordering, you can fetch the last meaningful value in real-world datasets, making it a useful function for summarizing and aggregation in PySpark pipelines.

If you need row-wise operations with ordering control, consider using the last_value() window function instead. Together, these functions make it easy to capture both earliest and latest values in your Spark workflows.

Happy learning!!

References

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.last.html