Explain PySpark last_value() Function with Examples

In PySpark, the last_value() function is a window function that returns the last value in an ordered partition of data. Unlike the last() aggregate function, which works on grouped data, last_value() operates over a Window specification, giving you row-wise control across partitions.

This makes last_value() especially useful in analytics where you need the latest value relative to each row within a group.

In this article, we’ll explore

What is the PySpark last_value() function
Syntax and parameters
Return value
Usage with and without partitions
Handling nulls using ignorenulls
Frequently Asked Questions (FAQs)
Key points

Key Points-

last_value() is a window function that returns the last value in a partition based on a specified order.
It must always be used with a window specification.
Without defining a frame, it only looks at the current row, not the full partition.
To fetch the true last row in a partition, use .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing).
It works with partitionBy() to calculate results separately for each group.
The result depends on the column used in orderBy(), as it defines what counts as the “last” value.
By default, null values are considered valid and can appear as the last value.
The ignoreNulls parameter can be set to True to skip null values and return the last non-null entry.
It differs from the last() aggregate function, which is used in groupBy aggregations, not with windowing.
It is especially useful in time-series analysis, ordered datasets, and when you need the most recent record within a group.

PySpark last_value() Function

last_value() is a window function in PySpark. It returns the last value in the window frame defined by the Window specification. It is part of the pyspark.sql.functions module. It is often used along with Window.partitionBy() and Window.orderBy() to compute results within partitions (groups) or across the entire DataFrame

Syntax of PySpark last_value()

Following is the syntax of the last_value() function.


# Syntax of the last_value()
# Syntax
pyspark.sql.functions.last_value(col, ignorenulls=False)

Parameters

col: Column name or expression.
ignorenulls (optional):
- False (default): Considers null values.
- True: Skips null values and returns the last non-null value.

Return Value

Returns a column containing the last value in the window frame, with the same data type as the input column.

Create Sample DataFrame

Let’s create a DataFrame with salaries, including null values.


# Create dataFrame
from pyspark.sql.window import Window
from pyspark.sql import SparkSession
from pyspark.sql.functions import last_value

# Initialize Spark session
spark = SparkSession.builder.appName("LastFunctionExample").getOrCreate()

# Sample data
data = [
    ("James", "Sales", 3000),
    ("Maria", "Finance", 2000),
    ("Robert", "Sales", 4100),
    ("Michael", "Sales", 4600),
    ("Jen", "Finance", None),
    ("Jeff", "Marketing", 3000)
]

columns = ["employee_name", "department", "salary"]

df = spark.createDataFrame(data, columns)
df.show()

Yields below output.

Use last_value() With Partitioning

We can select the last value from each group using the PySpark DataFrame API.
In this section, we will see how to use the window function last_value() with partitionBy().

Let’s create a DataFrame and define a Window specification with partitionBy() and orderBy() to get the last value of each group based on the specified ordering.


# Define window by department with ordering on salary
windowSpec = Window.partitionBy("department").orderBy("salary")\
    .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)

df.withColumn(
    "last_value_salary",
    last_value("salary").over(windowSpec)
).show()

Yields below the output.

Use last_value() With ignoreNulls=True

We can select the first non-null value from each group by using last_value() with the ignoreNulls parameter.


# # Use last_value with ignorenulls
df.withColumn(
    "last_value_salary_ignoreNulls",
    last_value("salary", True).over(windowSpec)
).show()

Yields below the output.


# Output:
+-------------+----------+------+-----------------------------+
|employee_name|department|salary|last_value_salary_ignoreNulls|
+-------------+----------+------+-----------------------------+
|          Jen|   Finance|  NULL|                         2000|
|        Maria|   Finance|  2000|                         2000|
|         Jeff| Marketing|  3000|                         3000|
|        James|     Sales|  3000|                         4600|
|       Robert|     Sales|  4100|                         4600|
|      Michael|     Sales|  4600|                         4600|
+-------------+----------+------+-----------------------------+

last_value() Without Partitioning

We can also use last_value() without partitioning, in this case, it returns the last value from the entire DataFrame after ordering.


# Window without partition, ordered by salary
windowSpec = Window.orderBy("salary") \
    .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)

df.withColumn(
    "last_value_global",
    last_value("salary", True).over(windowSpec)   # ignoreNulls=True
).show()

Yields below output.


# Output:
+-------------+----------+------+-----------------+
|employee_name|department|salary|last_value_global|
+-------------+----------+------+-----------------+
|          Jen|   Finance|  NULL|             4600|
|        Maria|   Finance|  2000|             4600|
|        James|     Sales|  3000|             4600|
|         Jeff| Marketing|  3000|             4600|
|       Robert|     Sales|  4100|             4600|
|      Michael|     Sales|  4600|             4600|
+-------------+----------+------+-----------------+

Difference Between last() and last_value()

The following table shows the difference between last() and last_value().

| Feature | last() (Aggregate Function) | last_value() (Window Function) | | --- | --- | --- | | Type | Aggregate function | Window function | | Module | pyspark.sql.functions | pyspark.sql.functions | | Usage | Used with groupBy() or as an aggregate to fetch the last element of a column or group | Used with Window specifications to get the last value within an ordered partition | | Ordering | Does not guarantee order unless combined with .orderBy() | Requires .orderBy() for deterministic results | | Null Handling | Supports ignorenulls parameter | Supports ignorenulls parameter | | Scope | Operates on entire DataFrame or groups | Operates row-by-row within partitions or globally with Window | | Return Type | Same data type as input column | Same data type as input column |

Frequently Asked Questions of PySpark last_value() Function

last(): Aggregate function → works on grouped data.

By default, it considers nulls. If the last row is null, the result is null.
Use ignorenulls=True to skip nulls.

You can define a window with only .orderBy() and apply last_value() to the entire dataset.

Use it when you need the latest value within a partition or the global last value across the dataset.

Conclusion

In this article, we explored the PySpark last_value() function, covering its syntax, parameters, and different use cases. We demonstrated how to retrieve the last value with and without partitions based on a specified ordering, and also explained how to handle null values using the ignoreNulls parameter. Finally, we provided a quick comparison between the last() and last_value() functions.

Happy learning!!

References

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.last_value.html