In PySpark, both first() and first_value() are used to retrieve the first element of a column. They might look similar, which often leads to confusion among beginners. However, their functionality and use cases are quite different.
In this article, I will break down the key differences between the two, covering their syntax, parameters, and practical examples. We’ll also explore when to use first() and when first_value() is the better choice.
Difference Between first() and first_value()
The following table shows the difference between first() and first_value().
| Feature | first() | first_value() |
| --- | --- | --- |
| Type | Aggregate function | Window function |
| Module | pyspark.sql.functions | pyspark.sql.functions |
| Usage | Used with groupBy() or as an aggregate to fetch the first element of a column or group | Used with Window specifications to get the first value within an ordered partition |
| Ordering | Does not guarantee order unless combined with .orderBy() | Requires .orderBy() for deterministic results |
| Null Handling | Supports ignorenulls parameter | Supports ignorenulls parameter |
| Scope | Operates on entire DataFrame or groups | Operates row-by-row within partitions or globally with Window |
| Return Type | Same data type as input column | Same data type as input column |
<a id="h-pyspark-first-function"></a>
PySpark first() Function
The first() function is an aggregate function. It is typically used with groupBy() or on a full DataFrame to return the first element in a group.
<a id="h-syntax"></a>
Syntax
Following is the Syntax of the first() function.
# Syntax of the first()
pyspark.sql.functions.first(col, ignorenulls=False)
<a id="h-parameters"></a>
Parameters
col:The column name or expression on which the function will operate.ignorenulls (optional):A boolean flagFalse(default): Considers null values. If the first value is null, it returns null.True: Skips null values and returns the first non-null value.
<a id="h-return-value"></a>
Return Value
Returns the first value in the specified column, with the same data type as the input column. It can return NULL if no non-null values exist (when ignorenulls=True).
<a id="h-using-first-with-groupby"></a>
Using first() with groupBy()
You can apply the first() function in combination with groupBy() to retrieve the first value of a column within each group. In this example, we group the data by a specific column and use first() on another column to fetch the first value in each group. Since ignorenulls is not specified, null values will be included in the resultant DataFrame.
# # Use first() with groupBy()
from pyspark.sql import SparkSession
from pyspark.sql.functions import first
spark = SparkSession.builder.appName("FirstFunctionExample").getOrCreate()
data = [
("James", "Sales", 3000),
("Michael", "Sales", 4600),
("Robert", "Sales", 4100),
("Maria", "Finance", None),
("Jen", "Finance", 2000),
("Jeff", "Marketing", 3000)
]
columns = ["employee_name", "department", "salary"]
df = spark.createDataFrame(data, columns)
# Use first() with groupBy()
df.groupBy("department").agg(first("salary")).show()
Yields below the output.
<a id="h-using-first-without-groupby"></a>
Using first() Without groupBy()
You can also use first() function directly on a DataFrame column (without groupBy()), it behaves as a global aggregate function.
# Apply first() directly on the DataFrame
df.select(first("salary")).show()
Yields below the output.
# Output:
+-------------+
|first(salary)|
+-------------+
| 3000|
+-------------+
<a id="h-using-first-with-ignorenulls-true"></a>
Using first() with ignorenulls=True
You can handle null values by using the first() function with the ignorenulls=True option, which skips nulls when retrieving the first element. By applying first() on grouped data, Spark returns the next available non-null value from the chosen column.
# Handle null values with ignorenulls
df.groupBy("department").agg(first("salary", ignorenulls=True)).show()
Yields below output.
# Output:
+----------+-------------+
|department|first(salary)|
+----------+-------------+
| Sales| 3000|
| Finance| 2000|
| Marketing| 3000|
+----------+-------------+
<a id="h-pyspark-first-value-function"></a>
PySpark first_value() Function
The first_value() function belongs to the pyspark.sql.functions module. It returns the first value of a column in an ordered group of rows. It is often used along with Window.partitionBy() and Window.orderBy() to compute results within partitions (groups) or across the entire DataFrame.
<a id="h-syntax-of-pyspark-first-value"></a>
Syntax of PySpark first_value()
The syntax of the first_value() function.
# Syntax of first_value()
pyspark.sql.functions.first_value(col, ignorenulls=False)
<a id="h-parameters"></a>
Parameters
col:The column or expression on which the function operates.ignorenulls (optional):Boolean flag to control null handling.False (default):Considers null values. If the first value is null, it returns null.True:Ignores nulls and returns the first non-null value.
Return Value
The function returns the first value in the window partition.
- The result has the same data type as the input column.
- If all values are null in the partition, the result is null (even if
ignorenulls=True).
<a id="h-use-first-value-with-partitioning"></a>
Use first_value() With Partitioning
We can select the first value from each group using the PySpark DataFrame API. In this section, we will see how to use the window function first_value() with partitionBy(). Let's create a DataFrame and define a window specification with partitionBy() and orderBy() to get the first value of each group based on the specified ordering.
# Create dataFrame
from pyspark.sql import SparkSession
from pyspark.sql.functions import first_value
from pyspark.sql.window import Window
# Initialize Spark session
spark = SparkSession.builder.appName("FirstValueExample").getOrCreate()
# Sample data
data = [
("James", "Sales", 3000),
("Michael", "Sales", 4600),
("Robert", "Sales", 4100),
("Maria", "Finance", None),
("Jen", "Finance", 3000),
("Jeff", "Marketing", 3000)
]
columns = ["employee_name", "department", "salary"]
# Assign DataFrame to df
df = spark.createDataFrame(data, columns)
# Only call show() to display
df.show()
# Define Window
windowSpec = Window.partitionBy("department").orderBy("salary")
# Apply first_value
df.withColumn("first_salary", first_value("salary").over(windowSpec)).show()
Yields below output.
<a id="h-use-first-value-with-ignorenulls"></a>
Use first_value with ignorenulls
We can select the first non-null value from each group by using first_value() with the ignoreNulls parameter.
# Use first_value with ignorenulls
df.withColumn(
"first_salary_non_null",
first_value("salary", True).over(windowSpec)
).show()
Yields below output.
<a id="h-use-first-value-without-partitioning"></a>
Use first_value() Without Partitioning
We can also use first_value() without partitioning, in this case, it returns the first value from the entire DataFrame after ordering.
# Define Window WITHOUT partitioning
windowSpec = Window.orderBy("salary")
# Apply first_value
df.withColumn("first_salary_global", first_value("salary").over(windowSpec)).show()
Yields below output.
# Output:
+-------------+----------+-----+------------------+
| employee_name|department|salary|first_salary_global|
+-------------+----------+-----+------------------+
| James | Sales | 3000 | 3000 |
| Jen | Finance | 3000 | 3000 |
| Jeff | Marketing| 3000 | 3000 |
| Robert | Sales | 4100 | 3000 |
| Michael | Sales | 4600 | 3000 |
| Maria | Finance | null | 3000 |
+-------------+----------+-----+------------------+
<a id="h-when-to-use-first-vs-first-value"></a>
When to Use first() vs first_value()
- You want row-level operations where the first value is repeated for all rows in the partition
- Use
first()when: - You’re working with aggregates like
groupBy() - You want the first element in each group without defining a window
- Use
first_value()when: - You’re performing window-based analytics
- You need the first ordered value within each partition or globally
<a id="h-conclusion"></a>
Conclusion
In this article, I have broken down the differences between first() and first_value() in PySpark. Both functions are used to retrieve the first element from data, but they differ in scope and application:
first()is an aggregate function, best suited for grouping and summarization.first_value()is a window function, ideal for analytics within partitions or ordered datasets.
By understanding these differences, you can confidently choose the right function for your PySpark pipelines and ensure accurate, meaningful results.
Happy Learning!!