Explain PySpark first() Function with Examples

The first() function in PySpark is an aggregate function that returns the first element of a column or expression, based on the specified order. It is commonly used with groupBy() or in queries where you need the first occurrence of a value from a group of rows.

This function is useful in scenarios such as:

Fetching the first value in a dataset.
Getting the earliest record per group.
Simplifying arrays or aggregates to just the first element.

In this article, we’ll explore:

What is the PySpark first() function.
Syntax and parameters.
Examples with groupBy().
Using ignorenulls.
Frequently Asked Questions (FAQs).

Key Points-

The first() function is an aggregate function in PySpark used to return the first element of a column.
It belongs to the pyspark.sql.functions module.
By default, if the first element of a column is NULL, it will return NULL.
The ignorenulls parameter can be set to True to skip null values and return the first non-null element.
It is commonly used with groupBy() to fetch the first value from each group.
The function returns a value with the same data type as the input column.
If no non-null values exist in a column, the result will still be NULL even with ignorenulls=True.
first() does not guarantee order unless combined with orderBy().
It is useful in summarizing datasets where only the initial or earliest value from a group is required.
When applied to an empty DataFrame, first() will return None

PySpark first() Function

The first() function, available in the pyspark.sql.functions module, returns the first element of a column. If the first value is NULL, it will return that NULL by default, but you can skip nulls by setting the ignorenulls parameter to True.

Syntax of PySpark first()

The syntax of the first() function.


# Syntax of the first()
pyspark.sql.functions.first(col, ignorenulls=False)

Parameters

col: The column name or expression on which the function will operate.
ignorenulls (optional): A boolean flag
- False (default): Considers null values. If the first value is null, it returns null.
- True: Skips null values and returns the first non-null value.

Return Value

Returns the first value in the specified column, with the same data type as the input column. It can return NULL if no non-null values exist (when ignorenulls=True).

Create Sample DataFrame

Create sample DataFrame having null values and use this to demonstrate how first() works with and without null values.


# Create dataFrame
from pyspark.sql import SparkSession
from pyspark.sql.functions import first

# Initialize Spark session
spark = SparkSession.builder.appName("FirstFunctionExample").getOrCreate()

# Sample data
data = [
    ("James", "Sales", 3000),
    ("Michael", "Sales", 4600),
    ("Robert", "Sales", 4100),
    ("Maria", "Finance", None),
    ("Jen", "Finance", 2000),
    ("Jeff", "Marketing", 3000)
]

columns = ["employee_name", "department", "salary"]

df = spark.createDataFrame(data, columns)
df.show()

Yields below output.

Using first() with groupBy()

You can apply the first() function in combination with groupBy() to retrieve the first value of a column within each group. In this example, we group the data by a specific column and use first() on another column to fetch the first value in each group. Since ignorenulls is not specified, null values will be included in the resultant DataFrame.


# Use first() with groupBy()
df.groupBy("department").agg(first("salary")).show()

Yields below the output.

Using first() Without groupBy()

You can also use first() function directly on a DataFrame column (without groupBy()), it behaves as a global aggregate function.


# Apply first() directly on the DataFrame
df.select(first("salary")).show()

Yields below the output.


# Output:
+-------------+
|first(salary)|
+-------------+
|         3000|
+-------------+

Using first() with ignorenulls=True

You can handle null values by using the first() function with the ignorenulls=True option, which skips nulls when retrieving the first element. By applying first() on grouped data, Spark returns the next available non-null value from the chosen column.


# Handle null values with ignorenulls
df.groupBy("department").agg(first("salary", ignorenulls=True)).show()

Yields below output.


# Output:
+----------+-------------+
|department|first(salary)|
+----------+-------------+
|     Sales|         3000|
|   Finance|         2000|
| Marketing|         3000|
+----------+-------------+

Difference Between first() and first_value()

The following table shows the difference between first() and first_value().

| Feature | first() | first_value() | | --- | --- | --- | | Type | Aggregate function | Window function | | Module | pyspark.sql.functions | pyspark.sql.functions | | Usage | Used with groupBy() or as an aggregate to fetch the first element of a column or group | Used with Window specifications to get the first value within an ordered partition | | Ordering | Does not guarantee order unless combined with .orderBy() | Requires .orderBy() for deterministic results | | Null Handling | Supports ignorenulls parameter | Supports ignorenulls parameter | | Scope | Operates on entire DataFrame or groups | Operates row-by-row within partitions or globally with Window | | Return Type | Same data type as input column | Same data type as input column |

Frequently Asked Questions of PySpark first() Function

Both first() and head(1) return the first row of a DataFrame. However, first() is simpler and directly returns the first row, whereas head(n) can return multiple rows if you specify n.

first() cannot be applied with conditions directly. If you want the first row that meets a condition, filter the DataFrame first and then apply first().

By itself, first() does not guarantee order since DataFrames are unordered. To get consistent results, you must apply .orderBy() before calling first().

If the DataFrame is empty, first() will return None. Always handle this case to avoid runtime errors.

You can use first() inside agg() with groupBy() to fetch the first value per group. This is useful when summarizing grouped data.

Conclusion

In this article, I have explored the PySpark first() function with syntax, parameters, and examples. We saw how it works with groupBy() and how the ignorenulls parameter helps skip null values.

By combining first() with grouping and ordering, you can easily fetch the first meaningful value in real-world datasets and simplify aggregation tasks in PySpark pipelines.

Happy Learning!!

References

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.first.html