How to check elements in the array columns of a PySpark DataFrame? PySpark provides two powerful higher-order functions, such as exists() and forall() to work with array columns. exists() function returns true if any element in the array satisfies the condition, whereas forall() returns true if all elements in the array satisfy the condition. For simple checks, the array_contains() function can be used to see if a specific value exists.
In this article, I will explain how to determine whether any element in a PySpark array meets a given condition using the exists() function, or if all elements in the array satisfy a condition using the forall() function. We’ll explore their syntax, practical examples with an array of strings.
- exists() → Returns
trueif any element in the array satisfies the condition. - forall() → Returns
trueif all elements satisfy the condition. - Both functions are part of higher-order array functions in PySpark SQL.
- You can use complex expressions (e.g., comparisons, LIKE, IS NULL) inside conditions.
- Useful for data validation, filtering, and feature engineering when working with array columns.
<a id="h-pyspark-exists-function"></a>
PySpark exists() Function
The exists() function is used to check if at least one element in the array satisfies a given condition.
If such an element is found, it returns true; otherwise, it returns false.
<a id="h-syntax-of-exists-function"></a>
Syntax of exists() Function
The following is the syntax of exists().
# Syntax of exists() Function
expr("exists(array_column, element -> condition)")
<a id="h-parameters"></a>
Parameters
array_column: The array column to process.element: A lambda variable representing each array element.condition: A boolean condition applied to each element.
<a id="h-return-value"></a>
Return Value
It returns True if “any” element of an array evaluates to True when passed as an argument to the given function and False otherwise.
<a id="h-sample-dataframe"></a>
Sample DataFrame
We’ll use the following DataFrame for all examples.
# Create DataFrame
from pyspark.sql import SparkSession
from pyspark.sql.types import StringType, ArrayType, StructType, StructField
from pyspark.sql.functions import expr
# Initialize Spark session
spark = SparkSession.builder.appName("ExistsForallExample").getOrCreate()
# Sample data
data = [
("James", ["Java", "Scala", "C++"]),
("Michael", ["Spark", "Java", "C++"]),
("Robert", ["CSharp", "VB"]),
("John", ["Java", None])
]
# Schema
schema = StructType([
StructField("name", StringType(), True),
StructField("languages", ArrayType(StringType()), True)
])
# Create DataFrame
df = spark.createDataFrame(data, schema)
df.printSchema()
df.show(truncate=False)
Yields below the output.
<a id="h-usage-of-exists"></a>
Usage of exists()
You can use the exists() function to determine whether at least one element in an array column matches a specific value. It returns True if the condition is satisfied by any element, and False otherwise.
# Check if any element in the array is 'Java'
df.select(
"name",
expr("exists(languages, x -> x = 'Java')").alias("has_java")
).show()
Yields below output.
In this example, we check if the languages array contains the value "Java".
Using exists() for Conditional Matching
You can use the exists() function to check whether at least one element in an array column matches a specified condition. It returns True if any element satisfies the condition, and False otherwise.
# Check if any element starts with 'S'
df.select(
"name",
expr("exists(languages, x -> x LIKE 'S%')").alias("starts_with_s")
).show()
# Output:
+-------+-------------+
| name|starts_with_s|
+-------+-------------+
| James| true|
|Michael| true|
| Robert| false|
| John| NULL|
+-------+-------------+
In this example, we verify if the languages array contains at least one value that starts with the letter 'S'.
<a id="h-pyspark-forall-function"></a>
PySpark forall() Function
The forall() function checks whether every element in an array column satisfies a given condition. It returns True if all elements meet the condition, and False otherwise.
<a id="h-syntax-of-the-forall-function"></a>
Syntax of the forall() Function
Following is the syntax of the forall() function.
# Syntax of forall()
expr("forall(array_column, element -> condition)")
<a id="h-parameters-0"></a>
Parameters
array_column: The array column to process.element: A lambda variable representing each array element.condition: A boolean condition applied to all elements.
<a id="h-return-value-0"></a>
Return Value
It returns True if “all” elements of an array evaluate to True when passed as an argument to the given function and False otherwise.
<a id="h-check-if-all-elements-are-non-null"></a>
Check If all Elements are Non-Null
You can use the forall() function to verify that all elements in an array column meet a specified condition. It returns True if every element satisfies the condition, and False otherwise. In this example, we check whether all elements in the languages array are non-null.
# Check if all elements are non-null
df.select(
"name",
expr("forall(languages, x -> x IS NOT NULL)").alias("all_not_null")
).show()
# Output:
+-------+------------+
| name|all_not_null|
+-------+------------+
| James| true|
|Michael| true|
| Robert| true|
| John| false|
+-------+------------+
<a id="h-combining-exists-and-forall"></a>
Combining exists() and forall()
You can combine the exists() and forall() functions to apply multiple conditions on an array column. The exists() function checks if at least one element matches the given condition, while the forall() function ensures that all elements satisfy another condition.
# Check if array has 'Java' and all elements are not null
df.select(
"name",
expr("exists(languages, x -> x = 'Java') AND forall(languages, x -> x IS NOT NULL)").alias("valid_array")
).show()
# Output:
+-------+-----------+
| name|valid_array|
+-------+-----------+
| James| true|
|Michael| true|
| Robert| false|
| John| false|
+-------+-----------+
In this example, we verify that the languages array contains "Java" and that none of its elements are null. If both conditions are met, the result is True; otherwise, it is False.
<a id="h-frequently-asked-questions-on-exists-and-forall-in-pyspark"></a>
Frequently Asked Questions on exists() and forall() in PySpark
exists() → Returns true if at least one element in an array satisfies a condition.
They are available in pyspark.sql.functions as higher-order functions.
exists() → Checks for any match.
forall() → Ensures every element matches.
No, They work on arrays of any type (numbers, strings, booleans).
exists([]) → false (no element matches).
forall([]) → true (vacuously true, since no element violates the condition).
Conclusion
The exists() and forall() functions in PySpark provide powerful ways to evaluate conditions across array elements.
- Use
exists()when you want to check if at least one element matches a condition. - Use
forall()when you need to ensure that every element in the array meets the requirement.
Together, they offer flexible tools for filtering, validation, and transformation of array-based data in PySpark workflows.
<a id="h-references"></a>