Built with Tutify.io

    AIStudyLabs

    Explain PySpark explode() Function

    Explain PySpark explode() Function

    In PySpark, the explode() function is used to explode an array or a map column into multiple rows, meaning one row per element. It is part of the pyspark.sql.functions module and is commonly used when dealing with nested structures like arrays, JSON, or structs.

    In this article, we’ll explore how explode() works, understand its behavior with nulls, and cover use cases such as exploding multiple columns, arrays, structs, and JSON. We’ll also compare explode() with explode_outer().

    Key Points-

    <a id="h-pyspark-explode-function"></a>

    PySpark explode() Function

    The PySpark explode() function creates a new row for each element in an array or map column. It by default assigns the column name col for arrays and key and value for maps unless you specify custom names.

    <a id="h-syntax"></a>

    Syntax

    Following is the syntax of the explode() function.

    
# Syntax of the explode()
from pyspark.sql.functions import explode
explode(col)

    <a id="h-parameters"></a>

    Parameters

    <a id="h-return-value"></a>

    Return Value

    It returns a new column in which each row represents an element of the array or a key-value pair from the map.

    Let’s start with a sample DataFrame containing arrays and maps.

    
# Create SparkSession and Prepare sample Data
from pyspark.sql import SparkSession
from pyspark.sql.functions import explode, col

spark = SparkSession.builder.appName('pyspark-by-examples').getOrCreate()

arrayData = [
    ('James', ['Java', 'Scala'], {'hair': 'black', 'eye': 'brown'}),
    ('Michael', ['Spark', 'Java', None], {'hair': 'brown', 'eye': None}),
    ('Robert', ['CSharp', ''], {'hair': 'red', 'eye': ''}),
    ('Washington', None, None),
    ('Jefferson', ['1', '2'], {})
]

df = spark.createDataFrame(data=arrayData, schema=['name', 'knownLanguages', 'properties'])
df.printSchema()
df.show(truncate=False)

    Yields below the output.

    <a id="h-pyspark-explode-on-array-column"></a>

    PySpark explode() on Array Column

    You can use the explode() function on an array-type column in a DataFrame to transform each element of the array into a separate row in a new column. By default, this function provides a new column name is col.

    
# Explode array column
df_exploded = df.select(df.name, explode(df.knownLanguages).alias("language"))
df_exploded.show(truncate=False)

    Yields below the output.

    <a id="h-pyspark-explode-on-map-column"></a>

    PySpark explode() on Map Column

    You can use the explode() function on a map-type column in a DataFrame to convert each key-value pair in the map into separate rows. By default, this function creates two new columns named key and value, unless you specify custom aliases.

    
# Explode map column
df_explode_map = df.select(df.name, explode(df.properties))
df_explode_map.show(truncate=False)

    Yields below the output.

    
# Output:
+-------+---+----+-----+
|name   |pos|key |value|
+-------+---+----+-----+
|James  |0  |eye |brown|
|James  |1  |hair|black|
|Michael|0  |eye |NULL |
|Michael|1  |hair|brown|
|Robert |0  |eye |     |
|Robert |1  |hair|red  |
+-------+---+----+-----+

    <a id="h-pyspark-explode-multiple-columns"></a>

    PySpark explode Multiple Columns

    You can use multiple explode() functions in a single select() statement to flatten multiple arrays or map columns simultaneously. Each explode() expands its respective column, creating separate rows for every element or key-value pair while keeping the other columns repeated accordingly.

    
# PySpark explode() Multiple Columns
df_multi_explode = df.select(
    df.name,
    explode(df.knownLanguages).alias("language"),
    explode(df.properties).alias("prop_key", "prop_value")
)
df_multi_explode.show(truncate=False)

    Yields below the output.

    
# Output:
+-------+--------+--------+----------+
|name   |language|prop_key|prop_value|
+-------+--------+--------+----------+
|James  |Java    |eye     |brown     |
|James  |Java    |hair    |black     |
|James  |Scala   |eye     |brown     |
|James  |Scala   |hair    |black     |
|Michael|Spark   |eye     |NULL      |
|Michael|Spark   |hair    |brown     |
|Michael|Java    |eye     |NULL      |
|Michael|Java    |hair    |brown     |
|Michael|NULL    |eye     |NULL      |
|Michael|NULL    |hair    |brown     |
|Robert |CSharp  |eye     |          |
|Robert |CSharp  |hair    |red       |
|Robert |        |eye     |          |
|Robert |        |hair    |red       |
+-------+--------+--------+----------+

    <a id="h-pyspark-explode-json-column"></a>

    PySpark explode() JSON Column

    You can use the from_json() function to parse a JSON string column into a structured format based on a defined schema. After parsing, the JSON data becomes accessible as a nested column. You can then apply the explode() function on an array field (like lang) within the parsed JSON to create separate rows for each element in that array, making the data easier to analyze and process.

    
# PySpark explode() JSON Column
from pyspark.sql.functions import from_json, schema_of_json

json_schema = schema_of_json('{"lang":["Python","Java"],"level":"Intermediate"}')

data = [("James", '{"lang":["Python","Java"],"level":"Intermediate"}')]
df_json = spark.createDataFrame(data, ["name", "json_data"])

df_parsed = df_json.withColumn("parsed", from_json(col("json_data"), json_schema))
df_exploded_json = df_parsed.select("name", explode(col("parsed.lang")).alias("language"))
df_exploded_json.show(truncate=False)

    Yields below the output.

    
# Output:
+-----+--------+
|name |language|
+-----+--------+
|James|Python  |
|James|Java    |
+-----+--------+

    The code returns a DataFrame where each element of the lang array from the parsed JSON is expanded into a separate row. Since the lang field contains ["Python", "Java"], the explode() function creates two rows for James — one for each language, resulting in the output with Python and Java listed separately.

    <a id="h-explode-an-array-inside-a-struct-column-in-pyspark"></a>

    Explode an Array Inside a Struct Column in PySpark

    In PySpark, you cannot directly explode a StructType column, but you can explode an array field inside the struct.

    This example demonstrates how to expand an array contained within a struct column. Each element of the array is converted into a separate row, while other fields in the struct are retained for each row. The result is a flattened dataset where nested array elements are accessible as individual rows, making the data easier to analyze.

    
# Explode array inside the struct
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, ArrayType, StringType
from pyspark.sql.functions import explode, col

# Create SparkSession
spark = SparkSession.builder.appName("StructExplodeExample").getOrCreate()

# Define schema for struct column
info_schema = StructType([
    StructField("knownLanguages", ArrayType(StringType()), True),
    StructField("level", StringType(), True)
])

# Sample data as tuples (name, knownLanguages, level)
data = [("James", ["Python", "Java"], "Intermediate")]

# Create DataFrame with struct column
df = spark.createDataFrame(data, ["name", "knownLanguages", "level"])

# Convert knownLanguages and level into a struct
from pyspark.sql.functions import struct
df_struct = df.withColumn("info", struct("knownLanguages", "level"))

# Explode the array inside the struct
df_exploded = df_struct.select(
    col("name"),
    explode(col("info.knownLanguages")).alias("language"),
    col("info.level")
)

df_exploded.show(truncate=False)

    Yields below the output.

    
# Output:
+-----+--------+------------+
|name |language|level       |
+-----+--------+------------+
|James|Python  |Intermediate|
|James|Java    |Intermediate|
+-----+--------+------------+

    The code returns a DataFrame where each element of the lang array from the parsed JSON is expanded into a

    explode() vs explode_outer()

    The table below shows how explode() and explode_outer() handle columns that contain null values.

    | Function | Description | | --- | --- | | explode() | Skips rows where array/map is null or empty. | | explode_outer() | Includes rows even when array/map is null or empty (returns null values). |

    Example:

    If the input column contains null values, the explode() function skips that record and returns no rows. To handle nulls gracefully, you can use explode_outer(), which provides that even when the column is null, the function returns a row with a null value in the exploded column instead of dropping the record.

    
# Explode_outer() on an array
from pyspark.sql.functions import explode_outer

df_outer = df.select(df.name, explode_outer(df.knownLanguages).alias("language"))
df_outer.show(truncate=False)

    Yields below the output.

    
# Output:
+----------+--------+
|name      |language|
+----------+--------+
|James     |Java    |
|James     |Scala   |
|Michael   |Spark   |
|Michael   |Java    |
|Michael   |NULL    |
|Robert    |CSharp  |
|Robert    |        |
|Washington|NULL    |
|Jefferson |1       |
|Jefferson |2       |
+----------+--------+

    <a id="h-frequently-asked-questions-of-pyspark-explode"></a>

    Frequently Asked Questions of PySpark explode()

    The explode() function is used to transform each element of an array or each key-value pair of a map column into a separate row, effectively flattening nested data structures in a DataFrame.

    explode() takes a column containing an array or map and returns a new row for every element (array) or key-value pair (map). Other columns are repeated for each new row.

    We use explode() to flatten nested data, making it easier to analyze, filter, aggregate, or join with other DataFrames. It simplifies working with arrays and maps in a structured way.

    If the column is null, explode() returns no rows for that record. To handle null values gracefully, you can use explode_outer(), which returns a row with null for the exploded column instead of dropping the record.

    You can apply multiple explode() functions in the same select() statement to flatten several array or map columns simultaneously, creating separate rows for each element in every column.

    explode() works only on arrays or maps. To work with a struct, you need to access the array field inside the struct and then apply explode() to it.

    For arrays, the default column name is col. For maps, the default column names are key and value. You can override these defaults by using alias().

    <a id="h-conclusion"></a>

    Conclusion

    In this article, you learned how to use the PySpark explode() function to transform arrays and maps into multiple rows. We covered exploding arrays, maps, structs, JSON, and multiple columns, as well as the difference between explode() and explode_outer().

    explode() is essential when dealing with nested or hierarchical data, making it easier to normalize complex data structures for analysis or transformation.

    Happy Learning!!

    <a id="h-reference"></a>

    Reference

    Related Articles