PySpark flatMap()

PySpark flatMap() transformation flattens the RDD/DataFrame (array/map DataFrame columns) after applying the function on every element and returns a new PySpark RDD/DataFrame.

The returned RDD/DataFrame can have the same count or more number of elements. This is one of the major differences between flatMap() and map(), where map() returns one element for each input element while flatMap() can return zero, one, or more elements for each input element.

Syntax

Syntax of flatMap()

flatMap(f, preservesPartitioning\=False)

First, let’s create an RDD from a Python list.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('AIStudyLabs.com').getOrCreate()
data = ["Project Gutenberg’s",
"Alice’s Adventures in Wonderland",
"Project Gutenberg’s",
"Adventures in Wonderland",
"Project Gutenberg’s"]
rdd = spark.sparkContext.parallelize(data)
for element in rdd.collect():
print(element)

This yields the below output.

Output:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Now, let’s see an example of how to apply the flatMap() transformation on an RDD. In the example below, it first splits each record by space in the RDD and then flattens the results. The resulting RDD contains a single word in each record.

Apply flatMap() transformation to split each record by space and flatten the results

rdd2 = rdd.flatMap(lambda x: x.split(" "))

Display each element from the resulting RDD

for element in rdd2.collect():
print(element)

This yields the below output.

Output:

Project
Gutenberg’s
Alice’s
Adventures
in
Wonderland
Project
Gutenberg’s
Adventures
in
Wonderland
Project
Gutenberg’s