RDD Transformations

Spark RDD Transformations are lazily evaluated operations used to transform/update RDDs. When executed on an RDD, they result in one or multiple new RDDs.

Since RDDs are immutable, transformations always create a new RDD without updating an existing one. Hence, a chain of RDD transformations creates an RDD lineage.

RDD Transformations are Lazy

RDD Transformations are lazy operations, meaning none of the transformations get executed until you call an action on the Spark RDD. Since RDDs are immutable, any transformations on them result in a new RDD leaving the current one unchanged.

RDD Transformation Types

There are two types of transformations.

Narrow Transformation

Narrow transformations result from methods like map() and filter(), and these compute data that resides on a single partition, meaning there will not be any data movement between partitions to execute narrow transformations.

Functions such as map(), mapPartitions(), flatMap(), filter(), and union() are examples of narrow transformations.

Wide Transformation

Wide transformations result from methods like groupByKey() and reduceByKey(), computing data across multiple partitions, meaning there will be data movement between partitions during execution. Since these cause shuffles, they are also called shuffle transformations.

Functions such as groupByKey(), aggregateByKey(), aggregate(), join(), and repartition() are examples of wide transformations.

Note: Compared to narrow transformations, wide transformations are more expensive operations due to shuffling.

Spark RDD Transformation Methods

Here are some RDD transformation methods.

| TRANSFORMATION METHODS | USAGE AND DESCRIPTION | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | cache() | Caches the RDD | | filter() | Returns a new RDD after applying the filter function on the source dataset. | | flatMap() | Returns a flattened map, meaning if you have a dataset with an array, it converts each element in an array as a row. In other words, it returns 0 or more items in output for each element in the dataset. | | map() | Applies a transformation function on the dataset and returns the same number of elements in the distributed dataset. | | mapPartitions() | Similar to map, but executes a transformation function on each partition, which provides better performance than the map function. | | mapPartitionsWithIndex() | Similar to mapPartitions, but also provides a function with an integer value representing the index of the partition. | | randomSplit() | Splits the RDD by the weights specified in the argument. For example, rdd.randomSplit(new double[]{0.7,0.3}) | | union() | Combines elements from the source dataset and the argument and returns the combined dataset. This is similar to the union function in Math set operations. | | sample() | Returns the sample dataset. | | intersection() | Returns the dataset, which contains elements present in both the source dataset and an argument. | | distinct() | Returns the dataset by eliminating all duplicate elements. | | repartition() | Returns a dataset with a number of partitions specified in the argument. This operation reshuffles the RDD randomly. It could either return a lesser or more partitioned RDD based on the input supplied. | | coalesce() | Similar to repartition, but operates better when we want to decrease the number of partitions. Betterment is achieved by reshuffling the data from fewer nodes compared with all nodes by repartition. |

Spark RDD Transformations with Examples

In this section, I will explain a few RDD Transformations with examples in Java Spark.

First, create a text file containing the text you want to analyze. For this example, let's assume you have a text file named test.txt with the following content:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

// Create SparkSession
SparkSession spark = SparkSession.builder().master("local[3]")
      .appName("AIStudyLabs.com").getOrCreate();

// Create JavaRDD
JavaSparkContext sc = new JavaSparkContext(spark.sparkContext());
JavaRDD<String> rdd = sc.textFile("/path/test.txt");

<img height="324" width="527" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/rdd-transformation-work-count.png" />

flatMap() Transformation

flatMap() transformation flattens the RDD after applying the function and returns a new RDD. In the example below, it first splits each record by space in an RDD and then flattens it. The resulting RDD consists of a single word in each record.

// Apply flatMap()
JavaRDD<String> rdd2 = rdd.flatMap(line -> Arrays.asList(line.split(" ")).iterator());

map() Transformation

map() transformation is used to apply any complex operation like adding a column, or updating a column, etc. The output of map transformations always has the same number of records as the input.

In our word count example, we add a new column with a value 1 for each word. The result of the RDD is a PairRDDFunctions which contains key-value pairs, where the word of type String is the Key, and 1 of type Integer is the value. For better understanding, I’ve defined the rdd3 variable with type.

// Apply map()
JavaPairRDD<String, Integer> rdd3 = rdd2.mapToPair(word -> new Tuple2<>(word, 1));

filter() Transformation

filter() transformation is used to filter records in an RDD. In our example, we filter all words starting with "a".

// Apply filter()
JavaPairRDD<String, Integer> rdd4 = rdd3.filter(tuple -> tuple._1().startsWith("a"));

reduceByKey() Transformation

reduceByKey() merges the values for each key with the specified function. In our example, it reduces the word strings by applying the sum function on each value. The result of our RDD contains unique words and their count.

// Apply reduceByKey()
JavaPairRDD<String, Integer> rdd5 = rdd3.reduceByKey(Integer::sum);

sortByKey() Transformation

sortByKey() transformation sorts RDD elements by key. In our example, we first convert RDD[(String, Integer)] to RDD[(Integer, String)] using map transformation, then apply sortByKey which ideally sorts on the integer value. Finally, a foreach loop with a println statement prints all words in the RDD and their count as a key-value pair to the console.

// Apply sortByKey()
JavaPairRDD<Integer, String> rdd6 = rdd5.mapToPair(tuple -> new Tuple2<>(tuple._2(), tuple._1())).sortByKey();

// Print rdd6 result to console
rdd6.foreach(tuple -> System.out.println(tuple));