Create RDD
SparkContext class provides the below methods to create an RDD. Let's see each with an example.
- parallelize()
- emptyRDD()
- textFile()
- wholeTextFiles()
Create RDD from Spark Shell or REPL
You have learned in the previous lesson that Spark shell provides sparkContext variable “sc”, hence, you can use sc.parallelize() to create an RDD. The below example creates an RDD with an Array of Integers.
scala> val rdd = sc.parallelize(Array(1,2,3,4,5,6,7,8,9,10))
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at :24
Creating RDD from the Application
You can create RDD in Spark from various data sources, such as an existing collection, HDFS (Hadoop Distributed File System), local files, S3, and more. RDDs are created primarily in two different ways, first parallelizing the existing collection and secondly referencing a Dataset from an external storage system like a local file system, HDFS, S3 and many more.
Before we create an RDD, we need to initialize the SparkSession object using the builder pattern by passing the master, appName details as shown below.
// Create SparkSession
import org.apache.spark.sql.SparkSession
val spark:SparkSession = SparkSession.builder().master("local[1]")
.appName("AIStudyLabs.com").getOrCreate()
Using parallelize()
sparkContext.parallelize() is used to parallelize an existing collection in your driver program. This is a basic method to create RDD and is used mainly during POCs or prototyping as it requires all data to be present on the driver program prior hence it is not most used for production applications. But it is widely used for testing purposes.
// Create RDD from parallelize
val dataSeq = Seq(("Java", 20000), ("Python", 100000), ("Scala", 3000))
val rddParallelize = spark.sparkContext.parallelize(dataSeq)
Create Empty RDD
Sometimes we may need to create an empty RDD and you can use either emptyRDD() method or parallelize()
// Creates empty RDD with no partition
val emptyRDD = spark.sparkContext.emptyRDD // creates EmptyRDD[0]
val emptyRDD2 = spark.sparkContext.emptyRDD[String] // creates EmptyRDD[1]
// Print
print("Is Empty RDD : "+emptyRDD.isEmpty())
// Output: Is Empty RDD : true
Further Reading
For additional learning on this topic, I would recommend reading the following from AIStudyLabs.com.