Create RDD

The SparkContext class in Java provides several methods to create an RDD. Let's explore each with corresponding examples.

• parallelize()
• emptyRDD()
• textFile()
• wholeTextFiles()

Create RDD from Spark Shell or REPL

In previous lessons, you learned that Spark shell provides a sparkContext variable "sc". In Java Spark environments, you can achieve the same by using the SparkConf and JavaSparkContext classes. Here’s how you can create an RDD with an Array of Integers using Java:

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.JavaRDD;
import java.util.Arrays;

public class CreateRDDExample {
    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("JavaSparkExample").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);
        
        JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
    }
}

Creating RDD from the Application

In Java, you can create an RDD from various data sources, such as existing collections, HDFS (Hadoop Distributed File System), local files, S3, and more. RDDs are primarily created in two different ways: by parallelizing an existing collection and by referencing a dataset from an external storage system like a local file system, HDFS, S3, and many more.

To create an RDD, initialize the Java SparkContext using SparkConf as shown below:

// Create JavaSparkContext
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;

SparkConf conf = new SparkConf().setAppName("AIStudyLabs").setMaster("local[1]");
JavaSparkContext sc = new JavaSparkContext(conf);

Using parallelize()

sc.parallelize() is used to parallelize an existing collection in your driver program. This method is primarily used during Proof of Concepts (POCs) or prototyping as it requires all data to be present on the driver program beforehand; therefore, it is not commonly used for production applications. Nonetheless, it is widely used for testing purposes.

// Create RDD from parallelize
JavaRDD<String> rddParallelize = sc.parallelize(Arrays.asList("Java", "Python", "Scala"));

Create Empty RDD

There might be situations when you need to create an empty RDD. In Java Spark, you can use the emptyRDD() method.

// Creates empty RDD with no partition
JavaRDD<String> emptyRDD = sc.emptyRDD();

// Print
System.out.println("Is Empty RDD : " + emptyRDD.isEmpty());

// Output: Is Empty RDD : true