RDD Parallelize

When we use parallelize(), textFile(), or wholeTextFiles() methods of JavaSparkContext to initiate RDD, it automatically splits the data into partitions based on resource availability. Spark provides methods to know how many partitions your data has been partitioned into, and a way to specify the partitions while creating an RDD.

getNumPartitions() – Returns the number of partitions our dataset is split into. Any transformations applied on the RDD execute in parallel. Spark will run one task for each partition of the cluster.

// Import necessary classes
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.SparkConf;

// Create Spark Configuration and Java Spark Context
SparkConf conf = new SparkConf().setAppName("JavaRDD Example").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);

// Create RDD
JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1,2,3,4,5,6,7,8,9,10));

// Get number of partitions
System.out.println("Initial partition count: " + rdd.partitions().size());

// Output: Initial partition count: 2

Note: This number might change based on your resource availability.

Set parallelize manually: We can also set the number of partitions manually. All we need is to pass the number of partitions as the second parameter to these methods. For example, here, I wanted to create 4 partitions.

// Create RDD with specified partitions
JavaRDD<Integer> rddWithPartitions = sc.parallelize(Arrays.asList(1,2,3,4,5,6,7,8,9,10), 4);

// Get number of partitions
System.out.println("Initial partition count: " + rddWithPartitions.partitions().size());

// Output: Initial partition count: 4

Spark also provides two transformations repartition() and coalesce() to increase or reduce the partitions. These will be covered in detail in the next lessons.