RDD Parallelize
When we use parallelize(), textFile(), or wholeTextFiles() methods of sparkContext to initiate RDD, it automatically splits the data into partitions based on resource availability. Spark provides methods to know how many partitions your data has been partitioned into. And a way to specify the partitions while creating an RDD.
getNumPartitions() – Returns a number of partitions our dataset split into. Any transformations applied on RDD execute parallelly. Spark will run one task for each partition of the cluster.
// Create RDD
val rdd = spark.sparkContext.parallelize(Array(1,2,3,4,5,6,7,8,9,10))
// Get number of partitions
println("Initial partition count:"+rdd.getNumPartitions)
// Output: Initial partition count:2
Note: This number might change based on your resource availability.
Set parallelize manually: We can also set a number of partitions manually, all, we need is, to pass a number of partitions as the second parameter to these methods for example here, I wanted to create 4 partitions.
// Create RDD with specified partitions
sparkContext.parallelize(Array(1,2,3,4,5,6,7,8,9,10), 4).
// Get number of partitions
print("Initial partition count:"+rdd.getNumPartitions)
// Output: Initial partition count:4
Spark also provides two transformations repartition() and coalesce() to increase or reduce the partitions. I have covered these in detail in the next lessons.
Further Reading
For additional learning on this topic, I would recommend reading the following from AIStudyLabs.com.