RDD Parallelize
When we use the parallelize(), textFile(), or wholeTextFiles() methods of PySpark's sparkContext to initiate RDD, it automatically splits the data into partitions based on resource availability. PySpark provides methods to know how many partitions your data has been partitioned into, and a way to specify the partitions while creating an RDD.
getNumPartitions() – Returns the number of partitions our dataset is split into. Any transformations applied on RDD execute in parallel. PySpark will run one task for each partition of the cluster.
# Create RDD
rdd = spark.sparkContext.parallelize([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
# Get number of partitions
print("Initial partition count:" + str(rdd.getNumPartitions()))
# Output: Initial partition count:2
Note: This number might change based on your resource availability.
Set parallelize manually: We can also set the number of partitions manually. All we need is to pass the number of partitions as the second parameter to these methods. For example, here, I wanted to create 4 partitions.
# Create RDD with specified partitions
rdd = spark.sparkContext.parallelize([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 4)
# Get number of partitions
print("Initial partition count:" + str(rdd.getNumPartitions()))
# Output: Initial partition count:4
PySpark also provides two transformations, repartition() and coalesce(), to increase or reduce the partitions. I have covered these in detail in the next lessons.
Further Reading
For additional learning on this topic, I would recommend reading the following from AIStudyLabs.com.