Cache and Persistence

Spark Cache and persist are optimization techniques for iterative and interactive Spark applications to improve the performance of subsequent operations. It is a mechanism to store RDD or DataFrame in memory so they can be reused effectively across multiple operations.

As you know, Spark provides computation 100x faster than traditional MapReduce jobs; if you have not designed the jobs to reuse the repeating computations, you will see a drop in performance when you are dealing with billions or trillions of data. Hence, we need to look at the stages and use optimization techniques as one of the ways to improve performance.

Caching can significantly improve the performance of Spark applications by reducing the need to recompute or reload data from disk. Caching helps avoid recomputation by storing intermediate or frequently used data in memory. cache() and persist() are two methods used for caching the RDD or DataFrame. The difference between cache() and persist() is that cache() saves the data to memory, whereas persist() saves it to the specified type of storage level.

Spark Cache vs Persist

Using cache() and persist() methods, Spark provides an optimization mechanism to store the intermediate computation of an RDD, DataFrame, and Dataset so they can be reused in subsequent actions (reusing the RDD, DataFrame, and Dataset computation results).

Both caching and persisting are used to save the Spark RDD, DataFrame, and Datasets. But, the difference is, the RDD cache() method default saves it to memory (MEMORY_ONLY), whereas the persist() method is used to store to the user-defined storage level.

cache() Method:

The cache() method is a convenient way to persist an RDD in memory by default. When you call cache() on an RDD, it is stored in memory, and Spark tries to keep it there as long as there's enough memory available.
You can use cache() without specifying storage levels, but it defaults to storing the RDD in memory only (MEMORY_ONLY level). This is suitable when you have enough memory to accommodate the RDD.

persist() Method:

The persist() method allows you to specify different storage levels and options for persisting the RDD. This gives you more control over where and how the data is stored.
persist() takes an optional argument that specifies the storage level, which can include options like memory-only, memory-and-disk, disk-only, and more. You can also specify replication levels.

Advantages of Caching and Persistence

Below are the advantages of using Spark Cache and Persist methods.

Cost efficient – Spark computations are very expensive hence reusing the computations is used to save cost.
Time efficient – Reusing the repeated computations saves lots of time.
Execution time – Saves execution time of the job and we can perform more jobs on the same cluster.

Spark Cache Syntax and Example

Following is the syntax and usage of the cache() method in Java.

Syntax

// Syntax
Dataset.cache()

Example

// Create SparkSession object
import org.apache.spark.sql.SparkSession;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.JavaRDD;

SparkSession spark = SparkSession.builder().master("local[1]")
        .appName("AIStudyLabs.com").getOrCreate();
JavaSparkContext sc = new JavaSparkContext(spark.sparkContext());

JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
rdd.cache();  // The RDD is cached in memory by default

Spark Persist Syntax and Example

Spark persist has two signatures: the first signature doesn’t take any argument, which by default saves it to MEMORY_AND_DISK storage level, and the second signature takes StorageLevel as an argument to store it at different storage levels.

Syntax

// Spark provides two overloaded methods
RDD.persist()
RDD.persist(StorageLevel newLevel)

Example

// Example
JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
rdd.persist(StorageLevel.MEMORY_AND_DISK());  // Specify storage level

Unpersist Syntax and Example

We can also unpersist or remove the persistent objects from the memory or storage by using the unpersist() method.

Syntax

// Spark provides two overloaded methods
RDD.unpersist()
RDD.unpersist(boolean blocking)

Example

// Example
JavaRDD<Integer> rdd2 = rdd.unpersist();

In summary, cache() is a convenient way to persist an RDD in memory with default settings, whereas persist() provides more flexibility by allowing you to specify custom storage levels and options. Depending on your specific use case and memory constraints, you can choose between these methods to optimize the performance and resource usage of your Spark application.

Cache and Persistence