Spark Persistence Storage Levels

In Java Spark, all different storage levels are available in the org.apache.spark.storage.StorageLevel class. The storage level specifies how and where to persist or cache a Spark DataFrame or Dataset.

MEMORY_ONLY – This is the default behavior of the RDD cache() method and stores the RDD or DataFrame as deserialized objects in JVM memory. When there is insufficient memory available, it will not save some partitions, and these will be re-computed as required. This level takes more memory.

MEMORY_ONLY_SER – This is similar to MEMORY_ONLY, but stores RDDs as serialized objects in JVM memory. It uses less memory (more space-efficient) than MEMORY_ONLY because it saves objects in a serialized form, requiring additional CPU cycles to deserialize.

MEMORY_ONLY_2 – Same as MEMORY_ONLY but replicates each partition to two cluster nodes.

MEMORY_ONLY_SER_2 – Same as MEMORY_ONLY_SER but replicates each partition to two cluster nodes.

MEMORY_AND_DISK – This is the default behavior for DataFrames or Datasets. In this storage level, the DataFrame stores as a deserialized object in JVM memory. When the storage requirement exceeds available memory, excess partitions are stored on disk, with data read from disk when needed. This level is slower due to I/O involvement.

MEMORY_AND_DISK_SER – Similar to MEMORY_AND_DISK, but serializes DataFrame objects in memory and on disk when space is not available.

MEMORY_AND_DISK_2 – Same as MEMORY_AND_DISK but replicates each partition to two cluster nodes.

MEMORY_AND_DISK_SER_2 – Same as MEMORY_AND_DISK_SER but replicates each partition to two cluster nodes.

DISK_ONLY – At this storage level, DataFrames are stored only on disk, leading to high CPU computation time due to I/O involvement.

DISK_ONLY_2 – Same as DISK_ONLY but replicates each partition to two cluster nodes.

Below is the table representation of the storage level, with their impact on space, CPU, and performance. Choose the one that best fits your needs.

Storage Level       Space used   CPU time  In memory  On-disk  Serialized   Recompute some partitions
--------------------------------------------------------------------------------------------------------
MEMORY_ONLY          High        Low       Y          N        N           Y    
MEMORY_ONLY_SER      Low         High      Y          N        Y           Y
MEMORY_AND_DISK      High        Medium    Some       Some     Some        N
MEMORY_AND_DISK_SER  Low         High      Some       Some     Y           N
DISK_ONLY            Low         High      N          Y        Y           N

Some Points to Note on Persistence

• In Java Spark, Spark automatically tracks each persist() and cache() call, monitors usage on each node, and drops persisted data that is unused based on the least-recently-used (LRU) algorithm. You can also manually remove data using the unpersist() method.
• Spark caching and persistence are crucial optimization techniques to enhance the performance of Spark jobs.
• For RDD cache(), the default storage level is MEMORY_ONLY, whereas for DataFrames and Datasets, the default is MEMORY_AND_DISK.
• The Spark UI's Storage tab displays where partitions exist in memory or disk across the cluster.
• The Dataset cache() method is an alias for persist(StorageLevel.MEMORY_AND_DISK).
• Caching a Spark DataFrame or Dataset is a lazy operation; a DataFrame will not be cached until an action triggers it.