Spark Persistence Storage Levels

All different storage levels of Spark supports are available in org.apache.spark.storage.StorageLevel class. The storage level specifies how and where to persist or cache a Spark DataFrame and Dataset.

MEMORY_ONLY – This is the default behavior of the RDD cache() method and stores the RDD or DataFrame as deserialized objects in JVM memory. When there is not enough memory available, it will not save RDD of some partitions and these will be re-computed as and when required. This takes more memory. 

MEMORY_ONLY_SER – This is the same as MEMORY_ONLY but the difference is that it stores RDD as serialized objects in JVM memory. It takes less memory (space-efficient) than MEMORY_ONLY as it saves objects as serialized and takes an additional few more CPU cycles in order to deserialize.

MEMORY_ONLY_2 – Same as MEMORY_ONLY storage level but replicate each partition to two cluster nodes.

MEMORY_ONLY_SER_2 – Same as MEMORY_ONLY_SER storage level but replicate each partition to two cluster nodes.

MEMORY_AND_DISK – This is the default behavior of the DataFrame or Dataset. In this Storage Level, The DataFrame will be stored in JVM memory as a deserialized object. When required storage is greater than available memory, it stores some of the excess partitions into the disk and reads the data from the disk when required. It is slower as there is I/O involved.

MEMORY_AND_DISK_SER – This is the same as MEMORY_AND_DISK storage level difference being it serializes the DataFrame objects in memory and on disk when space is not available.

MEMORY_AND_DISK_2 – Same as MEMORY_AND_DISK storage level but replicate each partition to two cluster nodes.

MEMORY_AND_DISK_SER_2 – Same as MEMORY_AND_DISK_SER storage level but replicate each partition to two cluster nodes.

DISK_ONLY – In this storage level, DataFrame is stored only on disk and the CPU computation time is high as I/O is involved.

DISK_ONLY_2 – Same as DISK_ONLY storage level but replicate each partition to two cluster nodes.

Below is the table representation of the Storage level, Go through the impact of space, CPU, and performance. Choose the one that best fits your needs.

Storage Level    Space used  CPU time  In memory  On-disk  Serialized   Recompute some partitions
----------------------------------------------------------------------------------------------------
MEMORY_ONLY          High        Low       Y          N        N         Y    
MEMORY_ONLY_SER      Low         High      Y          N        Y         Y
MEMORY_AND_DISK      High        Medium    Some       Some     Some      N
MEMORY_AND_DISK_SER  Low         High      Some       Some     Y         N
DISK_ONLY            Low         High      N          Y        Y         N

Some Points to note on Persistence

• Spark automatically monitors every persist() and cache() call you make and it checks usage on each node and drops persisted data if not used or using the least-recently-used (LRU) algorithm. As discussed in one of the above sections you can also manually remove using unpersist() method.
• Spark caching and persistence are just one of the optimization techniques to improve the performance of Spark jobs.
• For RDD cache() default storage level is ‘MEMORY_ONLY‘ but, for DataFrame and Dataset, the default is ‘MEMORY_AND_DISK‘
• On Spark UI, the Storage tab shows where partitions exist in memory or disk across the cluster.
• Dataset cache() is an alias for persist(StorageLevel.MEMORY_AND_DISK)
• Caching of Spark DataFrame or Dataset is a lazy operation, meaning a DataFrame will not be cached until you trigger an action.