Introduction

RDD is a fundamental data structure of Spark and it is the primary data abstraction in Apache Spark and the Spark Core. RDDs are fault-tolerant, immutable distributed collections of objects, which means once you create an RDD you cannot change it. Each dataset in an RDD is divided into logical partitions, which can be computed on different nodes of the cluster. 

In other words, RDDs in Java Spark are a collection of objects distributed across various nodes in a cluster, similar to collections in Java, with the key difference that RDD computations occur on multiple Java Virtual Machines (JVMs) across different physical servers (nodes), unlike a standard Java collection which resides on a single JVM.  

Additionally, RDDs provide data abstraction of partitioning and distribution of the data designed to run computations in parallel on several nodes. When performing transformations on RDDs, we often don't need to concern ourselves with parallelism, as Spark provides this automatically.

<img height="339" width="547" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/rdd.png" />

Characteristics of RDDs

• Immutable: Once created, an RDD cannot be modified. This ensures consistency and helps in easy recovery from failures.
• Distributed: RDDs divide tasks across nodes in a cluster, facilitating distributed processing.
• Fault-Tolerant: Built-in fault tolerance through lineage, allowing automatic recovery from failures.
• Lazy Evaluated: Operations are not executed immediately but are computed when an action is required.

Fault Tolerance in RDDs

Spark ensures fault tolerance through lineage tracking. If a partition is lost, RDD retains enough information (i.e., the lineage) to recompute the lost data from the original source.

Lineage Graphs

A lineage graph is a record of all transformations applied to RDDs. It allows Spark to reconstruct lost data easily.

How Lineage Ensures Resilience

When a node failure occurs, Spark leverages the lineage graph to rerun the missing or affected partitions, ensuring the job completes successfully without user intervention.

Summary / Key Takeaways

In this lesson on Understanding RDDs with Java Spark, you have learned:

• RDDs are immutable, distributed collections supporting parallel processing in a Java Spark environment.
• You can create RDDs by parallelizing collections or loading from external sources using Java Spark APIs.
• Transformations are lazy operations creating new RDDs; actions trigger execution and produce non-RDD results in Java.
• RDDs achieve fault tolerance through lineage graphs, allowing automatic recovery of lost partitions.
• Lineage graphs provide a history of operations, aiding in error rectification and data recovery.
• Practical use cases of RDDs in Java Spark demonstrate their power in processing large datasets efficiently.