Introduction

RDD is a fundamental data structure of PySpark and it is the primary data abstraction in Apache PySpark and the PySpark Core. RDDs are fault-tolerant, immutable distributed collections of objects, which means once you create an RDD you cannot change it. Each dataset in RDD is divided into logical partitions, which can be computed on different nodes of the cluster.

In other words, RDDs are a collection of objects similar to lists in Python, with the difference being RDD is computed on several processes scattered across multiple physical servers also called nodes in a cluster while a Python collection lives and processes in just one system.

Additionally, RDDs provide data abstraction of partitioning and distribution of the data which is designed to run computations in parallel on several nodes, while doing transformations on RDD most of the time we don’t have to worry about parallelism as PySpark by default provides it.

<img height="339" width="547" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/rdd.png" />

Characteristics of RDDs

• Immutable: Once created, an RDD cannot be modified. This ensures consistency and helps in easy recovery from failures.
• Distributed: RDDs divide tasks across nodes in a cluster, facilitating distributed processing.
• Fault-Tolerant: Built-in fault tolerance through lineage, allowing automatic recovery from failures.
• Lazy Evaluated: Operations are not executed immediately but are computed when an action is required.

Fault Tolerance in RDDs

PySpark ensures fault tolerance through lineage tracking. If a partition is lost, RDD retains enough information (i.e., the lineage) to recompute the lost data from the original source.

Lineage Graphs

A lineage graph is a record of all transformations applied to RDDs. It allows PySpark to reconstruct lost data easily.

How Lineage Ensures Resilience

When a node failure occurs, PySpark leverages the lineage graph to rerun the missing or affected partitions, ensuring the job completes successfully without user intervention.

Summary / Key Takeaways

In this lesson on Understanding RDDs, you have learned:

• RDDs are immutable, distributed collections supporting parallel processing.
• You can create RDDs by parallelizing collections or loading from external sources.
• Transformations are lazy operations creating new RDDs; actions trigger execution and produce non-RDD results.
• RDDs achieve fault tolerance through lineage graphs, allowing automatic recovery of lost partitions.
• Lineage graphs provide a history of operations, aiding in error rectification and data recovery.
• Practical use cases of RDDs demonstrate their power in processing large datasets efficiently.