Collect Data from RDD

PySpark collect() is an action operation that is used to retrieve all the elements of the RDD (from all nodes) to the driver program for further processing or analysis. We should use the collect() on smaller datasets usually after filtering, grouping, etc. Retrieving larger datasets results in out-of-memory errors in the driver program.

Here's how the collect() action works:

1. When you invoke collect() on an RDD, PySpark gathers all the data from the distributed partitions of the RDD and collects it into an array or collection in the driver program's memory. This means that all the data in the RDD will be moved to the driver, so it should be used with caution, especially when dealing with large datasets.
2. The data is collected in the form of an array or collection in the driver program's memory, making it available for further processing using standard programming constructs.
3. It's important to note that using collect() can be memory-intensive, and if the RDD contains a large amount of data, it may lead to memory issues or even cause the driver program to run out of memory.

Example:

Below is an example of how to use collect() on RDD in PySpark.

# Import Session
from pyspark.sql import SparkSession

# Create SparkSession
spark = SparkSession.builder \
    .master("local[1]") \
    .appName("AIStudyLabs.com") \
    .getOrCreate()

# Create RDD
simpleData = [("James", "Sales", "NY"),
              ("Michael", "Sales", "NY"),
              ("Robert", "Sales", "CA"),
              ("Maria", "Finance", "CA")]
rdd = spark.sparkContext.parallelize(simpleData)

# Collect data to the driver
colData = rdd.collect()

# Print the collected data to console
for data in colData:
    col1 = data[0]
    col2 = data[1]
    col3 = data[2]
    print(f"fname:{col1}, lname:{col2}, state:{col3}")

Output:

fname:James, lname:Sales, state:NY
fname:Michael, lname:Sales, state:NY
fname:Robert, lname:Sales, state:CA
fname:Maria, lname:Finance, state:CA