Collect Data from RDD

Spark collect() is an action operation that is used to retrieve all the elements of the RDD (from all nodes) to the driver program for further processing or analysis. We should use the collect() on smaller datasets usually after filtering, grouping, etc. Retrieving larger datasets results in out-of-memory errors in the driver program.

Here's how the collect() action works:

1. When you invoke collect() on an RDD, Spark gathers all the data from the distributed partitions of the RDD and collects it into an array or collection in the driver program's memory. This means that all the data in the RDD will be moved to the driver, so it should be used with caution, especially when dealing with large datasets.
2. The data is collected in the form of an array or collection in the driver program's memory, making it available for further processing using standard programming constructs.
3. It's important to note that using collect() can be memory-intensive, and if the RDD contains a large amount of data, it may lead to memory issues or even cause the driver program to run out of memory.

Example:

Below is an example of how to use collect() on RDD.

// Import Session
import org.apache.spark.sql.SparkSession

// Create SparkSession
val spark:SparkSession = SparkSession.builder()
  .master("local[1]")
  .appName("AIStudyLabs.com")
  .getOrCreate()

// Create RDD
val simpleData = Seq(("James","Sales","NY"),
  ("Michael","Sales","NY"),
  ("Robert","Sales","CA"),
  ("Maria","Finance","CA")
)
val rdd = spark.sparkContext.parallelize(simpleData)

// Collect data to the driver
val colData = rdd.collect()

// Print the collected data to console
colData.foreach(data =>{
  val col1 = data._1
  val col2 = data._2
  val col3 = data._3
  println("fname:"+col1+", lname:"+col2+", state:"+col3)
})

Output:

fname:James, lname:Sales, state:NY
fname:Michael, lname:Sales, state:NY
fname:Robert, lname:Sales, state:CA
fname:Maria, lname:Finance, state:CA