Introduction

Apache Spark is a computing framework that processes data in parallel in a distributed environment with low latency. Spark works on a master node (driver) and slave (worker) node architecture within a cluster. Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program). Below are the components of Spark.

<img height="208" width="435" alt="Spark Cluster Overview" title="Spark Cluster Overview" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/spark-cluster-overview.png" />

Cluster: A cluster is a platform to launch the Spark applications. A Spark cluster is a combination of a Driver Program, Cluster Manager, and Worker Nodes that work together to complete tasks. The SparkContext (Driver program) lets us coordinate processes across the cluster in order to execute the application. The SparkContext sends tasks to the Executors on the Worker Nodes to run. The cluster can be a single node or multinode cluster based on the resource availability.

Driver Program: The Spark driver program is the one that creates the SparkContext object in the application. As soon as we submit the Spark job, the driver program runs the main() method of your application and creates DAGs representing the data flow internally. Based on the DAG workflow, the driver requests the cluster manager to allocate the resources (workers) required for processing. Once the resources are allocated, the driver then uses the Spark context to send the serialized result (code+data) to the workers to execute as Tasks and their result is captured.

Cluster Manager: Cluster Manager is the key component in the Spark architecture. It plays a very important role in terms of resource management. Cluster Manager makes sure the application runs efficiently and effectively by utilizing the resources of the cluster. There are a variety of cluster managers such as Hadoop YARN, Apache Mesos, and Standalone Scheduler. Below are the key responsibilities of the Cluster Manager.

• Resource allocation: Cluster Manager plays a vital role in allocating the CPU cores, memory, and other resources to the application in a distributed environment.
• Task Scheduling: Spark application is a set of tasks that have to be executed. Cluster Manager schedules the tasks to the resources available in the cluster based on the priorities.
• Fault tolerance: Cluster Manager also handles the failures. In case a worker node fails or a task fails, the Cluster Manager reschedules the task to any available nodes in the cluster, so that the task can be completed.
• Scalability: Cluster Manager is designed to handle various sizes of clusters. They can scale resource allocation and scheduling based on the availability of nodes on the cluster.

Cluster Manager Types

Spark supports below cluster managers:

• Standalone – A simple cluster manager included with Spark that makes it easy to set up a cluster.
• Apache Mesos – Mesos is a Cluster manager that can also run Hadoop MapReduce and PySpark applications.
• Hadoop YARN – The resource manager in Hadoop 2. This is the mostly used cluster manager.
• Kubernetes – An open-source system for automating deployment, scaling, and management of containerized applications.
• local – This is not really a cluster manager but still, I wanted to mention that as we use “local” in order to run Spark on your laptop.