Install Apache Spark on Mac

Five easy steps to install the latest version of Apache Spark on Mac (macOS) – In recent days Apache Spark installation on Mac OS has become very easy using Homebrew. You can install it and start running examples in just 5 mins. There are multiple ways to install Apache Spark on Mac.

Note: Please use the latest version. The Java, Scala and Spark versions I used in this lesson might be older depending on when you are using it. I would highly recommend using the latest available versions for each tool.

Below I have explained the step-by-step installation of Apache Spark on Mac OS using Homebrew, validating the install, running spark-shell, and creating a Spark DataFrame.

Steps to Install the Latest Version of Apache Spark on Mac OS

• Step 1 – Install Homebrew
• Step 2 – Install Java
• Step 3 – Install Scala
• Step 4 – Install Apache Spark Latest Version
• Step 5 – Start Spark shell and Validate Installation

1. Install Apache Spark Latest Version on Mac

Homebrew is a Missing Package Manager for macOS that is used to install third-party packages like Java, and Apache Spark on Mac (macOS). In order to use Homebrew, first, you need to install it by using the below command.

// Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This prompts for the root password. You will need to type your root password to run this command. On a personal laptop, this is the same password you enter when you log into your Mac. If you don’t have root access, contact your system admin. You should see something like this below after the successful installation of homebrew.

<img height="260" width="634" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/homebrew.png" />

Post-installation, you may need to run the below command to set the brew to your $PATH.

// Set brew to Path
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> /Users/admin/.zprofile
eval "$(/opt/homebrew/bin/brew shellenv)"

If the above command has issues, you can find the latest command from Homebrew.

2. Install Java Version

Spark uses Java underlying hence you need to have Java on your Mac. Since Java is a third-party, you can install it using the Homebrew command brew. Since Oracle Java is not open source anymore, I am using OpenJDK version 21. Run the below command in the terminal to install it.

// Install OpenJDK 21
brew install openjdk@21

3. Install Scala

Since Apache Spark is written in Scala language it is obvious you would need Scala to run Spark programs.

// Install Scala 
brew install scala

4. Install Apache Spark on Mac

You can also download and install Apache Spark's latest version on Mac using Homebrew. 

// Install Apache Spark
brew install apache-spark

This installs the latest version of Apache Spark on your Mac OS.

Alternatively, you can also install Apache Spark latest version from https://spark.apache.org/downloads.html. If you use this option, you need to set the spark to your environment variables.

<img height="92" width="487" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/spark-download.png" />

After successful installation of Apache Spark latest version, run spark-shell from the Terminal to launch Spark shell. You should see something like this below (ignore the warning for now). spark-shell is a CLI utility that comes with Apache Spark distribution.

<img height="364" width="641" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/image.png" />

Note that it displays the Spark version and Java version you are using on the terminal. Also, notice this line 

"Spark context Web UI available at http://localhost:4041"

Note: Spark by default creates Spark context web UI with port 4040, if this port is already taken then it uses 4041. In my case, it is listening on 4041.

<img height="253" width="714" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/181/spark-web-ui-v4.png" />

5. Validate Spark Installation from Shell

Let’s create a Spark DataFrame with some sample data to validate the installation. Enter the following commands in the Spark Shell in the same order.

scala> val rdd = sc.parallelize(Array(1,2,3,4,5,6,7,8,9,10))

//Output
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at :24

Now access http://localhost:4041/jobs/ from your favorite web browser to access Spark Web UI to monitor your jobs. If this is your first time running Spark, you might have web UI with port 4040.

Summary and Key Takeaways

• Homebrew simplifies installing third-party tools on macOS, crucial for setting up Spark.
• Java and Scala are necessary dependencies for running Apache Spark.
• A successful Spark installation can be validated via the spark-shell and Spark’s Web UI.