Install PySpark on Windows

In this lesson, I will explain step-by-step how to do PySpark installation on Windows OS and explain how to start a history server and monitor your jobs using the Web UI.

Note: Please use the latest version. The Java, Scala, and Spark versions I used in this lesson might be older depending on when you are using it. I would highly recommend using the latest available versions for each tool.

Install Java 21 or Later

To install Apache Spark on Windows, you would need Java 17 or the latest version hence download the Java version from Oracle and install it on your system. If you wanted OpenJDK, you can download it from here. You can also download from this Microsoft link.

After downloading, double-click on the downloaded .exe (microsoft-jdk-21.0.10-windows-x64) file in order to install it on your Windows system. Choose any custom directory or keep the default location.

Note: This article explains installing Apache Spark on Java 21.

Apache Spark Installation on Windows

Apache Spark comes in compressed tar/zip files; hence installation on Windows is not much of a deal as you just need to download and untar the file. Download Apache Spark by accessing the Spark Download page and select the link from “Download Spark."

If you want to use a different version of Spark & Hadoop, select the one you wanted from the drop-down; the link on point 3 changes to the selected version and provides you with an updated link to download.

After download, untar the binary using 7zip or any zip utility to extract the zip file and copy the extracted directory spark-4.1.1-bin-hadoop3-connect to c:\apps\opt\spark-4.1.1-bin-hadoop3-connect.

Spark Environment Variables

Post Java and Apache Spark installation on Windows, set JAVA_HOME, SPARK_HOME, HADOOP_HOME, and PATH environment variables. If you know how to set the environment variable on Windows, add the following.

JAVA_HOME = C:\Program Files\Java\jdk21
PATH = %PATH%;%JAVA_HOME%
SPARK_HOME  = C:\apps\opt\spark-4.1.1-bin-hadoop3-connect
HADOOP_HOME = C:\apps\opt\spark-4.1.1-bin-hadoop3-connect
PATH=%PATH%;%SPARK_HOME%\bin

Follow the below steps if you are not aware of how to add or edit environment variables on Windows.

1. Open the System Environment Variables window and select Environment Variables.

1. On the following Environment variable screen, add SPARK_HOME, HADOOP_HOME, JAVA_HOME by selecting the New option.

Note: The below images are with older versions. Make sure you are using the right folder/file names as per your setup.

1. This opens up the New User Variables window where you can enter the variable name and value.
2. Now Edit the PATH variable.

1. Add Spark, Java, and Hadoop bin location by selecting the New option.

Spark with winutils.exe on Windows

Many beginners think PySpark needs a Hadoop cluster installed to run but that’s not true; PySpark can run on AWS by using S3, and Azure by using blob storage without Hadoop and HDFSe.t.c.

To run PySpark on Windows, you need winutils.exe as it uses POSIX like file access operations in Windows using Windows API.

winutils.exe enables PySpark to use Windows-specific services including running shell commands on a Windows environment.

Download winutils.exe for Hadoop 3 and copy it to %SPARK_HOME%\bin folder. Winutils are different for each Hadoop version hence download the right version based on your Spark vs Hadoop distribution from this GitHub link.

PySpark shell

pyspark is a CLI utility that comes with Apache Spark distribution. Open command prompt, go to cd %SPARK_HOME%/bin, and type the pyspark command to run PySpark shell. You should see something like this below (ignore the error you see at the end). Sometimes it may take a minute or two for your Spark instance to initialize to get to the below screen.

pyspark shell also creates a Spark context web UI and by default, it can be accessed from http://localhost:4041.

On the pyspark command line, you can run any PySpark statements like creating an RDD, getting a PySpark version, etc.

>>> spark.version
'3.0.0'
>>> rdd = sc.parallelize([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
>>> rdd.collect()
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

This completes the installation of PySpark on Windows.

Web UI on Windows

PySpark provides a suite of Web UIs (Jobs, Stages, Tasks, Storage, Environment, Executors, and SQL) to monitor the status of your Spark application, resource consumption of the Spark cluster, and Spark configurations. On Spark Web UI, you can see how the operations are executed.

History Server

The history server keeps a log of all Spark applications you submit by spark-submit, spark-shell. You can enable Spark to collect the logs by adding the below configs to spark-defaults.conf file, which is located at %SPARK_HOME%/conf directory.

spark.eventLog.enabled true
spark.history.fs.logDirectory file:///c:/logs/path

After setting the above properties, start the history server by executing the below command.

$SPARK_HOME/bin/spark-class.cmd org.apache.spark.deploy.history.HistoryServer

By default, the History server listens at the 18080 port, and you can access it from the browser using http://localhost:18080/.

By clicking on each App ID, you will get the details of the application in Spark web UI.

Conclusion

In conclusion, setting up a PySpark environment on Windows involves several steps, including installing Java, downloading the appropriate Spark package, configuring environment variables, and ensuring compatibility with Windows through utilities like winutils.exe. Additionally, understanding how to utilize the PySpark shell and Web UI can significantly enhance your ability to monitor and manage Spark applications effectively.

Key Takeaways

• Install Java 21 or later to ensure compatibility with Spark.
• Download and extract Apache Spark, and configure environment variables correctly.
• Use winutils.exe for compatibility with Windows OS.
• Utilize the PySpark shell and Web UI to manage and monitor Spark jobs efficiently.
• Enable and access the History server for logging and detailed application insights.