Introduction

pyspark.SparkContext is an entry point to the PySpark functionality that is used to communicate with the cluster and to create an RDD, accumulator, and broadcast variables.

Note: You can create only one active SparkContext per JVM. You should stop the active SparkContext with the stop() method before creating a new one.

The Spark driver program creates and uses SparkContext to connect to the cluster manager to submit PySpark jobs and know what resource manager (YARN, Mesos, or Standalone) to communicate to. It is the heart of the PySpark application.

Related: How to get current SparkContext & its configurations in Spark

SparkContext in PySpark shell

By default, the PySpark shell creates and provides sc object, which is an instance of the SparkContext class. We can directly use this object where required without the need to create.

sc.appName

Similar to the PySpark shell, in most of the tools, notebooks, and Azure Databricks, the environment itself creates a default SparkContext object for us to use.

Create SparkContext

Since PySpark 2.0, Creating a SparkSession creates a SparkContext internally and exposes the sparkContext variable to use. Hence, you don't have to explicitly create a SparkContext object.

# Create SparkSession from builder
from pyspark.sql import SparkSession

spark = SparkSession.builder.master("local[1]") \
                    .appName('AIStudyLabs.com') \
                    .getOrCreate()

# Access sparkContext variable
print(spark.sparkContext)
print("Spark App Name : " + spark.sparkContext.appName)

# Outputs:
# <SparkContext master=local[1] appName=AIStudyLabs.com>
# Spark App Name : AIStudyLabs.com

Note: As I explained in the SparkSession article, you can create any number of SparkSession objects however, for all those objects underlying there will be only one SparkContext.

Stop PySpark SparkContext

You can stop the SparkContext by calling the stop() method. As explained above you can have only one SparkContext per JVM. If you want to create another, you need to shut down it first by using the stop() method and create a new SparkContext.

# SparkContext stop() method
spark.sparkContext.stop()

When PySpark executes this statement, it logs the message INFO SparkContext: Successfully stopped SparkContext to the console or to a log file.

When you try to create multiple SparkContext you will get the below error.

ValueError: Cannot run multiple SparkContexts at once;

Summary and Key Takeaways

• pyspark.SparkContext serves as an entry point for cluster communication and creating core items like RDDs.
• Only one active SparkContext can exist per JVM at any given time; to create another, the existing one must be stopped.
• PySpark shell and many environments provide a default SparkContext object, eliminating the need for manual creation.
• Creating a SparkSession in PySpark automatically provides a sparkContext variable for use.
• Attempting to create multiple SparkContexts results in an error, emphasizing the importance of managing SparkContext carefully.