PySpark SparkSession

SparkSession was introduced in version Spark 2.0. It provides an entry point to underlying Spark functionality to programmatically create Spark RDD, DataFrame, and DataSet. SparkSession’s object spark is the default variable available in pysparkshell and it can be created programmatically using the SparkSession builder pattern.

SparkSession in Spark 2.0

• With Spark 2.0, a new class pyspark.sql.SparkSession was introduced as a unified class for all the different contexts we had before Spark 2.0 (such as SQLContext, HiveContext). Hence, Spark Session can replace SQLContext, HiveContext, and other contexts.
• As mentioned earlier, SparkSession is an entry point to Spark, and creating a SparkSession instance is the first step to program with RDD, DataFrame, and Dataset. Under PySpark, this is how it’s done:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("example") \
    .getOrCreate()

• Prior to Spark 2.0, SparkContext was the main entry point. It hasn't been completely replaced by SparkSession; many features of SparkContext are still available in Spark 2.0 and later. Note that SparkSession internally creates SparkContext with the configuration provided.

sc = spark.sparkContext

• Spark Session in PySpark includes all the APIs available in different contexts:
  • SparkContext
  • SQLContext (now part of SparkSession)
  • HiveContext (accessible if Hive support is enabled)

How many SparkSessions can you create in an application?

In a PySpark application, you can create multiple SparkSession instances using either SparkSession.builder() or SparkSession.newSession(). This is useful when you need to keep Spark tables (relational entities) logically separated.

Summary and Key Takeaways

In this lesson, we explored what SparkSession is and how it serves as a unified entry point in Apache Spark 2.0 and later. SparkSession combines multiple contexts into a single entry point and offers various APIs for Spark functionalities.