SparkSession in spark-shell

By default, Spark shell provides spark object which is an instance of the SparkSession class. We can directly use this object when required in spark-shell

// Usage of spark variable
scala> spark.version

Similar to the Spark shell, In most of the tools, notebooks, and Azure Databricks, the environment itself creates a default SparkSession object for us to use so you don’t have to worry about creating a Spark session.

Create SparkSession From Scala Program

To create SparkSession in Scala or Python, you need to use the builder pattern method builder() and calling getOrCreate() method. If SparkSession already exists it returns otherwise creates a new SparkSession.

// Imports
import org.apache.spark.sql.SparkSession
object SparkSessionTest extends App { 
    // Create SparkSession object
    val spark = SparkSession.builder().master("local[1]")
             .appName("AIStudyLabs.com").getOrCreate() 
     println(spark)  
     println("Spark Version : "+spark.version)
}

// Output:
// org.apache.spark.sql.SparkSession@2fdf17dc
// Spark Version : 4.1.1

From the above code: 

SparkSession.builder() – Return SparkSession.Builder class. This is a builder for SparkSession. master(), appName(), and getOrCreate() are methods of SparkSession.Builder.

master() – This allows Spark applications to connect and run in different modes (local, standalone cluster, Mesos, YARN), depending on the configuration.

• Use local[x] when running on your local laptop. x should be an integer value and should be greater than 0; this represents how many partitions it should create when using RDD, DataFrame, and Dataset. Ideally, x value should be the number of CPU cores you have.
• For standalone use spark://master:7077
• appName() – Sets a name to the Spark application that shows in the Spark web UI. If no application name is set, it sets a random name.
• getOrCreate() – This returns a SparkSession object if already exists. Creates a new one if not exist.

Get Existing SparkSession

You can get the existing SparkSession in Scala programmatically using the below example.

// Get existing SparkSession 
import org.apache.spark.sql.SparkSession
val spark2 = SparkSession.builder().getOrCreate()
print(spark2)

// Output:
// org.apache.spark.sql.SparkSession@2fdf17dc

Compare the hash with the previous example. Since it returned the existing session object, both objects spark and spark2 have the same hash value.

Create Another SparkSession

Sometimes you might be required to create multiple sessions, which you can easily achieve by using newSession() method. This uses the same app name, master as the existing session. Underlying SparkContext will be the same for both sessions as you can have only one context per Spark application.

// Create a new SparkSession
val spark3 = spark.newSession()
print(spark3)

// Output:
// org.apache.spark.sql.SparkSession@692dba54

Compare this hash with the hash from the above example, it should be different.

Setting Spark Configs

If you want to set some configs to SparkSession, use the config() method.

// Usage of config()
val spark = SparkSession.builder().master("local[1]").appName("AIStudyLabs.com")                     .config("spark.some.config.option","config-value").getOrCreate()

Summary and Key Takeaways

• SparkSession is a unified entry point for programming Spark applications.
• Spark shell, notebooks, and other environments automatically provide SparkSession objects.
• Through the builder pattern, creating a SparkSession is straightforward, supporting various configurations.