SparkSession in spark-shell

By default, Spark shell provides spark object which is an instance of the SparkSession class. We can directly use this object when required in spark-shell.

// Usage of spark variable
scala> spark.version

Similar to the Spark shell, in most of the tools, notebooks, and Azure Databricks, the environment itself creates a default SparkSession object for us to use, so you don’t have to worry about creating a Spark session.

Create SparkSession From Java Program

To create SparkSession in Java, you use the builder pattern method builder() and call the getOrCreate() method. If a SparkSession already exists, it returns it; otherwise, it creates a new SparkSession.

// Imports
import org.apache.spark.sql.SparkSession;

public class SparkSessionTest {
    public static void main(String[] args) {
        // Create SparkSession object
        SparkSession spark = SparkSession.builder().master("local[1]")
                .appName("AIStudyLabs.com").getOrCreate();
        System.out.println(spark);
        System.out.println("Spark Version : " + spark.version());
    }
}

// Output:
// org.apache.spark.sql.SparkSession@2fdf17dc
// Spark Version : 4.1.1

From the above code: 

SparkSession.builder() – Returns a SparkSession.Builder class. This is a builder for SparkSession. master(), appName(), and getOrCreate() are methods of SparkSession.Builder.

master() – This allows Spark applications to connect and run in different modes (local, standalone cluster, Mesos, YARN), depending on the configuration.

• Use local[x] when running on your local machine. x should be an integer value greater than 0; this represents how many partitions it should create when using RDD, DataFrame, and Dataset. Ideally, x value should be the number of CPU cores you have.
• For standalone use spark://master:7077
• appName() – Sets a name to the Spark application that shows in the Spark web UI. If no application name is set, it sets a random name.
• getOrCreate() – This returns a SparkSession object if it already exists. Creates a new one if not.

Get Existing SparkSession

You can get the existing SparkSession in a Java programmatically using the example below.

// Get existing SparkSession 
import org.apache.spark.sql.SparkSession;

public class ExistingSparkSession {
    public static void main(String[] args) {
        SparkSession spark2 = SparkSession.builder().getOrCreate();
        System.out.println(spark2);
    }
}

// Output:
// org.apache.spark.sql.SparkSession@2fdf17dc

Compare the hash with the previous example. Since it returned the existing session object, both objects spark and spark2 have the same hash value.

Create Another SparkSession

Sometimes you might be required to create multiple sessions, which you can easily achieve by using the newSession() method. This uses the same app name, and master as the existing session. The underlying SparkContext will be the same for both sessions as you can have only one context per Spark application.

// Create a new SparkSession
SparkSession spark3 = spark.newSession();
System.out.println(spark3);

// Output:
// org.apache.spark.sql.SparkSession@692dba54

Compare this hash with the hash from the above example; it should be different.

Setting Spark Configs

If you want to set some configs to SparkSession, use the config() method.

// Usage of config()
SparkSession spark = SparkSession.builder().master("local[1]").appName("AIStudyLabs.com")
        .config("spark.some.config.option", "config-value").getOrCreate();

Summary and Key Takeaways

• SparkSession is a unified entry point for programming Spark applications.
• Spark shell, notebooks, and other environments automatically provide SparkSession objects.
• Through the builder pattern, creating a SparkSession is straightforward, supporting various configurations.