Spark SparkSession

Since Apache Spark 2.0, SparkSession has become a unified entry point to Spark to work with RDD, DataFrame, and Dataset. Prior to 2.0, SparkContext used to be an entry point. spark-shell which is a CLI to interact with Spark provides a default Spark Session ‘spark‘ variable hence you don’t have to create one if you are using spark-shell.

Note: The Apache Spark doesn't support Java in spark-shell CLI. you have to Scala when you are using spark-shell. In the course, we utilize spark-shell just to explain the setup and some basic concepts and I will use Scala in these cases. If you are not familiar with Scala please skip these sections.

What is SparkSession

SparkSession was introduced in version Spark 2.0, It is an entry point to underlying Spark functionality in order to programmatically create Spark RDD, DataFrame, and DataSet. SparkSession’s object spark is the default variable available in spark-shell and it can be created programmatically using SparkSession builder pattern.

SparkSession in Spark 2.0

• With Spark 2.0 a new class org.apache.spark.sql.SparkSession has been introduced which is a combined class for all the different contexts we used to have prior to 2.0 (SQLContext, HiveContext. StreamingContext e.t.c) release hence, Spark Session can be used in the place of SQLContext, HiveContext, and other contexts.
• As mentioned in the beginning, SparkSession is an entry point to Spark, and creating a SparkSession instance would be the first statement you would write to program with RDD, DataFrame, and Dataset. SparkSession will be created using SparkSession.builder() builder pattern.
• Prior to Spark 2.0, SparkContext used to be an entry point, and it’s not been completely replaced with SparkSession. Many features of SparkContext are still available and used in Spark 2.0 and later. You should also know that SparkSession internally creates SparkConfig and SparkContext with the configuration provided with SparkSession.
• Spark Session also includes all the APIs available in different contexts –
  • SparkContext
  • SQLContext
  • StreamingContext
  • HiveContext

How many SparkSessions can you create in an application?

You can create as many SparkSession as you want in a Spark application using either SparkSession.builder() or SparkSession.newSession(). Many Spark session objects are required when you want to keep Spark tables (relational entities) logically separated.

Table: Differences between MapReduce and Apache Spark

| Feature | MapReduce | Apache Spark | | -------------------- | ----------------- | ---------------------- | | Processing Model | Disk-based | In-memory | | Speed | Slower | Faster | | Iterative Processing | Not efficient | Efficient | | Fault Tolerance | High | Moderate | | Ease of Use | Complex API | Simple API | | Language Support | Java, Python, C++ | Java, Python, Scala, R | | Streaming Support | Limited | Advanced |

Summary and Key Takeaways

In this lesson, we explored what SparkSession is and how it serves as a unified entry point in Apache Spark 2.0 and later. SparkSession combines multiple contexts into a single entry point and offers various APIs for Spark functionalities. We also compared MapReduce and Apache Spark to highlight their differences.