SparkSession Most Used Methods

The following are the most used methods of SparkSession. For the complete list, refer to the PySpark documentation.

You don't have to run these examples explained here, will explain these methods while explaining other concepts in this course.

version – Returns the PySpark version where your application is running, likely the version your cluster is configured with. You can access it using:

spark.version

catalog – Returns the catalog object to access metadata. It can be accessed with:

spark.catalog

conf – Returns the runtime configuration object, accessed via:

spark.conf

builder() – Used to create a new SparkSession. Use it as follows:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ExampleApp").getOrCreate()

newSession() – Creates a new SparkSession:

new_spark = spark.newSession()

range(n) – Returns a single column DataFrame with LongType and column named id, containing elements in a range from 0 to n (exclusive) with step value 1:

spark.range(10).show()

createDataFrame() – Creates a DataFrame from a collection or RDD:

data = [("John", 25), ("Doe", 29)]
df = spark.createDataFrame(data, ["Name", "Age"])

emptyDataFrame() – Use createDataFrame() to create an empty DataFrame:

empty_df = spark.createDataFrame([], schema="Name STRING, Age INT")

getActiveSession() – Returns an active Spark session for the current thread, accessed as:

SparkSession.getActiveSession()

getDefaultSession() – Returns the default SparkSession created by the builder:

SparkSession.getDefaultSession()

read() – Returns a DataFrameReader to read records from various file formats into DataFrame:

df = spark.read.csv("file.csv")

readStream() – Returns a DataStreamReader, used for reading streaming data into a DataFrame:

stream_df = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).load()

sparkContext() – Returns a SparkContext:

sc = spark.sparkContext

sql(String sql) – Executes the given SQL query string and returns a DataFrame:

sql_df = spark.sql("SELECT * FROM my_table")

stop() – Stops the current SparkSession:

spark.stop()

table() – Returns a DataFrame of a table or view:

table_df = spark.table("my_table")

udf() – Creates a user-defined function (UDF) for use in DataFrame transformations:

from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

def squared(x):
    return x * x

squared_udf = udf(squared, IntegerType())

I have used most of these methods in detail with examples while teaching Spark SQL & DataFrames.

Summary and Key Takeaways

• This lesson covers the most frequently used methods of SparkSession such as version, catalog, builder, and more.
• These methods help manage sessions, configure settings, create and handle DataFrames, read data, run SQL queries, and more within PySpark.
• Understanding these methods is essential for efficiently utilizing PySpark's capabilities in data processing and analysis.