SparkSession Most Used Methods
The following are the most used methods of SparkSession. For the complete list, refer to Spark documentation.
version – Returns Spark version where your application is running, probably the Spark version your cluster is configured with.
catalog – Returns the catalog object to access metadata.
conf – Returns the RuntimeConfig object.
builder() – builder() is used to create a new SparkSession, this return SparkSession.Builder
newSession() – Creaetes a new SparkSession.
range(n) – Returns a single column Dataset with LongType and column named id, containing elements in a range from 0 to n (exclusive) with step value 1. There are several variations of this function, refer to Spark documentation.
createDataFrame() – This creates a DataFrame from a collection and an RDD
emptyDataFrame() – Creates an empty DataFrame.
emptyDataset() – Creates an empty Dataset.
getActiveSession() – Returns an active Spark session for the current thread.
getDefaultSession() – Returns the default SparkSession that is returned by the builder.
implicits() – To access the nested Scala object.
read() – Returns an instance of DataFrameReader class, this is used to read records from CSV, Parquet, Avro, and more file formats into DataFrame.
readStream() – Returns an instance of DataStreamReader class, this is used to read streaming data. that can be used to read streaming data into DataFrame.
sparkContext() – Returns a SparkContext.
sql(String sql) – Returns a DataFrame after executing the SQL mentioned.
sqlContext() – Returns SQLContext.
stop() – Stop the current SparkSession.
table() – Returns a DataFrame of a table or view.
udf() – Creates a Spark UDF to use it on DataFrame, Dataset, and SQL.
I have used most of these methods in detail with examples while teaching Spark SQL & DataFrames.