SparkContext Most Used Methods

The following are the most commonly used methods of SparkContext. For the complete list, refer to Spark documentation.

longAccumulator() – It creates an accumulator variable of a long data type. Only a driver can access accumulator variables.

doubleAccumulator() – It creates an accumulator variable of a double data type. Only a driver can access accumulator variables.

applicationId – Returns a unique ID of a Spark application.

appName – Return an app name that was given when creating SparkContext

broadcast – read-only variable broadcast to the entire cluster. You can broadcast a variable to a Spark cluster only once.

emptyRDD – Creates an emptyRDD

getPersistentRDDs – Returns all persisted RDDs

getOrCreate() – Creates or returns a SparkContext

hadoopFile – Returns an RDD of a Hadoop file

master()–  Returns master that set while creating SparkContext

newAPIHadoopFile – Creates an RDD for a Hadoop file with a new API InputFormat.

sequenceFile – Get an RDD for a Hadoop SequenceFile with given key and value types.

setLogLevel – Change log level to debug, info, warn, fatal, and error

textFile – Reads a text file from HDFS, local or any Hadoop-supported file systems, and returns an RDD

union – Union two RDDs

wholeTextFiles – Reads a text file in the folder from HDFS, local, or any Hadoop-supported file systems and returns an RDD of Tuple2. The first element of the tuple consists file name and the second element consists context of the text file.