SparkContext Most Used Methods
The following are the most commonly used methods of JavaSparkContext. For the complete list, refer to Spark documentation.
longAccumulator() – It creates an accumulator variable of a long data type. Only a driver can access accumulator variables.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
LongAccumulator longAcc = sc.sc().longAccumulator();
doubleAccumulator() – It creates an accumulator variable of a double data type. Only a driver can access accumulator variables.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
DoubleAccumulator doubleAcc = sc.sc().doubleAccumulator();
applicationId – Returns a unique ID of a Spark application.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
String appId = sc.applicationId();
appName – Return an app name that was given when creating JavaSparkContext.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
String appName = sc.appName();
broadcast – Read-only variable broadcast to the entire cluster. You can broadcast a variable to a Spark cluster only once.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
Broadcast<String> broadcastVar = sc.broadcast("Broadcast Value");
emptyRDD – Creates an emptyRDD.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> emptyRDD = sc.emptyRDD();
getPersistentRDDs – Returns all persisted RDDs.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
Map<Integer, JavaRDD<?>> persistedRDDs = sc.getPersistentRDDs();
getOrCreate() – Creates or returns a JavaSparkContext.
Example:
JavaSparkContext sc = JavaSparkContext.getOrCreate(conf);
hadoopFile – Returns an RDD of a Hadoop file.
Example:
JavaRDD<Text> hadoopRDD = sc.hadoopFile("hdfs://input/path", TextInputFormat.class, LongWritable.class, Text.class).values();
master() – Returns master that set while creating JavaSparkContext.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
String master = sc.master();
newAPIHadoopFile – Creates an RDD for a Hadoop file with a new API InputFormat.
Example:
JavaRDD<String> newHadoopRDD = sc.newAPIHadoopFile("hdfs://input/path", NewInputFormat.class, Text.class, LongWritable.class).map(tuple -> tuple._1.toString());
sequenceFile – Get an RDD for a Hadoop SequenceFile with given key and value types.
Example:
JavaRDD<Tuple2<Text, Text>> sequenceRDD = sc.sequenceFile("hdfs://input/path", Text.class, Text.class);
setLogLevel – Change log level to debug, info, warn, fatal, and error.
Example:
JavaSparkContext sc = new JavaSparkContext(conf);
sc.setLogLevel("ERROR");
textFile – Reads a text file from HDFS, local, or any Hadoop-supported file systems, and returns an RDD.
Example:
JavaRDD<String> textRDD = sc.textFile("hdfs://input/path");
union – Union two RDDs.
Example:
JavaRDD<String> rdd1 = sc.textFile("hdfs://input/path1");
JavaRDD<String> rdd2 = sc.textFile("hdfs://input/path2");
JavaRDD<String> unionRDD = rdd1.union(rdd2);
wholeTextFiles – Reads a text file in the folder from HDFS, local, or any Hadoop-supported file systems and returns an RDD of Tuple2. The first element of the tuple consists of the file name and the second element consists of the content of the text file.
Example:
JavaPairRDD<String, String> filesRDD = sc.wholeTextFiles("hdfs://input/folder");