Reading Text Files

First, create a text file containing the text you want to analyze. For this example, let's assume you have a text file named test.txt with the following content:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Using sparkContext.textFile()

For production applications, we often create RDDs by using external storage systems like HDFS, S3, HBase, etc. To read from external sources and create an RDD, use the sparkContext.textFile() method.

To simplify this Spark tutorial, we are using files from the local system to create an RDD.

// Create a SparkConf and JavaSparkContext
SparkConf conf = new SparkConf().setAppName("ReadTextFile").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);

// Create RDD from an external data source
JavaRDD<String> rdd2 = sc.textFile("/path/test.txt");

// Collect RDD to driver
List<String> colData = rdd2.collect();

// Print the collected data to console
for (String data : colData) {
    System.out.println(data);
}

sc.close();

# Output:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Note: Besides using text files, we can also create RDDs from CSV files, JSON, and other formats.

Using sparkContext.wholeTextFiles()

The wholeTextFiles() method returns a JavaPairRDD with the key being the file path and the value being the file content.

// Create a SparkConf and JavaSparkContext
SparkConf conf = new SparkConf().setAppName("ReadWholeTextFiles").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);

// Reads entire file into a RDD as a single record
JavaPairRDD<String, String> rdd3 = sc.wholeTextFiles("/path/test.txt");

// Collect RDD to driver
List<Tuple2<String, String>> colData = rdd3.collect();

// Print the collected data to console
for (Tuple2<String, String> data : colData) {
    System.out.println("File Name : " + data._1());
    System.out.println("File Data : " + data._2());
}

sc.close();

# Output:

File Name : file:/Users/admin/test.txt
File Data : Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s