Reading Text Files

First, create a text file containing the text you want to analyze. For this example, let's assume you have a text file named test.txt with the following content:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Using sparkContext.textFile()

For production applications, we mostly create RDD by using external storage systems like HDFS, S3, HBase, etc. So, in order to read from external sources and create an RDD, use sparkContext.textFile() method.

To make it simple for this Spark tutorial, we are using files from the local system to create an RDD.

// Create RDD from external data source
val rdd2 = spark.sparkContext.textFile("/path/test.txt")

// Collect RDD to driver
val colData = rdd2.collect()

// Print the collected data to console
colData.foreach(data =>{
  println(data)
})

# Output:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Note: Besides using text files, we can also create RDD from CSV files, JSON, and more formats.

Using sparkContext.wholeTextFiles()

The wholeTextFiles() method returns a pairRDD with the key being the file path and the value being file content.

// Reads entire file into a RDD as single record.
val rdd3 = spark.sparkContext.wholeTextFiles("/path/test.txt")

// Collect RDD to driver
val colData = rdd3.collect()

// Print the collected data to console
colData.foreach(data =>{
    println("File Name : "+data._1)
    println("File Data : "+data._2)
})

# Output:

File Name : file:/Users/admin/test.txt
File Data : Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s