Reading Text Files

First, create a text file containing the text you want to analyze. For this example, let's assume you have a text file named test.txt with the following content:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Using sparkContext.textFile()

For production applications, we mostly create RDD by using external storage systems like HDFS, S3, HBase, etc. So, in order to read from external sources and create an RDD, use sparkContext.textFile() method.

To make it simple for this PySpark tutorial, we are using files from the local system to create an RDD.

# Create RDD from external data source
rdd2 = sc.textFile("/path/test.txt")

# Collect RDD to driver
colData = rdd2.collect()

# Print the collected data to console
for data in colData:
    print(data)

# Output:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Note: Besides using text files, we can also create RDD from CSV files, JSON, and more formats.

Using pyspark.wholeTextFiles()

The wholeTextFiles() method returns a pairRDD with the key being the file path and the value being file content.

# Reads entire file into a RDD as single record.
rdd3 = sc.wholeTextFiles("/path/test.txt")

# Collect RDD to driver
colData = rdd3.collect()

# Print the collected data to console
for data in colData:
    print("File Name : " + data[0])
    print("File Data : " + data[1])

# Output:

File Name : file:/Users/admin/test.txt
File Data : Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s