RDD - Word Count Example - Apache Spark Tutorial with Scala Examples

RDD - Word Count Example

Word count example is a most asked interview question. In this lesson, we'll count the occurrences of each word in a text file. Here are the steps:

First, create a text file containing the text you want to analyze. For this example, let's assume you have a text file named test.txt with the following content:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Now, let's create a SparkSession and read this file into an RDD.

// Imports
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession 

// Create SparkSession 
val spark = SparkSession.builder().master("local[1]")
                .appName("AIStudyLabs.com")
                .getOrCreate()

// Load the input text file into an RDD
val rdd = spark.sparkContext.textFile("test.txt")

Now, let's follow the below steps to get the unique words and count of each word.

1. flatMap() Transformation

flatMap() transformation flattens the RDD after applying the function and returns a new RDD. In the below example, first, it splits each record by space in an RDD and finally flattens it. The resulting RDD consists of a single word on each record.

// Split each line into words and flatten the result
val rdd2 = rdd.flatMap(f=>f.split(" "))

2. map() Transformation

map() transformation is used the apply any complex operations like adding a column, or updating a column e.t.c, the output of map transformations would always have the same number of records as the input.

In our word count example, we are adding a new column with a value of 1 for each word, the result of the RDD is PairRDDFunctions which contains key-value pairs, word of type String as Key, and 1 of type Int as value. For your understanding, I’ve defined the rdd3 variable with type.

// Map each word to a key-value pair (word, 1)
val rdd3:RDD[(String,Int)] = rdd2.map(m=>(m,1))

3. reduceByKey() Transformation

reduceByKey() merges the values for each key with the function specified. In our example, it reduces the word string by applying the sum function to the value. The result of our RDD contains unique words and their count.

// Reduce by key to count the occurrences of each word
val rdd4 = rdd3.reduceByKey(_ + _)

4. Collect and Print the Result

Finally, collect the results and print them to the console.

// Collect the result
val result = rdd4.collect()

// Print the result to console
result.foreach(rec =>{
  println(rec._1 +"->"+rec._2)
})

Alternatively, you can write this RDD into a text file using the saveAsTextFile() method.

# Write final RDD back to text filer
dd4.saveAsTextFile("/tmp/word-count.txt")

In this example, we used Spark to read the input text file, split the lines into words, and then performed a word count using RDD transformations like flatMap, map, and reduceByKey. Finally, we collected and printed the results.

Keep in mind that this is a basic example. In a real-world scenario, you might want to clean and preprocess the data further, handle punctuation, and perform more advanced text analysis tasks as needed.

Complete Word Count Example

Following is a complete example of a word count by using several RDD transformations.

// Imports
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession

object TestRDD extends App {

   // Create SparkSession
   val spark = SparkSession.builder().master("local[1]")
     .appName("AIStudyLabs.com")
     .getOrCreate()

   // Load the input text file into an RDD
   val rdd = spark.sparkContext.textFile("test.txt")

   // Split each line into words and flatten the result
   val rdd2 = rdd.flatMap(f=>f.split(" "))

   // Map each word to a key-value pair (word, 1)
   val rdd3:RDD[(String,Int)] = rdd2.map(m=>(m,1))

   // Reduce by key to count the occurrences of each word
   val rdd4 = rdd3.reduceByKey(_ + _)

  // Collect the result
  val result = rdd4.collect()

  // Print the result to console
  result.foreach(rec =>{
    println(rec._1 +"->"+rec._2)
  }
}

This example yields the below output:

// Output:
Gutenberg’s->3	
Wonderland->2		
in->2		
Alice’s->1		
Project->3		
Adventures->2

RDD - Word Count Example