RDD - Word Count Example - Apache Spark Tutorial with Java Examples

RDD - Word Count Example

Word count example is a frequently asked interview question. In this lesson, we'll count the occurrences of each word in a text file. Here are the steps:

First, create a text file containing the text you want to analyze. For this example, assume you have a text file named test.txt with the following content:

Project Gutenberg’s
Alice’s Adventures in Wonderland
Project Gutenberg’s
Adventures in Wonderland
Project Gutenberg’s

Now, let's create a SparkSession and read this file into an RDD.

// Imports
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SparkSession;

// Create SparkSession
SparkSession spark = SparkSession.builder().master("local[1]")
                .appName("AIStudyLabs.com")
                .getOrCreate();

// Create JavaSparkContext from SparkSession
JavaSparkContext sparkContext = new JavaSparkContext(spark.sparkContext());

// Load the input text file into an RDD
JavaRDD<String> rdd = sparkContext.textFile("test.txt");

Follow the steps below to get the unique words and count of each word.

1. flatMap() Transformation

The flatMap() transformation flattens the RDD after applying the function and returns a new RDD. In the example below, it splits each record by a space in an RDD and finally flattens it. The resulting RDD consists of a single word on each record.

// Split each line into words and flatten the result
JavaRDD<String> rdd2 = rdd.flatMap(line -> Arrays.asList(line.split(" ")).iterator());

2. mapToPair() Transformation

The mapToPair() transformation is used to apply any operations that involve generating key-value pairs. In our word count example, we add a key-value pair of (word, 1) for each word. The result is an RDD that contains key-value pairs, where the word is the key, and the count, initialized to 1, is the value.

// Map each word to a key-value pair (word, 1)
JavaPairRDD<String, Integer> rdd3 = rdd2.mapToPair(word -> new Tuple2<>(word, 1));

3. reduceByKey() Transformation

The reduceByKey() method merges the values for each key using the specified function. In our example, it reduces the word string by applying the sum function to the value. The result of our RDD contains unique words and their counts.

// Reduce by key to count the occurrences of each word
JavaPairRDD<String, Integer> rdd4 = rdd3.reduceByKey(Integer::sum);

4. Collect and Print the Result

Finally, collect the results and print them to the console.

// Collect the result
List<Tuple2<String, Integer>> result = rdd4.collect();

// Print the result to console
result.forEach(record -> {
  System.out.println(record._1() + "->" + record._2());
});

Alternatively, you can write this RDD into a text file using the saveAsTextFile() method.

// Write final RDD back to text file
rdd4.saveAsTextFile("/tmp/word-count.txt");

In this example, we used Spark to read the input text file, split the lines into words, and then performed a word count using RDD transformations like flatMap, mapToPair, and reduceByKey. Finally, we collected and printed the results.

Keep in mind that this is a basic example. In a real-world scenario, you might want to clean and preprocess the data further, handle punctuation, and perform more advanced text analysis tasks as needed.

Complete Word Count Example

Following is a complete example of a word count using several RDD transformations.

// Imports
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SparkSession;
import scala.Tuple2;

import java.util.Arrays;
import java.util.List;

public class TestRDD {

    public static void main(String[] args) {

        // Create SparkSession
        SparkSession spark = SparkSession.builder()
                .master("local[1]")
                .appName("AIStudyLabs.com")
                .getOrCreate();

        // Create JavaSparkContext from SparkSession
        JavaSparkContext sparkContext = new JavaSparkContext(spark.sparkContext());

        // Load the input text file into an RDD
        JavaRDD<String> rdd = sparkContext.textFile("test.txt");

        // Split each line into words and flatten the result
        JavaRDD<String> rdd2 = rdd.flatMap(line -> Arrays.asList(line.split(" ")).iterator());

        // Map each word to a key-value pair (word, 1)
        JavaPairRDD<String, Integer> rdd3 = rdd2.mapToPair(word -> new Tuple2<>(word, 1));

        // Reduce by key to count the occurrences of each word
        JavaPairRDD<String, Integer> rdd4 = rdd3.reduceByKey(Integer::sum);

        // Collect the result
        List<Tuple2<String, Integer>> result = rdd4.collect();

        // Print the result to console
        result.forEach(record -> {
            System.out.println(record._1() + "->" + record._2());
        });
    }
}

This example yields the following output:

// Output:
Gutenberg’s->3	
Wonderland->2		
in->2		
Alice’s->1		
Project->3		
Adventures->2

RDD - Word Count Example