Built with Tutify.io

    AIStudyLabs

    Run Spark Hello World Example in IntelliJ

    Run Spark Hello World Example in IntelliJ

    I will explain how to run Apache Spark Hello world example in IntelliJ on Windows using Scala & Maven. I created a Spark basic example in Apache Spark GitHub Examples project hence, I will clone this and use it to make it simple.

    A simple "Hello, World!" example in Apache Spark typically involves setting up a Spark session, Creating an RDD or DataFrame, and performing a basic transformation on a dataset.

    Make sure you have the IntelliJ IDE Setup and run Spark Application with Scala on Windows before you proceed.

    <a id="h-1-spark-maven-dependency"></a>

    1. Spark Maven Dependency

    In order to run the Apache Spark Hello World Example on IntelliJ, you would need to have the below Scala and Spark Maven dependencies.

    
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>

    <dependency>
      <groupId>org.specs</groupId>
      <artifactId>specs</artifactId>
      <version>1.2.5</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
      <scope>compile</scope>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
      <scope>compile</scope>
    </dependency>

    <a id="h-2-apache-spark-hello-world-example"></a>

    2. Apache Spark Hello World Example

    In other languages to demonstrate Hello World, we would just print the Hello World statement in the console, since Apache Spark is a framework to process data in memory, I will show how to create a Spark Session object and print some details from the Spark session object.

    
import org.apache.spark.sql.SparkSession

object SparkSessionTest {

  def main(args:Array[String]): Unit ={

    val spark = SparkSession.builder()
      .master("local[1]")
      .appName("SparkByExample")
      .getOrCreate();
    
    println("First SparkContext:")
    println("APP Name :"+spark.sparkContext.appName);
    println("Deploy Mode :"+spark.sparkContext.deployMode);
    println("Master :"+spark.sparkContext.master);

    val sparkSession2 = SparkSession.builder()
      .master("local[1]")
      .appName("SparkByExample-test")
      .getOrCreate();

    println("Second SparkContext:")
    println("APP Name :"+sparkSession2.sparkContext.appName);
    println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
    println("Master :"+sparkSession2.sparkContext.master);

  }
}

    <a id="h-3-spark-github-clone-hello-world-example-project"></a>

    3. Spark GitHub Clone - Hello World Example Project

    To make things simple, I have created a Spark Hello World project in GitHub and I will use this to run the example. First let's clone the project, build, and run.

    Using this option, we are going to import the project directly from the GitHub repository.

    
https://github.com/spark-examples/spark-hello-world-example

    Once the cloning completes, you will see the project workspace structure on IntelliJ.

    <a id="h-4-run-maven-build"></a>

    4. Run Maven build

    Now run the Maven build. First, select the Maven from the right corner, navigate to Lifecycle > install, right-click, and select Run Maven Build.

    This downloads all dependencies mentioned in the pom.xml file and compiles all examples in this tutorial. This also takes a few minutes to complete and you should see the below message after a successful build.

    <a id="h-5-run-apache-spark-hello-world-program"></a>

    5. Run Apache Spark Hello World Program

    After a successful Maven build, run src/main/scala/com.aistudylabs.spark.SparkSessionTest

    In case you still get errors during the running of the Spark application, please restart the IntelliJ IDE and run the application again. Now you should see the below message in the console.

    <a id="h-6-create-dataframe"></a>

    6. Create DataFrame

    To complete this Apache Spark Hello World program let's create a RDD and create a DataFrame from it; run the below program and explore the output.

    
// Import
import org.apache.spark.sql.SparkSession

// Create SparkSession and Prepare Data
val spark:SparkSession = SparkSession.builder()
   .master("local[1]").appName("AIStudyLabs.com")
   .getOrCreate()

import spark.implicits._
val columns = Seq("language","users_count")
val data = Seq(("Java", "20000"), ("Python", "100000"), ("Scala", "3000"))

// Spark Create DataFrame from RDD
val rdd = spark.sparkContext.parallelize(data)

// Create DataFrame
val dfFromRDD1 = rdd.toDF()
dfFromRDD1.printSchema()

    <a id="h-where-to-go-next"></a>

    Where to go next?

    Once you are able to run the Spark Hello Work example, you should readSpark RDD, Create Spark DataFrame, How to read CSV file into Spark

    Happy Learning !!

    <a id="h-related-articles"></a>

    Related Articles