Built with Tutify.io

    AIStudyLabs

    Spark SqlContext explained with Examples

    Spark SqlContext explained with Examples

    In Spark Version 1.0 SQLContext (org.apache.spark.sql.SQLContext ) is an entry point to SQL in order to work with structured data (rows and columns) however with 2.0 SQLContext has been replaced with SparkSession.

    Spark org.apache.spark.sql.SQLContext is a deprecated class that contains several useful functions to work with Spark SQL and it is an entry point o Spark SQL however, this has been deprecated since Spark 2.0 and recommends using SparkSession.

    <a id="h-sqlcontext-in-spark-shell"></a>

    SQLContext in spark-shell

    You can create an SQLContext in Spark shell by passing a default SparkContext object (sc) as a parameter to the SQLContext constructor.

    
scala> val sqlcontext = new org.apache.spark.sql.SQLContext(sc)

    <a id="h-creating-sqlcontext-from-scala-program"></a>

    Creating SQLContext from Scala program

    In Spark 1.0, you would need to pass a SparkContext object to a constructor in order to create SQL Context instance, In Scala, you do this as explained in the below example.

    
val spark = SparkSession.builder()
    .master("local[1]")
    .appName("AIStudyLabs.com")
    .getOrCreate();
val sqlContext = new org.apache.spark.sql.SQLContext(spark.sparkContext)

    However, In 2.0 SQLContext() constructor has been deprecated and recommend to use sqlContext method from SparkSession for example spark.sqlContext

    
val sqlContext = spark.sqlContext

    <a id="h-create-a-dataframe-by-reading-a-file"></a>

    Create a DataFrame by reading a file

    
  val sqlContext:SQLContext = spark.sqlContext

  //read csv with options
  val df = sqlContext.read.options(Map("inferSchema"->"true","delimiter"->",","header"->"true"))
    .csv("src/main/resources/zipcodes.csv")
  df.show()
  df.printSchema()

    <a id="h-execute-sql-query-from-sqlcontext-object"></a>

    Execute SQL query from SQLContext object

    
  df.createOrReplaceTempView("TAB")
  sqlContext.sql("select * from TAB")
    .show(false)

    Note: Since 2.0, SQLContext is replaced by SparkSession and SparkSession contains all methods that are present in SQLContext.

    <a id="h-complete-sqlcontext-example"></a>

    Complete SQLContext Example

    
package com.aistudylabs.spark

import org.apache.spark.sql.{SQLContext, SparkSession}

object SQLContextExample extends App {

  val spark = SparkSession.builder()
    .master("local[1]")
    .appName("AIStudyLabs.com")
    .getOrCreate();

  spark.sparkContext.setLogLevel("ERROR")

  val sqlContext:SQLContext = spark.sqlContext

  //read csv with options
  val df = sqlContext.read.options(Map("inferSchema"->"true","delimiter"->",","header"->"true"))
    .csv("src/main/resources/zipcodes.csv")
  df.show()
  df.printSchema()

  df.createOrReplaceTempView("TAB")
  sqlContext.sql("select * from TAB")
    .show(false)

}

    This example is also available at GitHub project for reference

    <a id="h-conclusion"></a>

    Conclusion

    In this article, you have learned how to create an SQLContext object from Spark shell and through programming using Scala example and also learned how to read a file and create a DataFrame. Finally learned SQLContext has been deprecated and to use SparkSession instead.

    <a id="h-related-articles"></a>

    Related Articles

    <a id="h-references"></a>

    References

    Happy Learning !!