Built with Tutify.io

    AIStudyLabs

    Spark - explode Array of Map to rows

    Spark - explode Array of Map to rows

    Problem: How to explode the Array of Map DataFrame columns to rows using Spark.

    Solution: Spark explode function can be used to explode an Array of Map ArrayType(MapType) columns to rows on Spark DataFrame using scala example.

    Before we start, let's create a DataFrame with map column in an array. From below example column "properties" is an array of MapType which holds properties of a person with key & value pair.

    
  val arrayMapSchema = new StructType().add("name",StringType)
    .add("properties",ArrayType(new MapType(StringType,StringType,true)))

  val arrayMapData = Seq(
    Row("James",List(Map("hair"->"black","eye"->"brown"), Map("height"->"5.9"))),
    Row("Michael",List(Map("hair"->"brown","eye"->"black"),Map("height"->"6"))),
    Row("Robert",List(Map("hair"->"red","eye"->"gray"),Map("height"->"6.3")))
  )

  val df = spark.createDataFrame(
    spark.sparkContext.parallelize(arrayMapData),arrayMapSchema)
  df.printSchema()
  df.show(false)

    df.printSchema() and df.show() returns the following schema and table.

    
root
 |-- name: string (nullable = true)
 |-- properties: array (nullable = true)
 |    |-- element: map (containsNull = true)
 |    |    |-- key: string
 |    |    |-- value: string (valueContainsNull = true)

+-------+------------------------------------------------+
|name   |properties                                      |
+-------+------------------------------------------------+
|James  |[[hair -> black, eye -> brown], [height -> 5.9]]|
|Michael|[[hair -> brown, eye -> black], [height -> 6]]  |
|Robert |[[hair -> red, eye -> gray], [height -> 6.3]]   |
+-------+------------------------------------------------+

    Now, let's explode "properties" array column to map rows. after exploding, it creates a new column 'col' with rows represents a map.

    
  import spark.implicits._

  val df2 = df.select($"name",explode($"properties"))
  df2.printSchema()
  df2.show(false)

    Outputs:

    
// Outputs:
root
 |-- name: string (nullable = true)
 |-- col: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

+-------+-----------------------------+
|name   |col                          |
+-------+-----------------------------+
|James  |[hair -> black, eye -> brown]|
|James  |[height -> 5.9]              |
|Michael|[hair -> brown, eye -> black]|
|Michael|[height -> 6]                |
|Robert |[hair -> red, eye -> gray]   |
|Robert |[height -> 6.3]              |
+-------+-----------------------------+

    <a id="h-complete-example"></a>

    Complete Example

    
// Complete Example
package com.aistudylabs.spark.dataframe.functions.collection

import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.functions.{explode}
import org.apache.spark.sql.types._

object ArrayOfMapType extends App {
  val spark = SparkSession.builder().appName("AIStudyLabs.com")
    .master("local[1]")
    .getOrCreate()

  val arrayMapSchema = new StructType().add("name",StringType)
    .add("properties",
      ArrayType(new MapType(StringType,StringType,true)))

  val arrayMapData = Seq(
    Row("James",List(Map("hair"->"black","eye"->"brown"), Map("height"->"5.9"))),
    Row("Michael",List(Map("hair"->"brown","eye"->"black"),Map("height"->"6"))),
    Row("Robert",List(Map("hair"->"red","eye"->"gray"),Map("height"->"6.3")))
  )

  val df = spark.createDataFrame(
    spark.sparkContext.parallelize(arrayMapData),arrayMapSchema)
  df.printSchema()
  df.show(false)

  import spark.implicits._

  val df2 = df.select($"name",explode($"properties"))
  df2.printSchema()
  df2.show(false)
}

    Happy Learning !!

    <a id="h-related-articles"></a>

    Related Articles