Built with Tutify.io

    AIStudyLabs

    Spark Read JSON from multiline

    Spark Read JSON from multiline

    Spark JSON data source API provides the multiline option to read records from multiple lines. By default, spark considers every record in a JSON file as a fully qualified record in a single line hence, we need to use the multiline option to process JSON from multiple lines.

    <a id="h-using-multiline-option-read-json-multiple-lines"></a>

    Using multiline Option - Read JSON multiple lines

    In this example, we set multiline option to true to read JSON records from multiple lines into Spark DataFrame. By default, this option is set to false. Let's consider we have a below JSON file by name "multiline-zipcode.json". this file is also available at GitHub for reference

    
[{
  "RecordNumber": 2,
  "Zipcode": 704,
  "ZipCodeType": "STANDARD",
  "City": "PASEO COSTA DEL SUR",
  "State": "PR"
},
{
  "RecordNumber": 10,
  "Zipcode": 709,
  "ZipCodeType": "STANDARD",
  "City": "BDA SAN LUIS",
  "State": "PR"
}]

    Use spark.read.option("multiline", "true") while reading

    
    //read multiline json file
    val multiline_df = spark.read.option("multiline", "true")
      .json("src/main/resources/multiline-zipcode.json")
    multiline_df.printSchema()
    multiline_df.show(false)

    multiline_df.printSchema() yields below schema

    
root
 |-- City: string (nullable = true)
 |-- RecordNumber: long (nullable = true)
 |-- State: string (nullable = true)
 |-- ZipCodeType: string (nullable = true)
 |-- Zipcode: long (nullable = true)

    multiline_df.show(false) statement yields below output

    
+-------------------+------------+-----+-----------+-------+
|City               |RecordNumber|State|ZipCodeType|Zipcode|
+-------------------+------------+-----+-----------+-------+
|PASEO COSTA DEL SUR|2           |PR   |STANDARD   |704    |
|BDA SAN LUIS       |10          |PR   |STANDARD   |709    |
+-------------------+------------+-----+-----------+-------+

    <a id="h-complete-example-of-reading-json-using-multiline-option"></a>

    Complete example of reading JSON using multiline option

    
    val spark: SparkSession = SparkSession.builder()
      .master("local[3]")
      .appName("AIStudyLabs.com")
      .getOrCreate()

    This complete example is available at GitHub.

    <a id="h-conclusion"></a>

    Conclusion

    In this article, you have learned how to read a JSON from multiline and convert it into Spark DataFrame using a Scala example.

    Happy Learning !!

    <a id="h-related-articles"></a>

    Related Articles