Spark Convert CSV to Avro, Parquet & JSON

In this Spark article, you will learn how to read a CSV file into DataFrame and convert or save DataFrame to Avro, Parquet and JSON file formats using Scala examples.

Though the below examples explain with the CSV in context, once we have data in DataFrame, we can convert it to any format Spark supports regardless of how and from where you have read it.

Read CSV into DataFrame
Convert CSV to Avro
Convert CSV to Parquet
Convert CSV to JSON
Complete Example

1. Read CSV into DataFrame

Using spark.read.csv("path") or spark.read.format("csv").load("path") you can read a CSV file into a Spark DataFrame, Thes method takes a file path to read as an argument. By default read method considers header as a data record hence it reads column names on file as data, To overcome this we need to explicitly mention “true” for header option. It also reads all columns as a string (StringType) by default.


  // Read csv with options
  val df = spark.read.options(Map("inferSchema"->"true","delimiter"->",","header"->"true"))
    .csv("src/main/resources/zipcodes.csv")
  df.show()
  df.printSchema()

In this example, we are using the option inferSchema to true, with this option, Spark looks at the data and identifies the column type. This snippet prints the schema and sample data to the console.

Spark also supports many other options while reading a CSV file.

2. Spark Convert CSV to Avro file

Once we convert the CSV into Spark DataFrame, we can write the DataFrame to AVRO file format. first, let's see what is Avro file format and then will see some examples in Scala.

Apache Avro is an open-source, row-based, data serialization and data exchange framework for Hadoop projects, originally developed by databricks as an open-source library that supports reading and writing data in Avro file format. it is mostly used in Apache Spark especially for Kafka-based data pipelines. When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program.

Since Avro library is external to Spark, it doesn’t provide avro() function on DataFrameWriter , hence we should use DataSource “avro” or “org.apache.spark.sql.avro” to write Spark DataFrame to Avro file.


df.write.format("avro").save("/tmp/avro/zipcodes.avro")

Spark DataFrameWriter provides partitionBy() function to partition the Avro at the time of writing. Partition improves performance on reading by reducing Disk I/O.


df.write.partitionBy("State","Zipcode")
        .format("avro").save("/tmp/avro/zipcodes_partition.avro")

If you want to read more on Avro, I would recommend checking how to Read and Write Avro file with a specific schema along with the dependencies it needed.

3. Spark Convert CSV to Parquet file

Let's see how to convert the Spark DataFrame that created from CSV to the Parquet file, first let's see what is Parquet file format is and then will see some examples in Scala.

Apache Parquet is a columnar file format that provides optimizations to speed up queries and is a far more efficient file format than CSV or JSON, supported by many data processing systems.

Using spark.write.parquet() function we can write Spark DataFrame to Parquet file, and parquet() function is provided in DataFrameWriter class. As mentioned earlier Spark doesn’t need any additional packages or libraries to use Parquet as it by default provides with Spark. easy isn’t it? as we don’t have to worry about version and compatibility issues. In this example, we are writing DataFrame to “people.parquet” file


df.write.parquet("/tmp/parquet/zipcodes.parquet")

If you want to read more on Parquet, I would recommend checking how to Read and Write Parquet file with a specific schema along with the dependencies and how to use partitions.

4. Spark Convert CSV to JSON file

Similar to Avro and Parquet, once we have a DataFrame created from CSV file, we can easily convert or save it to JSON file using dataframe.write.json("path")


// Spark Convert CSV to JSON file
df.write
.json("/tmp/json/zipcodes.json")

In this example, we have used the head option to write the CSV file with the header, Spark also supports multiple options to read and write CSV files

5. Complete Example of converting CSV to Avro, Parquet and JSON file


package com.aistudylabs.spark.dataframe

import org.apache.spark.sql.{SaveMode, SparkSession}

object CsvToAvroParquetJson extends App {

  val spark: SparkSession = SparkSession.builder()
    .master("local[1]")
    .appName("SparkByExample")
    .getOrCreate()

  spark.sparkContext.setLogLevel("ERROR")

  //read csv with options
  val df = spark.read.options(Map("inferSchema"->"true","delimiter"->",","header"->"true"))
    .csv("src/main/resources/zipcodes.csv")
  df.show()
  df.printSchema()

  //convert to avro
  df.write.format("avro").mode(SaveMode.Overwrite)
    .save("/tmp/avro/zipcodes.avro")

  //convert to avro by partition
  df.write.partitionBy("State","Zipcode")
    .format("avro")
    .mode(SaveMode.Overwrite)
    .save("/tmp/avro/zipcodes_partition.avro")

  //convert to parquet
  df.write.mode(SaveMode.Overwrite).parquet("/tmp/parquet/zipcodes.parquet")

  //convert to csv
  df.write.mode(SaveMode.Overwrite).json("/tmp/json/zipcodes.json")
}

Conclusion

In this Spark article, you have learned how to convert a CSV file to Avro, Parquet and JSON file with Scala examples. Though we literally don't convert from one format to another straight, first we convert it to DataFrame and then DataFrame can be converted to any format Spark supports.

Happy Learning !!

Spark Convert CSV to Avro, Parquet &amp; JSON