Introduction

Some of the examples you have seen so far used StructType. Let's dive deep into StructType and StructFields and their usage.

Spark SQL StructType & StructField classes are used to programmatically specify the schema to the DataFrame and create complex columns like nested struct, array, and map columns. StructType is a collection of StructFields. Using StructField, we can define column name, column data type, a nullable column (a boolean to specify if the field can be nullable or not), and metadata.

Though Spark infers a schema from data, sometimes we may need to define our own column names and data types. 

StructType – Defines the structure of the DataFrame

Spark provides spark.sql.types.StructType to define the structure of the DataFrame and it is a collection or list of StructField objects. By calling Spark DataFrame printSchema() we can print the schema on the console where StructType columns are represented as struct.

StructField – Defines the metadata of the DataFrame column

Spark provides spark.sql.types.StructField class to define the column name (String), column type (DataType), nullable column (Boolean), and metadata (MetaData).

Using StructType & StructField with DataFrame

While creating a Spark DataFrame we can specify the structure using StructType and StructField classes. As specified in the introduction, StructType is a collection of StructFields which are used to define the column name, data type, and a flag for nullable or not. Using StructField we can also add a nested struct schema, ArrayType for arrays, and MapType for key-value pairs which we will discuss in detail in later sections.

The StructType & StructField in Java Spark are instantiated as shown below:

StructType simpleSchema = new StructType(new StructField[]{
    new StructField("firstname", DataTypes.StringType, true, Metadata.empty()),
    new StructField("middlename", DataTypes.StringType, true, Metadata.empty()),
    new StructField("lastname", DataTypes.StringType, true, Metadata.empty()),
    new StructField("id", DataTypes.StringType, true, Metadata.empty()),
    new StructField("gender", DataTypes.StringType, true, Metadata.empty()),
    new StructField("salary", DataTypes.IntegerType, true, Metadata.empty())
});

Below is an example of how to create a Spark DataFrame using StructType & StructField on DataFrame with sample data.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.Metadata;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
import java.util.Arrays;
import java.util.List;

public class DataFrameCreator {

    public static void main(String[] args) {
        SparkSession spark = SparkSession.builder()
                .appName("DataFrame Example")
                .master("local")
                .getOrCreate();

        List<Row> data = Arrays.asList(
                RowFactory.create("James", "", "Smith", "36636", "M", 3000),
                RowFactory.create("Michael", "Rose", "", "40288", "M", 4000),
                RowFactory.create("Robert", "", "Williams", "42114", "M", 4000),
                RowFactory.create("Maria", "Anne", "Jones", "39192", "F", 4000),
                RowFactory.create("Jen", "Mary", "Brown", "", "F", -1)
        );

        StructType schema = new StructType(new StructField[]{
                new StructField("firstname", DataTypes.StringType, true, Metadata.empty()),
                new StructField("middlename", DataTypes.StringType, true, Metadata.empty()),
                new StructField("lastname", DataTypes.StringType, true, Metadata.empty()),
                new StructField("id", DataTypes.StringType, true, Metadata.empty()),
                new StructField("gender", DataTypes.StringType, true, Metadata.empty()),
                new StructField("salary", DataTypes.IntegerType, true, Metadata.empty())
        });

        Dataset<Row> df = spark.createDataFrame(data, schema);

        df.printSchema();
        df.show();
    }
}

By running the above snippet, it displays the following outputs.

root
 |-- firstname: string (nullable = true)
 |-- middlename: string (nullable = true)
 |-- lastname: string (nullable = true)
 |-- id: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary: integer (nullable = true)

+---------+----------+--------+-----+------+------+
|firstname|middlename|lastname|   id|gender|salary|
+---------+----------+--------+-----+------+------+
|    James|          |   Smith|36636|     M|  3000|
|  Michael|      Rose|        |40288|     M|  4000|
|   Robert|          |Williams|42114|     M|  4000|
|    Maria|      Anne|   Jones|39192|     F|  4000|
|      Jen|      Mary|   Brown|     |     F|    -1|
+---------+----------+--------+-----+------+------+

Defining Nested StructType object struct

We often need to work with the nested struct column and this can be defined using SQL StructType.

In the below example, we instantiate a StructType and use the add method (instead of StructField) to add column names and datatype. Notice that for column “name” data type is StructType which is nested.

List<Row> structureData = Arrays.asList(
        RowFactory.create(RowFactory.create("James", "", "Smith"), "36636", "M", 3100),
        RowFactory.create(RowFactory.create("Michael", "Rose", ""), "40288", "M", 4300),
        RowFactory.create(RowFactory.create("Robert", "", "Williams"), "42114", "M", 1400),
        RowFactory.create(RowFactory.create("Maria", "Anne", "Jones"), "39192", "F", 5500),
        RowFactory.create(RowFactory.create("Jen", "Mary", "Brown"), "", "F", -1)
);

StructType nestedSchema = new StructType()
        .add("name", new StructType()
                .add("firstname", DataTypes.StringType)
                .add("middlename", DataTypes.StringType)
                .add("lastname", DataTypes.StringType))
        .add("id", DataTypes.StringType)
        .add("gender", DataTypes.StringType)
        .add("salary", DataTypes.IntegerType);

Dataset<Row> df2 = spark.createDataFrame(structureData, nestedSchema);

df2.printSchema();
df2.show();

Outputs below show the schema and the DataFrame.

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- id: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary: integer (nullable = true)

+--------------------+-----+------+------+ 
|                name|   id|gender|salary| 
+--------------------+-----+------+------+ 
|   [James , , Smith]|36636|     M|  3100| 
|  [Michael , Rose, ]|40288|     M|  4300| 
|[Robert , , Willi...|42114|     M|  1400| 
|[Maria , Anne, Jo...|39192|     F|  5500| 
|  [Jen, Mary, Brown]|     |     F|    -1| 
+--------------------+-----+------+------+ 

Creating StructType object struct from JSON file

If you have too many columns and the structure of the DataFrame changes now and then, it’s a good practice to load the SQL StructType schema from a JSON file. Note the definition in JSON uses a different layout and you can get this by using schema.prettyJson().

{
  "type" : "struct",
  "fields" : [ 
    {
    "name" : "name",
    "type" : {
      "type" : "struct",
      "fields" : [ 
        {
        "name" : "firstname",
        "type" : "string",
        "nullable" : true,
        "metadata" : { }
      }, 
      {
        "name" : "middlename",
        "type" : "string",
        "nullable" : true,
        "metadata" : { }
      }, 
      {
        "name" : "lastname",
        "type" : "string",
        "nullable" : true,
        "metadata" : { }
      } ]
    },
    "nullable" : true,
    "metadata" : { }
  }, 
  {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, 
  {
    "name" : "gender",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, 
  {
    "name" : "salary",
    "type" : "integer",
    "nullable" : true,
    "metadata" : { }
  } 
  ]
}

import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.StructType;

public class JsonSchemaLoader {

    public static void main(String[] args) {
        SparkSession spark = SparkSession.builder()
                .appName("JSON Schema Loader")
                .master("local")
                .getOrCreate();

        String schemaSource = new String(Files.readAllBytes(Paths.get("path/to/schema.json")));
        StructType schemaFromJson = (StructType) DataType.fromJson(schemaSource);

        Dataset<Row> df3 = spark.createDataFrame(structureData, schemaFromJson);

        df3.printSchema();
    }
}

This prints the same output as the previous section. You can also have a name, type, and flag for nullable in a comma-separated file and we can use these to create a StructType programmatically, which is left for you to explore.

Adding & Changing struct of the DataFrame

Using Spark SQL function struct(), we can change the struct of the existing DataFrame and add a new StructType to it. The below example demonstrates how to copy the columns from one structure to another and add a new column.

import static org.apache.spark.sql.functions.*;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

Dataset<Row> updatedDF = df2.withColumn("OtherInfo",
        struct(
                col("id").as("identifier"),
                col("gender").as("gender"),
                col("salary").as("salary"),
                when(col("salary").cast(DataTypes.IntegerType).lt(2000), "Low")
                        .when(col("salary").cast(DataTypes.IntegerType).lt(4000), "Medium")
                        .otherwise("High").alias("Salary_Grade")
        )
).drop("id", "gender", "salary");

updatedDF.printSchema();
updatedDF.show(false);

Here, it copies “gender“, “salary” and “id” to the new struct “OtherInfo” and adds a new column “Salary_Grade”.

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- OtherInfo: struct (nullable = false)
 |    |-- identifier: string (nullable = true)
 |    |-- gender: string (nullable = true)
 |    |-- salary: integer (nullable = true)
 |    |-- Salary_Grade: string (nullable = false)

Using SQL ArrayType and MapType

SQL StructType also supports ArrayType and MapType to define the DataFrame columns for array and map collections respectively. In the below example, column “hobbies” is defined as ArrayType(StringType) and “properties” is defined as MapType(StringType,StringType) meaning both key and value as String.

import org.apache.spark.sql.types.ArrayType;
import org.apache.spark.sql.types.MapType;

List<Row> arrayStructureData = Arrays.asList(
        RowFactory.create(RowFactory.create("James", "", "Smith"), Arrays.asList("Cricket", "Movies"), Map.of("hair", "black", "eye", "brown")),
        RowFactory.create(RowFactory.create("Michael", "Rose", ""), Arrays.asList("Tennis"), Map.of("hair", "brown", "eye", "black")),
        RowFactory.create(RowFactory.create("Robert", "", "Williams"), Arrays.asList("Cooking", "Football"), Map.of("hair", "red", "eye", "gray")),
        RowFactory.create(RowFactory.create("Maria", "Anne", "Jones"), null, Map.of("hair", "blond", "eye", "red")),
        RowFactory.create(RowFactory.create("Jen", "Mary", "Brown"), Arrays.asList("Blogging"), Map.of("white", "black", "eye", "black"))
);

StructType arrayStructureSchema = new StructType()
        .add("name", new StructType()
                .add("firstname", DataTypes.StringType)
                .add("middlename", DataTypes.StringType)
                .add("lastname", DataTypes.StringType))
        .add("hobbies", DataTypes.createArrayType(DataTypes.StringType))
        .add("properties", DataTypes.createMapType(DataTypes.StringType, DataTypes.StringType));

Dataset<Row> df5 = spark.createDataFrame(arrayStructureData, arrayStructureSchema);

df5.printSchema();
df5.show();

Outputs below the schema and the DataFrame data. Note that field “hobbies” is array type and “properties” is map type.

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- hobbies: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- properties: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

+---------------------+-------------------+------------------------------+
|name                 |hobbies            |properties                    |
+---------------------+-------------------+------------------------------+
|[James , , Smith]    |[Cricket, Movies]  |[hair -> black, eye -> brown] |
|[Michael , Rose, ]   |[Tennis]           |[hair -> brown, eye -> black] |
|[Robert , , Williams]|[Cooking, Football]|[hair -> red, eye -> gray]    |
|[Maria , Anne, Jones]|null               |[hair -> blond, eye -> red]   |
|[Jen, Mary, Brown]   |[Blogging]         |[white -> black, eye -> black]|
+---------------------+-------------------+------------------------------+

Convert case class to Spark StructType

Spark SQL also provides Encoders to convert Java Beans to StructType objects. If you are using older versions of Spark, encourage users to transform the Java class to the schema using reflection.

import org.apache.spark.sql.Encoder;
import org.apache.spark.sql.Encoders;

public class Employee implements Serializable {
    private Name fullName;
    private Integer age;
    private String gender;

    // getters and setters
}

public class Name implements Serializable {
    private String first;
    private String last;
    private String middle;

    // getters and setters
}

Encoder<Employee> employeeEncoder = Encoders.bean(Employee.class);
StructType beanSchema = employeeEncoder.schema();

System.out.println(beanSchema.treeString());

treeString() outputs the schema.

root
 |-- fullName: struct (nullable = true)
 |    |-- first: string (nullable = true)
 |    |-- last: string (nullable = true)
 |    |-- middle: string (nullable = true)
 |-- age: integer (nullable = true)
 |-- gender: string (nullable = true)

Creating StructType object struct from DDL String

Like loading a structure from a JSON string, we can also create it from a DDL (by using fromDDL() static function on SQL StructType class StructType.fromDDL). You can also generate DDL from a schema using toDDL(). This allows for easy schema generation and validation.

StructType ddlSchema = StructType.fromDDL("`fullName` STRUCT<`first`: STRING, `last`: STRING, `middle`: STRING>,`age` INT,`gender` STRING");

System.out.println(ddlSchema.treeString());

Checking if a field exists in a DataFrame

If you want to perform some checks on metadata of the DataFrame, for example, if a column or field exists in a DataFrame or the data type of the column; we can easily do this using several functions on SQL StructType and StructField.

System.out.println(df.schema().fieldNames().contains("firstname"));
System.out.println(df.schema().getFieldIndex("firstname").isDefined());

This example returns “true” for both scenarios if the "firstname" field exists. Similarly, you can also check if two schemas are equal and more.

The complete example explained here is available at GitHub project.