Introduction

The union() method of the DataFrame in Spark is used to combine two DataFrames with the same structure/schema. If schemas are not the same, it returns an error.

unionAll() is deprecated since Spark version “2.0.0” and replaced with union().

Note: In traditional SQL, Union eliminates duplicates, but UnionAll combines two datasets, including duplicate records. However, in Spark, both behave the same.

Before using these methods, it's essential to understand StructType and StructField, which define the schema of DataFrames.

• StructType is a collection of StructFields that define the column name, column type, nullable property, and metadata.
• StructField is essentially a field containing a column's metadata, including column name and data type.

First, let’s create two DataFrames with the same schema.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;

import java.util.Arrays;

SparkSession spark = SparkSession.builder()
        .appName("Java Spark Union Example")
        .master("local")
        .getOrCreate();

StructType schema = new StructType(new StructField[]{
        DataTypes.createStructField("employee_name", DataTypes.StringType, true),
        DataTypes.createStructField("department", DataTypes.StringType, true),
        DataTypes.createStructField("state", DataTypes.StringType, true),
        DataTypes.createStructField("salary", DataTypes.IntegerType, false),
        DataTypes.createStructField("age", DataTypes.IntegerType, false),
        DataTypes.createStructField("bonus", DataTypes.IntegerType, false)
});

Dataset<Row> df = spark.createDataFrame(Arrays.asList(
        RowFactory.create("James", "Sales", "NY", 90000, 34, 10000),
        RowFactory.create("Michael", "Sales", "NY", 86000, 56, 20000),
        RowFactory.create("Robert", "Sales", "CA", 81000, 30, 23000),
        RowFactory.create("Maria", "Finance", "CA", 90000, 24, 23000)
), schema);

df.printSchema();
df.show();

root
 |-- employee_name: string (nullable = true)
 |-- department: string (nullable = true)
 |-- state: string (nullable = true)
 |-- salary: integer (nullable = false)
 |-- age: integer (nullable = false)
 |-- bonus: integer (nullable = false)

+-------------+----------+-----+------+---+-----+
|employee_name|department|state|salary|age|bonus|
+-------------+----------+-----+------+---+-----+
|        James|     Sales|   NY| 90000| 34|10000|
|      Michael|     Sales|   NY| 86000| 56|20000|
|       Robert|     Sales|   CA| 81000| 30|23000|
|        Maria|   Finance|   CA| 90000| 24|23000|
+-------------+----------+-----+------+---+-----+

Now, let’s create a second DataFrame with new records and some records from the above DataFrame but with the same schema.

Dataset<Row> df2 = spark.createDataFrame(Arrays.asList(
        RowFactory.create("James", "Sales", "NY", 90000, 34, 10000),
        RowFactory.create("Maria", "Finance", "CA", 90000, 24, 23000),
        RowFactory.create("Jen", "Finance", "NY", 79000, 53, 15000),
        RowFactory.create("Jeff", "Marketing", "CA", 80000, 25, 18000),
        RowFactory.create("Kumar", "Marketing", "NY", 91000, 50, 21000)
), schema);

df2.show();

This yields below output:

+-------------+----------+-----+------+---+-----+
|employee_name|department|state|salary|age|bonus|
+-------------+----------+-----+------+---+-----+
|        James|     Sales|   NY| 90000| 34|10000|
|        Maria|   Finance|   CA| 90000| 24|23000|
|          Jen|   Finance|   NY| 79000| 53|15000|
|         Jeff|Marketing |   CA| 80000| 25|18000|
|        Kumar|Marketing |   NY| 91000| 50|21000|
+-------------+----------+-----+------+---+-----+

1. Combine two or more DataFrames using union

DataFrame union() method combines two DataFrames and returns a new DataFrame with all rows from two DataFrames regardless of duplicate data.

// Combine two or more DataFrames using union
Dataset<Row> df3 = df.union(df2);
df3.show(false);

As you see below, it returns all records:

+-------------+----------+-----+------+---+-----+
|employee_name|department|state|salary|age|bonus|
+-------------+----------+-----+------+---+-----+
|        James|     Sales|   NY| 90000| 34|10000|
|      Michael|     Sales|   NY| 86000| 56|20000|
|       Robert|     Sales|   CA| 81000| 30|23000|
|        Maria|   Finance|   CA| 90000| 24|23000|
|        James|     Sales|   NY| 90000| 34|10000|
|        Maria|   Finance|   CA| 90000| 24|23000|
|          Jen|   Finance|   NY| 79000| 53|15000|
|         Jeff|Marketing |   CA| 80000| 25|18000|
|        Kumar|Marketing |   NY| 91000| 50|21000|
+-------------+----------+-----+------+---+-----+

2. Combine DataFrames using unionAll

The DataFrame unionAll() method is deprecated since Spark version “2.0.0” and recommends using the union() method.

// Combine DataFrames using unionAll (deprecated, use union())
Dataset<Row> df4 = df.union(df2);
df4.show(false);

Returns the same output as above.

3. Combine without Duplicates

Since the union() method returns all rows without distinct records, we will use the distinct() function to return just one record when duplicates exist.

// Combine without Duplicates
Dataset<Row> df5 = df.union(df2).distinct();
df5.show(false);

Yields the output below. As you see, this returns only distinct rows:

+-------------+----------+-----+------+---+-----+
|employee_name|department|state|salary|age|bonus|
+-------------+----------+-----+------+---+-----+
|        James|     Sales|   NY| 90000| 34|10000|
|        Maria|   Finance|   CA| 90000| 24|23000|
|         Jeff|Marketing |   CA| 80000| 25|18000|
|          Jen|   Finance|   NY| 79000| 53|15000|
|        Kumar|Marketing |   NY| 91000| 50|21000|
|      Michael|     Sales|   NY| 86000| 56|20000|
|       Robert|     Sales|   CA| 81000| 30|23000|
+-------------+----------+-----+------+---+-----+

This complete example is also available at the GitHub project.