Introduction

In Spark, withColumnRenamed() is used to rename one column or multiple DataFrame column names. Depending on the DataFrame schema, renaming columns might get simple to complex, especially when a column is nested with struct type it gets complicated.

First, let’s create our data and DataFrame. We are using the Row class to convert this data to a Spark DataFrame.

import org.apache.spark.sql.Row;
import java.util.Arrays;
import java.util.List;

List<Row> data = Arrays.asList(
  RowFactory.create(RowFactory.create("James", "", "Smith"), "36636", "M", 3000),
  RowFactory.create(RowFactory.create("Michael", "Rose", ""), "40288", "M", 4000),
  RowFactory.create(RowFactory.create("Robert", "", "Williams"), "42114", "M", 4000),
  RowFactory.create(RowFactory.create("Maria", "Anne", "Jones"), "39192", "F", 4000),
  RowFactory.create(RowFactory.create("Jen", "Mary", "Brown"), "", "F", -1)
);

Our base schema with a nested structure.

import org.apache.spark.sql.types.*;

StructType schema = new StructType(new StructField[] {
  new StructField("name", new StructType(new StructField[] {
    new StructField("firstname", DataTypes.StringType, true, Metadata.empty()),
    new StructField("middlename", DataTypes.StringType, true, Metadata.empty()),
    new StructField("lastname", DataTypes.StringType, true, Metadata.empty())
  }), true, Metadata.empty()),
  new StructField("dob", DataTypes.StringType, true, Metadata.empty()),
  new StructField("gender", DataTypes.StringType, true, Metadata.empty()),
  new StructField("salary", DataTypes.IntegerType, true, Metadata.empty())
});

Let’s create the DataFrame by using parallelize and provide the above schema.

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

Dataset<Row> df = spark.createDataFrame(spark.sparkContext().parallelize(data), schema);
df.printSchema();

Output:

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- dob: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary: integer (nullable = true)

1. Using withColumnRenamed() to Rename Column Name

This is the most straightforward approach; this function takes two parameters: the first is your existing column name, and the second is the new column name you wish for.

Syntax:

DataFrame withColumnRenamed(String existingName, String newName)

existingName – The existing column name you want to change

newName – New name of the column

Returns a new DataFrame (Dataset<Row>) with a column renamed. This is a no-op if the schema doesn’t contain existingName.

Example:

Dataset<Row> dfRenamed = df.withColumnRenamed("dob", "DateOfBirth");
dfRenamed.printSchema();

Output:

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- DateOfBirth: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary: integer (nullable = true)

The above example changes the column name from “dob” to “DateOfBirth” on the Spark DataFrame. Note that withColumnRenamed function returns a new DataFrame and doesn’t modify the current DataFrame.

2. Rename Multiple Columns

To change multiple column names, we should chain withColumnRenamed functions as shown below.

Dataset<Row> df2 = df.withColumnRenamed("dob", "DateOfBirth")
                     .withColumnRenamed("salary", "salary_amount");
df2.printSchema();

Output:

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- DateOfBirth: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary_amount: integer (nullable = true)

This creates a new DataFrame df2 after renaming dob and salary columns.

3. Using Spark StructType – To Rename a Nested Column in Dataframe

Changing a column name on nested data is not straightforward forward, and we can do this by creating a new schema with new DataFrame columns using StructType and using it with the cast function as shown below.

StructType schema2 = new StructType(new StructField[] {
  new StructField("fname", DataTypes.StringType, true, Metadata.empty()),
  new StructField("middlename", DataTypes.StringType, true, Metadata.empty()),
  new StructField("lname", DataTypes.StringType, true, Metadata.empty())
});

import org.apache.spark.sql.functions;

df.select(functions.col("name").cast(schema2), functions.col("dob"), 
          functions.col("gender"), functions.col("salary"))
  .printSchema();

Output:

root
 |-- fname: string (nullable = true)
 |-- middlename: string (nullable = true)
 |-- lname: string (nullable = true)
 |-- dob: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary: integer (nullable = true)

This statement renames firstname to fname and lastname to lname within the name structure.

4. Using select() to Rename Nested Elements

Let’s see another way to change nested columns by transposing the structure to flat.

df.select(
  functions.col("name.firstname").as("fname"),
  functions.col("name.middlename").as("mname"),
  functions.col("name.lastname").as("lname"),
  functions.col("dob"), functions.col("gender"), functions.col("salary"))
  .printSchema();

Output:

root
 |-- fname: string (nullable = true)
 |-- mname: string (nullable = true)
 |-- lname: string (nullable = true)
 |-- dob: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary: integer (nullable = true)

5. Using withColumn() to Rename Nested Columns

When you have nested columns on a Spark DataFrame, and if you want to rename it, use withColumn on a DataFrame object to create a new column from an existing one, and we will need to drop the existing column. Below example creates a “fname” column from “name.firstname” and drops the “name” column.

Dataset<Row> df4 = df.withColumn("fname", functions.col("name.firstname"))
                     .withColumn("mname", functions.col("name.middlename"))
                     .withColumn("lname", functions.col("name.lastname"))
                     .drop("name");
df4.printSchema();

Output:

root
 |-- dob: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- salary: integer (nullable = true)
 |-- fname: string (nullable = true)
 |-- mname: string (nullable = true)
 |-- lname: string (nullable = true)

6. Using col() Function to Dynamically Rename All or Multiple Columns

Another way to change all column names on a DataFrame is to use the col() function; you can also use map() to rename multiple columns.

String[] oldColumns = new String[]{"dob", "gender", "salary", "fname", "mname", "lname"};
String[] newColumns = new String[]{"DateOfBirth", "Sex", "salary", "firstName", "middleName", "lastName"};

Seq<Column> columnsList = java.util.stream.IntStream.range(0, oldColumns.length)
  .mapToObj(i -> functions.col(oldColumns[i]).as(newColumns[i]))
  .collect(Collectors.toList());

Dataset<Row> df5 = df4.select(columnsList.toArray(new Column[0]));
df5.printSchema();

Output:

root
 |-- DateOfBirth: string (nullable = true)
 |-- Sex: string (nullable = true)
 |-- salary: integer (nullable = true)
 |-- firstName: string (nullable = true)
 |-- middleName: string (nullable = true)
 |-- lastName: string (nullable = true)

7. Using toDF() – To Change All Columns in a Spark DataFrame

When we have data in a flat structure (without nested), use toDF() with a new schema to change all column names.

Dataset<Row> renamedDf = df.toDF("newCol1", "newCol2", "newCol3");
renamedDf.printSchema();

Output:

root
 |-- newCol1: string (nullable = true)
 |-- newCol2: string (nullable = true)
 |-- newCol3: string (nullable = true)