Introduction

Spark DataFrame provides a drop() method to drop a column/field from a DataFrame/Dataset. The drop() method is also used to remove multiple columns at a time from a Spark DataFrame/Dataset.

First, let’s create a DataFrame.

import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.types.{IntegerType, StringType, StructType}

val structureData = Seq(
  Row("James", "", "Smith", "36636", "NewYork", 3100),
  Row("Michael", "Rose", "", "40288", "California", 4300),
  Row("Robert", "", "Williams", "42114", "Florida", 1400),
  Row("Maria", "Anne", "Jones", "39192", "Florida", 5500),
  Row("Jen", "Mary", "Brown", "34561", "NewYork", 3000)
)

val structureSchema = new StructType()
  .add("firstname", StringType)
  .add("middlename", StringType)
  .add("lastname", StringType)
  .add("id", StringType)
  .add("location", StringType)
  .add("salary", IntegerType)

val df = spark.createDataFrame(
  spark.sparkContext.parallelize(structureData), structureSchema
)

df.printSchema()

This yields the output:

root
 |-- firstname: string (nullable = true)
 |-- middlename: string (nullable = true)
 |-- lastname: string (nullable = true)
 |-- id: string (nullable = true)
 |-- location: string (nullable = true)
 |-- salary: integer (nullable = true)

1. drop() syntax

Spark drop() has 3 different signatures. I’ve explained using all these signatures with examples in the below sections.

// Spark DataFrame drop() syntax
1) drop(colName: scala.Predef.String): org.apache.spark.sql.DataFrame
2) drop(colNames: scala.Predef.String*): org.apache.spark.sql.DataFrame
3) drop(col: org.apache.spark.sql.Column): org.apache.spark.sql.DataFrame

2. Drop one column

The first and third signature takes the column name as String type and Column type, respectively. When you use the third signature make sure you import org.apache.spark.sql.functions.col

// Drop one column from DataFrame
val df2 = df.drop("firstname") // First signature
df2.printSchema()

df.drop(df("firstname")).printSchema()

import org.apache.spark.sql.functions.col
df.drop(col("firstname")).printSchema() // Third signature

The above 3 examples remove the column “firstname” from the DataFrame. You can use either one of these, depending on your needs.

root
 |-- middlename: string (nullable = true)
 |-- lastname: string (nullable = true)
 |-- id: string (nullable = true)
 |-- location: string (nullable = true)
 |-- salary: integer (nullable = true)

3. Drop Multiple Columns from DataFrame

This uses the second signature of the drop(), which removes more than one column from a DataFrame.

// Refering more than one column
df.drop("firstname", "middlename", "lastname")
  .printSchema()

// Using array/sequence of columns
val cols = Seq("firstname", "middlename", "lastname")
df.drop(cols: _*)
  .printSchema()

The above two examples remove more than one column at a time from DataFrame. These both yield the same output.

root
 |-- id: string (nullable = true)
 |-- location: string (nullable = true)
 |-- salary: integer (nullable = true)

The complete example is also available at Spark Examples Github project for references.