Introduction

PySpark DataFrame provides a drop() method to drop a column/field from a DataFrame. The drop() method is also used to remove multiple columns at a time from a PySpark DataFrame.

First, let’s create a DataFrame.

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession.builder.appName("DropColumnExample").getOrCreate()

structure_data = [
    ("James", "", "Smith", "36636", "NewYork", 3100),
    ("Michael", "Rose", "", "40288", "California", 4300),
    ("Robert", "", "Williams", "42114", "Florida", 1400),
    ("Maria", "Anne", "Jones", "39192", "Florida", 5500),
    ("Jen", "Mary", "Brown", "34561", "NewYork", 3000)
]

structure_schema = StructType([
    StructField("firstname", StringType(), True),
    StructField("middlename", StringType(), True),
    StructField("lastname", StringType(), True),
    StructField("id", StringType(), True),
    StructField("location", StringType(), True),
    StructField("salary", IntegerType(), True)
])

df = spark.createDataFrame(data=structure_data, schema=structure_schema)

df.printSchema()

This yields the output:

root
 |-- firstname: string (nullable = true)
 |-- middlename: string (nullable = true)
 |-- lastname: string (nullable = true)
 |-- id: string (nullable = true)
 |-- location: string (nullable = true)
 |-- salary: integer (nullable = true)

1. drop() syntax

PySpark drop() can be used in several ways. We will explain these signatures with examples in the below sections.

# PySpark DataFrame drop() usage
df.drop("columnName")  # Dropping a single column by name
df.drop("columnName1", "columnName2")  # Dropping multiple columns by names

2. Drop one column

You can use the column name in string format to drop a single column.

# Drop one column from DataFrame
df2 = df.drop("firstname") 
df2.printSchema()

The above example removes the column “firstname” from the DataFrame.

root
 |-- middlename: string (nullable = true)
 |-- lastname: string (nullable = true)
 |-- id: string (nullable = true)
 |-- location: string (nullable = true)
 |-- salary: integer (nullable = true)

3. Drop Multiple Columns from DataFrame

This removes more than one column from a DataFrame.

# Dropping more than one column
df3 = df.drop("firstname", "middlename", "lastname")
df3.printSchema()

# Using a list of columns
cols = ["firstname", "middlename", "lastname"]
df4 = df.drop(*cols)
df4.printSchema()

The above examples remove more than one column at a time from DataFrame. These yield the same output.

root
 |-- id: string (nullable = true)
 |-- location: string (nullable = true)
 |-- salary: integer (nullable = true)