Select DataFrame Columns

PySpark SQL select() and selectExpr() are used to select the columns from the DataFrame.

Both these are transformation operations and return a new DataFrame based on the usage of untyped and typed columns.

PySpark select()

PySpark select() function is used to select specific columns from a DataFrame. As a transformation operation, it returns a new DataFrame with the selected columns.

Below are various methods for selecting columns from a DataFrame.

# Select columns in different ways
df.select("firstname", "lastname").show()

df.select(df.firstname, df.lastname).show()

df.select(df["firstname"], df["lastname"]).show()

# By using col() function
from pyspark.sql.functions import col

df.select(col("firstname"), col("lastname")).show()

# Select columns by regular expression
df.select(df.colRegex("`^.*name*`")).show()

PySpark selectExpr()

PySpark selectExpr() is a function of DataFrame that is similar to select(), the difference is it takes a set of SQL expressions in a string to execute. This gives the ability to run SQL-like expressions without creating a temporary table and views.

selectExpr() just has one syntax that takes SQL expressions in a string and returns a new DataFrame. Note that unlike select(), it doesn’t have a signature to take the Column type.

The following is the syntax of selectExpr() function.

# Syntax of selectExpr()
DataFrame.selectExpr(*expr)

Example

# Using selectExpr()
# Increment month of the date
df.selectExpr(
    "date", 
    "increment", 
    "add_months(to_date(date,'yyyy-MM-dd'), cast(increment as int)) as inc_date"
).show()