Problem 1: When I try to add a month to the data column with a value from another column I am getting a PySpark error TypeError: Column is not iterable.
from pyspark.sql.functions import add_months
data=[("2019-01-23",1),("2019-06-24",2),("2019-09-20",3)]
df=spark.createDataFrame(data).toDF("date","increment")
df.select(df.date,df.increment,add_months(df.date,df.increment)).show()
Get's below PySpark Error during run-time.
TypeError: Column is not iterable
<a id="h-solution-for-typeerror-column-is-not-iterable"></a>
Solution for TypeError: Column is not iterable
PySpark add_months() function takes the first argument as a column and the second argument is a literal value. if you try to use Column type for the second argument you get "TypeError: Column is not iterable". In order to fix this use expr() function as shown below.
df.select(df.date,df.increment,
expr("add_months(date,increment)")
.alias("inc_date")).show()
<a id="h-related-articles"></a>
Related Articles
- PySpark Shell Command Usage with Examples
- PySpark Find Maximum Row per Group in DataFrame
- PySpark Column Class | Operators & Functions
- PySpark Column alias after groupBy() Example
- PySpark Get Number of Rows and Columns
- PySpark Groupby on Multiple Columns
- PySpark alias() Column & DataFrame Examples
- PySpark Replace Column Values in DataFrame