Built with Tutify.io

    AIStudyLabs

    Pyspark to_date() vs date_format()

    Pyspark to_date() vs date_format()

    Working with dates in PySpark can be challenging, especially when the input data is in various string formats. Fortunately, PySpark offers built-in functions such as to_date() and date_format() from the pyspark.sql.functions module to make this easier. The to_date() function is used to convert a string into a DateType, whereas date_format() is used toconvert a DateType into a formatted string representation.

    In this article, you’ll learn the difference between to_date() and date_format() functions in PySpark using a simple real-world example.

    Key Points-

    <a id="ember55"></a>

    Sample Dataset

    Let's say you have a DataFrame with dates in string format

    
# Create DataFrame
# Imports
from pyspark.sql.functions import *
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DateFunctions").getOrCreate()

# Create a sample DataFrame with string dates
df = spark.createDataFrame([["02-03-2013"], ["05-06-2023"]], ["input"])
df.show()
df.printSchema()

    Yields below the output.

    <a id="h-pyspark-to-date"></a>

    PySpark to_date()

    The to_date() function is used to convert a string column into a DateType column based on the format you specify. If no format is given, it defaults to 'yyyy-MM-dd'.

    It can accept date strings in any format as the first argument. However, if the input values do not match the specified format in the second argument, to_date() will return null for those rows.

    
# Convert string column to Datetype
df_date = df.select(col("input"),to_date(col("input"),"MM-dd-yyyy").alias("date"))
df_date.show()
df_date.printSchema()

    Yields below the output.

    <a id="h-pyspark-date-format"></a>

    PySpark date_format()

    The date_format() function in PySpark is used to convert a DateType or TimestampType column into a formatted string using a specified pattern, such as "dd/MM/yyyy" or "yyyy-MM". It cannot work directly on string columns. If your column contains date values as strings, you must first convert them using to_date() or to_timestamp() to provide the input in the correct type.

    Additionally, the date_format() function requires a valid format string as its second argument—if it's missing, PySpark will raise a TypeError. The first argument should be a column of DateType or TimestampType. If the input is not in the correct format (like 'yyyy-MM-dd'), the function may return null. When the input is valid, date_format() converts the date or timestamp into the string format specified in the second argument.

    
# Convert date column to string
df_string = df.select(
    col("input"),
    to_date(col("input"), "MM-dd-yyyy").alias("date"),
    date_format(to_date(col("input"), "MM-dd-yyyy"), "dd/MM/yyyy").alias("formatted_date"))
df_string.show()
df_string.printSchema()

    Yields below the output.

    
# Output:
+----------+----------+--------------+
|     input|      date|formatted_date|
+----------+----------+--------------+
|02-03-2013|2013-02-03|    03/02/2013|
|05-06-2023|2023-05-06|    06/05/2023|
+----------+----------+--------------+

root
 |-- input: string (nullable = true)
 |-- date: date (nullable = true)
 |-- formatted_date: string (nullable = true)

    <a id="h-pyspark-to-date-vs-date-format"></a>

    PySpark to_date() vs date_format()

    The following table provides a complete guide to the differences between PySpark to_date() and date_format() functions.

    | Feature | to_date() | date_format() | | --- | --- | --- | | Purpose | Converts a string to DateType | Formats a DateType/TimestampType column as a string | | Returns | DateType column | StringType column | | Input | String column + input format | Date/Timestamp column + desired format | | Common Use Case | Data cleaning, type casting | Date formatting for display or export | | Example Output | 2013-02-03 | 03/02/2013 |

    <a id="h-frequently-asked-questions-of-pyspark-to-date-vs-date-format"></a>

    Frequently Asked Questions of Pyspark to_date() vs date_format()

    to_date() converts a string column into a DateType column.
    date_format() converts a DateType or TimestampType column into a formatted string.

    date_format() requires a column of type DateType or TimestampType.
    It cannot be used directly on a string column. If your input is a string (e.g., "MM-dd-yyyy"), you must first convert it using to_date() or to_timestamp() before applying date_format().

    to_date() returns the date in yyyy-MM-dd format (standard ISO format), even if the input was in a different format.

    to_date() returns a column of type DateType.

    date_format() returns a column of type StringType.

    You cannot use to_date() to change the format of a date — it’s only used to convert a string into a DateType column based on the input format.

    If the format is incorrect, to_date() returns null for that row.

    You can use the date_format() to implement both DateType and TimestampType columns.

    Use to_date(col, "MM/dd/yyyy"). Example:
    to_date(col("input"), "MM/dd/yyyy")

    <a id="h-conclusion"></a>

    Conclusion

    I have explained the key differences between PySpark to_date() and date_format() functions using a practical example. Also learned how to_date() it helps convert date strings into actual DateType columns, and how date_format() can be used to format those dates into a specific string representation. Understanding when and how to use these functions is essential for processing and presenting date values correctly in PySpark. By combining them effectively, you can handle real-world scenarios like parsing inconsistent date strings and formatting dates for reports or data exports with confidence.

    Happy Learning!!

    <a id="h-related-articles"></a>

    Related Articles: