Built with Tutify.io

    AIStudyLabs

    Spark Epoch time to timestamp and Date

    Spark Epoch time to timestamp and Date

    In this Spark article, you will learn how to convert or cast Epoch time to Timestamp and Date using SQL function from_unixtime() and Scala language

    Epoch time is also known as Unix time which specifies the moment in time since 1970-01-01 00:00:00 UTC. It is the number of seconds passed since Epoch time. Epoch time is widely used in Unix like operating systems.

    <a id="h-1-convert-epoch-time-to-timestamp"></a>

    1. Convert Epoch time to timestamp

    from_unixtime() SQL function is used to convert or cast Epoch time to timestamp string and this function takes Epoch time as a first argument and formatted string time as the second argument. As a first argument, we use unix_timestamp() which returns the current timestamp in Epoch time (Long) as an argument.

    
  //Convert unix timestamp to timestamp
  val timeDF = Seq(1).toDF("seq").select(
    from_unixtime(unix_timestamp()).as("timestamp_1"),
    from_unixtime(unix_timestamp(),"MM-dd-yyyy HH:mm:ss").as("timestamp_2"),
    from_unixtime(unix_timestamp(),"dd-MM-yyyy HH:mm:ss").as("timestamp_3"),
    from_unixtime(unix_timestamp(),"HH:mm:ss").as("timestamp_4")
  ).show()

    Yields below output

    
// Output:
+-------------------+-------------------+-------------------+-----------+
|        timestamp_1|        timestamp_2|        timestamp_3|timestamp_4|
+-------------------+-------------------+-------------------+-----------+
|2019-12-24 10:36:12|12-24-2019 10:36:12|24-12-2019 10:36:12|   10:36:12|
+-------------------+-------------------+-------------------+-----------+

    <a id="h-1-convert-epoch-time-to-date"></a>

    1. Convert Epoch time to Date

    Similar to above, here we use the same from_unixtime() SQL function to convert Epoch time to date besides we need to provide formatted string date as a second argument to the function

    
  // Convert unix timestamp to date
  val dateDF = Seq(1).toDF("seq").select(
    from_unixtime(unix_timestamp(),"MM-dd-yyyy").as("date_1"),
    from_unixtime(unix_timestamp(),"dd-MM-yyyy HH:mm:ss").as("date_2"),
    from_unixtime(unix_timestamp(),"yyyy-MM-dd").as("date_3")
  ).show(false)

    Yields below output

    
// Output:
+----------+-------------------+----------+
|date_1    |date_2             |date_3    |
+----------+-------------------+----------+
|12-24-2019|24-12-2019 10:36:12|2019-12-24|
+----------+-------------------+----------+

    The complete example is available at GitHub project for reference

    <a id="h-conclusion"></a>

    Conclusion

    In this Spark article, you have learned how to convert or cast the Epoch time to Date or Timestamp using from_unixtime() function along with Scala example.

    <a id="h-related-articles"></a>

    Related Articles