Introduction
Following are sort functions
| SPARK SQL SORT FUNCTION SYNTAX | SPARK FUNCTION DESCRIPTION | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------- | | asc(columnName: String): Column | asc function is used to specify the ascending order of the sorting column on DataFrame | | asc_nulls_first(columnName: String): Column | Similar to asc function but null values return first and then non-null values | | asc_nulls_last(columnName: String): Column | Similar to asc function but non-null values return first and then null values | | desc(columnName: String): Column | desc function is used to specify the descending order of the DataFrame sorting column. | | desc_nulls_first(columnName: String): Column | Similar to desc function but null values return first and then non-null values. | | desc_nulls_last(columnName: String): Column | Similar to desc function but non-null values return first and then null values. |
asc() – ascending function
asc function is used to specify the ascending order of the sorting column on DataFrame.
Example in PySpark:
from pyspark.sql import SparkSession
from pyspark.sql.functions import asc
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.createDataFrame([(1, "apple"), (3, "banana"), (2, "lemon")], ["id", "fruit"])
df.sort(asc("id")).show()
output:
+---+------+
| id| fruit|
+---+------+
| 1| apple|
| 2| lemon|
| 3|banana|
+---+------+
asc_nulls_first() – ascending with nulls first
Similar to asc function but null values return first and then non-null values.
Example in PySpark:
from pyspark.sql import SparkSession
from pyspark.sql.functions import asc_nulls_first
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.createDataFrame([(None, "kiwi"), (3, "banana"), (2, "lemon")], ["id", "fruit"])
df.sort(asc_nulls_first("id")).show()
output:
+----+------+
| id| fruit|
+----+------+
|null| kiwi|
| 2| lemon|
| 3|banana|
+----+------+
asc_nulls_last() – ascending with nulls last
Similar to asc function but non-null values return first and then null values.
Example in PySpark:
from pyspark.sql import SparkSession
from pyspark.sql.functions import asc_nulls_last
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.createDataFrame([(None, "kiwi"), (3, "banana"), (2, "lemon")], ["id", "fruit"])
df.sort(asc_nulls_last("id")).show()
output:
+----+------+
| id| fruit|
+----+------+
| 2| lemon|
| 3|banana|
|null| kiwi|
+----+------+
desc() – descending function
desc function is used to specify the descending order of the DataFrame sorting column.
Example in PySpark:
from pyspark.sql import SparkSession
from pyspark.sql.functions import desc
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.createDataFrame([(1, "apple"), (3, "banana"), (2, "lemon")], ["id", "fruit"])
df.sort(desc("id")).show()
output:
+---+------+
| id| fruit|
+---+------+
| 3|banana|
| 2| lemon|
| 1| apple|
+---+------+
desc_nulls_first() – descending with nulls first
Similar to desc function but null values return first and then non-null values.
Example in PySpark:
from pyspark.sql import SparkSession
from pyspark.sql.functions import desc_nulls_first
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.createDataFrame([(None, "kiwi"), (3, "banana"), (2, "lemon")], ["id", "fruit"])
df.sort(desc_nulls_first("id")).show()
output:
+----+------+
| id| fruit|
+----+------+
|null| kiwi|
| 3|banana|
| 2| lemon|
+----+------+
desc_nulls_last() – descending with nulls last
Similar to desc function but non-null values return first and then null values.
Example in PySpark:
from pyspark.sql import SparkSession
from pyspark.sql.functions import desc_nulls_last
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.createDataFrame([(None, "kiwi"), (3, "banana"), (2, "lemon")], ["id", "fruit"])
df.sort(desc_nulls_last("id")).show()
output:
+----+------+
| id| fruit|
+----+------+
| 3|banana|
| 2| lemon|
|null| kiwi|
+----+------+