In PySpark, the rank() window function adds a new column by assigning a rank to each row within a partition of a dataset based on the specified order criteria. When multiple rows have the same value for the order column, they receive the same rank, but subsequent ranks are skipped. This function is especially useful in data analysis tasks such as identifying top performers within a group, filtering top-N records, or sorting data within subgroups.
In this article, you’ll learn how to use the rank() function along with partitionBy() and orderBy() to group data and rank rows based on column values.
Key Points-
rank()is a PySpark window function for ranking rows.- Same values (ties) get the same rank.
- The next rank is skipped when there's a tie.
- Requires a window specification (
Window.partitionBy().orderBy()). - Without
partitionBy(), it treats the entire DataFrame as one group. - Supports ascending or descending order.
- Useful in analytics, leaderboards, and grouped ranking.
- Works well for “top-N by group” queries.
rank()vsdense_rank(): onlyrank()skips ranks on ties.row_number()is used to assign a unique sequential number to each row.
<a id="h-pyspark-rank-window-function"></a>
PySpark rank Window Function
The rank() function assigns a rank to each row within a certain partition based on the specified order.
If rows with the same value in the ordering column, they receive the same rank, and skip the next rank.
<a id="h-syntax-of-the-rank-function"></a>
Syntax of the rank() Function
The following is the syntax of the rank() function.
# Syntax of the rank() function
pyspark.sql.functions.rank()
<a id="h-parameters"></a>
Parameters
- It has no parameters.
<a id="h-return-value"></a>
Return Value
It returns a new column with ranks (IntegerType), where rows with the same value share the same rank, and skipped ranks occur when duplicates exist.
<a id="h-pyspark-rank-partition-by"></a>
PySpark rank Partition by
You can use the rank() function to add a new column to DataFrame by defining a window with a partition and a specified order. It adds a rank number to each row. If some rows have the same value, they get the same rank, and the next rank numbers are skipped.
# Add the rank to each row of a specified order within a partition
# Create DataFrame
# Imports
from pyspark.sql import SparkSession
from pyspark.sql.functions import row_number, rank, dense_rank, col
from pyspark.sql.window import Window
# Create SparkSession
spark = SparkSession.builder.appName("aistudylabs").getOrCreate()
# Sample data
data = [
("James", "Sales", 3000),
("Michael", "Sales", 4600),
("Robert", "Sales", 4100),
("Maria", "Finance", 3000),
("Scott", "Finance", 3300),
("Jen", "Finance", 3900),
("Jeff", "Marketing", 3000),
("Kumar", "Marketing", 2000),
("Saif", "Sales", 4100)
]
columns = ["employee_name", "department", "salary"]
df = spark.createDataFrame(data, columns)
df.show()
window_spec = Window.partitionBy("department").orderBy(col("salary"))
df.withColumn("rank", rank().over(window_spec)).show()
Yields below the output.
<a id="h-pyspark-rank-without-partition"></a>
PySpark rank without Partition
You can also use the rank() function to add a row number(rank) as a new column to a DataFrame without applying any partitioning. To do this by using the rank() function along with Window.orderBy()function, specifying only the ordering criteria. In this case, this function considers the entire DataFrame as a single group and adds row numbers based on the global order of the specified column.
# Add the rank to each row without partition
global_window = Window.orderBy(col("salary").desc())
df.withColumn("rank", rank().over(global_window)).show()
# Output:
# +-------------+----------+------+----+
# |employee_name|department|salary|rank|
# +-------------+----------+------+----+
# | Michael| Sales| 4600| 1|
# | Robert| Sales| 4100| 2|
# | Saif| Sales| 4100| 2|
# | Jen| Finance| 3900| 4|
# | Scott| Finance| 3300| 5|
# | James| Sales| 3000| 6|
# | Maria| Finance| 3000| 6|
# | Jeff| Marketing| 3000| 6|
# | Kumar| Marketing| 2000| 9|
# +-------------+----------+------+----+
<a id="h-pyspark-rank-order-by-desc"></a>
PySpark rank Order by Desc
To add ranks within each group based on descending order, you can use the rank() function along with a window specification.
# Add the rank to each row within a partition by descending order
window_spec = Window.partitionBy("department").orderBy(col("salary").desc())
df.withColumn("rank", rank().over(window_spec)).show()
# Output:
# +-------------+----------+------+----+
# |employee_name|department|salary|rank|
# +-------------+----------+------+----+
# | Jen| Finance| 3900| 1|
# | Scott| Finance| 3300| 2|
# | Maria| Finance| 3000| 3|
# | Jeff| Marketing| 3000| 1|
# | Kumar| Marketing| 2000| 2|
# | Michael| Sales| 4600| 1|
# | Robert| Sales| 4100| 2|
# | Saif| Sales| 4100| 2|
# | James| Sales| 3000| 4|
# +-------------+----------+------+----+
<a id="h-pyspark-rank-vs-dense-rank"></a>
PySpark rank vs dense_rank
In this example, shows how these functions handle ranking when some rows have the same values.
# Difference between PySpark rank() and dense_rank()
window_spec = Window.partitionBy("department").orderBy(col("salary"))
result_df = df.withColumn("rank", rank().over(window_spec)) \
.withColumn("dense_rank", dense_rank().over(window_spec))
result_df.show()
# Output:
# +-------------+----------+------+----+----------+
# |employee_name|department|salary|rank|dense_rank|
# +-------------+----------+------+----+----------+
# | Maria| Finance| 3000| 1| 1|
# | Scott| Finance| 3300| 2| 2|
# | Jen| Finance| 3900| 3| 3|
# | Kumar| Marketing| 2000| 1| 1|
# | Jeff| Marketing| 3000| 2| 2|
# | James| Sales| 3000| 1| 1|
# | Robert| Sales| 4100| 2| 2|
# | Saif| Sales| 4100| 2| 2|
# | Michael| Sales| 4600| 4| 3|
# +-------------+----------+------+----+----------+
<a id="h-pyspark-rank-vs-dense-rank-vs-row-number"></a>
PySpark rank() vs dense_rank() vs row_number()
This example shows the differences between the rank(), dense_rank(), and row_number() functionsin PySpark with a window partition. We’ll apply these functions to a DataFrame to add columns that represent row rankings based on the specified partition.
# Complete example of Difference between PySpark rank(), dense_rank(),and row_number()
# Applying partitionBy() and orderBy()
window_spec = Window.partitionBy("department").orderBy(col("salary"))
result_df = df.withColumn("row_number", row_number().over(window_spec))\
.withColumn("rank", rank().over(window_spec)) \
.withColumn("dense_rank", dense_rank().over(window_spec))
# Show the result
result_df.show()
# Output:
# +-------------+----------+------+----------+----+----------+
# |employee_name|department|salary|row_number|rank|dense_rank|
# +-------------+----------+------+----------+----+----------+
# | Maria| Finance| 3000| 1| 1| 1|
# | Scott| Finance| 3300| 2| 2| 2|
# | Jen| Finance| 3900| 3| 3| 3|
# | Kumar| Marketing| 2000| 1| 1| 1|
# | Jeff| Marketing| 3000| 2| 2| 2|
# | James| Sales| 3000| 1| 1| 1|
# | Robert| Sales| 4100| 2| 2| 2|
# | Saif| Sales| 4100| 3| 2| 2|
# | Michael| Sales| 4600| 4| 4| 3|
# +-------------+----------+------+----------+----+----------+
<a id="h-frequently-asked-questions-of-pyspark-rank-function"></a>
Frequently Asked Questions of PySpark rank() Function
The rank() function assigns a ranking number to each row within a partition based on a specified order. Rows with the same value receive the same rank, and the next ranks are skipped.
rank() gives the same rank to duplicate values and skips the next ranks.
You can use rank() without partitionBy(). In that case, the entire DataFrame is treated as a single group.
They receive the same rank, and the next rank(s) are skipped. For example, if two rows are tied at rank 1, the next rank will be 3.
It returns a column of type IntegerType, representing the rank assigned to each row.
You can use rank() with partitionBy() and orderBy(), then filter rows where rank is less than or equal to N.
<a id="h-conclusion"></a>
Conclusion
In this article, I explained the rank() function and how to use it with a window specification to create a new column in a DataFrame. The function handles rows with identical values both with and without partitioning, based on the specified order. I also discussed the differences between rank(), dense_rank(), and row_number(), helping you choose the most suitable function for your specific data analysis needs.
<a id="h-references"></a>
References
<a id="h-related-articles"></a>
Related Articles
- PySpark Select Top N Rows From Each Group
- PySpark Find Maximum Row per Group in DataFrame
- PySpark Select First Row of Each Group?
- PySpark Window Functions
- Pyspark Select Distinct Rows
- PySpark Distinct to Drop Duplicate Rows
- Explain PySpark repartition() vs partitionBy() functions
- Explain PySpark dense_rank() function with examples
- Explain PySpark row_number() function with examples