How to Add Row Number without partition in PySpark?

How to add a new column with a row number to a PySpark DataFrame without partitioning? The pyspark.sql.window module provides functions like row_number(), rank(), and dense_rank() to add ranking-based columns to a DataFrame. Among them, row_number() assigns a unique sequential number to each row based on the ordering specified in a window specification, whereas rank() assigns the same rank to rows with duplicate values but skips subsequent ranks. The dense_rank() function assigns the same rank to duplicates but does not skip ranks. This approach is useful for scenarios where global row numbering is required.

In this article, you will learn multiple methods to achieve this and explore practical use cases for full-DataFrame ordering.

Key Points-

The row_number() function assigns a unique sequential number to each row according to the specified sort order.
The rank() function gives identical ranks to rows with duplicate values but leaves gaps in the ranking sequence for subsequent rows.
The dense_rank() function also assigns the same rank to duplicates but ensures that ranks remain consecutive without gaps.
When no partition is defined, the entire DataFrame is treated as a single group for ranking.
All three functions require a Window specification with at least an orderBy() clause.
Using these functions without partitioning is ideal for global row numbering, which is useful for tasks like pagination, selecting the top N records, or removing duplicates.
You can apply all three ranking functions in the same DataFrame to compare their behaviors.
These functions are provided by the pyspark.sql.functions module.
Sorting can be performed on one or multiple columns, including descending order using col("column_name").desc().

PySpark Add Row Number without Partition

You can use the row_number() function to add a row number as a new column to a DataFrame without applying any partitioning. To do this by using the row_number() function along with Window.orderBy() function, specifying only the ordering criteria. In this case, this function considers the entire DataFrame as a single group and adds row numbers based on the global order of the specified column.


# Add row number without partitioning
# Imports 
from pyspark.sql import SparkSession
from pyspark.sql.functions import row_number, rank, dense_rank, col
from pyspark.sql.window import Window

# Start Spark session
spark = SparkSession.builder.appName("RowNumberExample").getOrCreate()

# Sample data
data = [
    ("James", "Sales", 3000),
    ("Michael", "Sales", 4600),
    ("Robert", "Sales", 4100),
    ("Maria", "Finance", 3000),
    ("Scott", "Finance", 3300),
    ("Jen", "Finance", 3900),
    ("Jeff", "Marketing", 3000),
    ("Kumar", "Marketing", 2000),
    ("Saif", "Sales", 4100)
]

columns = ["employee_name", "department", "salary"]
df = spark.createDataFrame(data, columns)
df.show()

# Define a global window ordered by salary
global_window = Window.orderBy(col("salary"))

# Add row_number column
result_df = df.withColumn("row_number", row_number().over(global_window ))

result_df.show()

Yields below the output.

The above code has returned the final DataFrame with an additional row_number column. Rows are numbered globally from 1, starting with the lowest salary.

PySpark rank without Partition

You can also use the rank() function to add a row number(rank) as a new column to a DataFrame without applying any partitioning. To do this by using the rank() function along with Window.orderBy()function, specifying only the ordering criteria. In this case, this function considers the entire DataFrame as a single group and adds row numbers based on the global order of the specified column.


# Add the rank to each row without partition
global_window = Window.orderBy(col("salary"))
df.withColumn("rank", rank().over(global_window)).show()

# Output:
# +-------------+----------+------+----+
# |employee_name|department|salary|rank|
# +-------------+----------+------+----+
# |        Kumar| Marketing|  2000|   1|
# |        James|     Sales|  3000|   2|
# |        Maria|   Finance|  3000|   2|
# |         Jeff| Marketing|  3000|   2|
# |        Scott|   Finance|  3300|   5|
# |          Jen|   Finance|  3900|   6|
# |       Robert|     Sales|  4100|   7|
# |         Saif|     Sales|  4100|   7|
# |      Michael|     Sales|  4600|   9|
# +-------------+----------+------+----+

The above code has returned the final DataFrame with an additional rank column. Rows with the same salary are assigned the same rank, and the next rank is skipped accordingly. Ranking is applied globally, starting from the lowest salary.

PySpark dense_rank Without Partition

To use the dense_rank() function you can assign ranks without partitioning by using orderBy(). It will treat the whole DataFrame as a single group and add row numbers based on the global order of the specified column.


# Add the rank to each row without partition
global_window = Window.orderBy(col("salary"))
df.withColumn("dense_rank", dense_rank().over(global_window)).show()

# Output:
# +-------------+----------+------+----------+
# |employee_name|department|salary|dense_rank|
# +-------------+----------+------+----------+
# |        Kumar| Marketing|  2000|         1|
# |        James|     Sales|  3000|         2|
# |        Maria|   Finance|  3000|         2|
# |         Jeff| Marketing|  3000|         2|
# |        Scott|   Finance|  3300|         3|
# |          Jen|   Finance|  3900|         4|
# |       Robert|     Sales|  4100|         5|
# |         Saif|     Sales|  4100|         5|
# |      Michael|     Sales|  4600|         6|
# +-------------+----------+------+----------+

The above code has returned the final DataFrame with an additional dense_rank column. Rows with the same salary receive the same rank, but unlike rank(), no ranks are skipped. Dense ranking is applied globally, beginning from the lowest salary.

Difference between row_number(), rank(), and dense_rank() without Partition

This example shows the differences between the rank(), dense_rank(), and row_number() functionsin PySpark without a window partition.


# Complete example of Difference between PySpark rank(), dense_rank(),and row_number() 
# by adding rank number without partition

window_spec = Window.orderBy(col("salary"))
result_df = df.withColumn("row_number", row_number().over(window_spec))\
                  .withColumn("rank", rank().over(window_spec)) \
                  .withColumn("dense_rank", dense_rank().over(window_spec))

# Show the result
result_df.show()

# Output:
# +-------------+----------+------+----------+----+----------+
# |employee_name|department|salary|row_number|rank|dense_rank|
# +-------------+----------+------+----------+----+----------+
# |        Kumar| Marketing|  2000|         1|   1|         1|
# |        James|     Sales|  3000|         2|   2|         2|
# |        Maria|   Finance|  3000|         3|   2|         2|
# |         Jeff| Marketing|  3000|         4|   2|         2|
# |        Scott|   Finance|  3300|         5|   5|         3|
# |          Jen|   Finance|  3900|         6|   6|         4|
# |       Robert|     Sales|  4100|         7|   7|         5|
# |         Saif|     Sales|  4100|         8|   7|         5|
# |      Michael|     Sales|  4600|         9|   9|         6|
# +-------------+----------+------+----------+----+----------+

Frequently Asked Questions of Add Row Number without partition

It means numbering rows across the entire DataFrame without dividing it into groups; the whole dataset is treated as a single partition.

Use row_number() with a Window specification that only has orderBy(), omitting partitionBy()

Skipping partitionBy() ensures that numbering is global across the full DataFrame instead of restarting within groups.

All treat the entire DataFrame as one group, but row_number() gives unique sequential numbers, rank() skips numbers after ties, and dense_rank() doesn’t skip ranks.

Adding a new row_number column will overwrite the existing one unless you choose a different column name.

Add the global row number column, then filter rows where row_number <= N.

Examples include creating unique indexes, selecting top records across the entire dataset, and deduplicating rows in a globally sorted manner.

Conclusion

In this article, you have learned how to add row numbers or rankings in a PySpark DataFrame without using partitioning. Using row_number(), rank(), and dense_rank() with only orderBy() in a Window specification you can treat the entire DataFrame as one global group. This technique is especially useful for use cases like generating globally sorted indexes, selecting the top N records, and displaying data in pages for distributed data processing.

Understanding how these ranking functions behave without partitions, particularly how they handle ties, helps you make informed decisions in your data pipelines.

Happy learning!!