PySpark is widely adopted by Data Engineers and Big Data professionals because of its capability to process massive datasets efficiently using distributed computing. Whether you’re preparing for a data engineering interview or working on real-world big data projects, having a strong command of PySpark functions can significantly improve your productivity and problem-solving skills.
This guide compiles the Top 100 PySpark functions every data engineer should know, grouped into practical categories:
- Basic DataFrame Operations
- Column Operations
- String Functions
- Date and Time Functions
- Aggregation and Grouping
- Window Functions
- Null Handling
- Array and Map Functions
- Joins and Merging
- File I/O Functions
- UDFs and Performance
- Miscellaneous Utility Functions
Each function is explained in a clear, concise manner with one-line descriptions, code examples, and outputs, making this guide a quick reference for interviews and on-the-job problem-solving.
By the end of this article, you’ll have a ready-to-use PySpark function reference that not only helps in cracking technical interviews but also serves as a practical cheat sheet for everyday data engineering tasks.
Now that you know the key categories, let’s explore each one in detail with examples, syntax, and outputs.
<a id="basic-dataframe-operations"></a>
Basic DataFrame Operations
Basic DataFrame operations provide the foundation for working with PySpark, allowing you to create, inspect, filter, and transform data. These functions are essential for any PySpark project, forming the starting point before applying advanced transformations.
<a id="h-1-create-an-empty-dataframe"></a>
1. Create an empty DataFrame
Use spark.createDataFrame() with an empty RDD and a schema.
# Create an empty DataFrame
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('AIStudyLabs.com').getOrCreate()
# Creates Empty RDD
emptyRDD = spark.sparkContext.emptyRDD()
print(emptyRDD)
# Create empty DataFrame from empty RDD
df = spark.createDataFrame(emptyRDD,schema)
df.printSchema()
Yields below the output.
<a id="h-2-convert-rdd-to-dataframe"></a>
2. Convert RDD to DataFrame
Use toDF() to convert RDD into a DataFrame.
# Convert Rdd to DataFrame
dept = [("Finance",10),("Marketing",20),("Sales",30),("IT",40)]
rdd = spark.sparkContext.parallelize(dept)
df = rdd.toDF()
df.show(truncate=False)
Yields below the output.
3. Select Specific Columns
Use select() to return specific columns.
# Select Specific Column
df.select("_1").show()
Yields below the output.
# Output:
+---------+
| _1 |
+---------+
| Finance |
|Marketing|
| Sales |
| IT |
+---------+
<a id="h-4-filter-rows"></a>
4. Filter Rows
Use filter() or where() to filter data.
# Use filter() or where() to filter data.
df.filter(df._2 > 10).show(truncate=False)
Yields below the output.
# Output:
+---------+---+
| _1 | _2|
+---------+---+
|Marketing| 20|
| Sales | 30|
| IT | 40|
+---------+---+
<a id="h-5-add-new-column"></a>
5. Add New Column
You can add a new column using withColumn() function.
# Add new column using df.withColumn
df.withColumn("new_id", df._2 + 100).show(truncate=False)
Yields below the output.
# Output:
+---------+---+-------+
| _1 | _2| new_id|
+---------+---+-------+
| Finance | 10| 110|
|Marketing| 20| 120|
| Sales | 30| 130|
| IT | 40| 140|
+---------+---+-------+
<a id="h-6-rename-column"></a>
6. Rename Column
To rename a specified column, you can use the withColumnRenamed().
# Rename specific column name
df.withColumnRenamed("_1", "department").show(truncate=False)
Yields below the output.
# Output:
+----------+---+
|department| _2|
+----------+---+
| Finance| 10|
| Marketing| 20|
| Sales| 30|
| IT| 40|
+----------+---+
<a id="h-7-drop-column"></a>
7. Drop Column
You can drop a specified column using drop() function.
# Drop specified column
df.drop("_2").show(truncate=False)
Yields below the output.
# Output:
+---------+
| _1 |
+---------+
| Finance |
|Marketing|
| Sales |
| IT |
+---------+
<a id="h-8-sort-rows"></a>
8. Sort Rows
To sort rows of a DataFrame using orderBy().
# Sort Rows
df.orderBy(df._2.desc()).show(truncate=False)
Yields below the output.
# Output:
+---------+---+
| _1 | _2|
+---------+---+
| IT | 40|
| Sales | 30|
|Marketing| 20|
| Finance | 10|
+---------+---+
<a id="h-9-count-rows-in-dataframe"></a>
9. Count Rows in DataFrame
Count the total number of rows present in a DataFrame using .count().
# Count Rows in DataFrame
df.count()
Yields below the output.
# Output:
4
<a id="h-10-display-dataframe-content"></a>
10. Display DataFrame Content
Use .show() to display the DataFrame rows in a tabular format.
# Display the content
df.show()
Yields below the output.
# Output:
+---------+---+
| _1 | _2|
+---------+---+
| Finance | 10|
|Marketing| 20|
| Sales | 30|
| IT | 40|
+---------+---+
<a id="column-operations"></a>
Column Operations
Column operations allow you to add, rename, drop, and transform DataFrame columns. These functions help manage schema changes and apply specific logic to individual columns for data cleaning and preparation.
<a id="h-11-add-literal-column"></a>
11. Add Literal Column
Add a column with a constant literal value for all rows, you can use the lit().
# Add a column with a constant literal value
df.withColumn("constant_col", lit("constant")).show()
Yields below the output.
# Output:
+---------+---+------------+
| _1 | _2|constant_col|
+---------+---+------------+
| Finance | 10| constant|
|Marketing| 20| constant|
| Sales | 30| constant|
| IT | 40| constant|
+---------+---+------------+
<a id="h-12-change-column-datatype"></a>
12. Change Column DataType
Change the data type of a specified column using .cast().
# Change the data type of a specified column
df.withColumn("_2", df._2.cast("string")).printSchema()
Yields below the output.
# Output:
root
|-- _1: string (nullable = true)
|-- _2: string (nullable = true)
<a id="h-13-replace-existing-column"></a>
13. Replace Existing Column
You can overwrite an existing column by applying transformations and assigning it back using withColumn()
# Replace Existing Column
df.withColumn("_2", df._2 + 100).show()
Yields below the output.
# Output:
+---------+---+
| _1 | _2|
+---------+---+
| Finance |110|
|Marketing|120|
| Sales |130|
| IT |140|
+---------+---+
<a id="h-14-add-column-based-on-condition"></a>
14. Add Column Based on Condition
Create a new column based on a condition using when().otherwise().
# Add Column Based on Condition
df.withColumn("high_dept", when(df._2 > 20, True).otherwise(False)).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2|high_dept |
+---------+---+----------+
| Finance | 10| False|
|Marketing| 20| False|
| Sales | 30| True|
| IT | 40| True|
+---------+---+----------+
<a id="h-15-alias-column"></a>
15. Alias Column
Rename a column using .alias().
# Rename a column using .alias().
df.select(df._1.alias("department")).show()
Yields below the output.
# Output:
+----------+
|department|
+----------+
| Finance|
| Marketing|
| Sales|
| IT|
+----------+
<a id="h-16-concatenate-columns"></a>
16. Concatenate Columns
Join multiple columns into a single column using concat().
# Concatenate Columns
df.withColumn("dept_info", concat(df._1, lit("-"), df._2)).show()
Yields below the output.
# Output:
+---------+---+------------+
| _1 | _2| dept_info |
+---------+---+------------+
| Finance | 10| Finance-10 |
|Marketing| 20|Marketing-20|
| Sales | 30| Sales-30 |
| IT | 40| IT-40 |
+---------+---+------------+
<a id="h-17-upper-case-a-column"></a>
17. Upper Case a Column
Convert all text in a column to uppercase using the upper()
# Upper Case a Column
df.withColumn("upper_name", upper(df._1)).show()
Yields below the output.
# Output:
+---------+---+-----------+
| _1 | _2|upper_name |
+---------+---+-----------+
| Finance | 10| FINANCE |
|Marketing| 20| MARKETING |
| Sales | 30| SALES |
| IT | 40| IT |
+---------+---+-----------+
<a id="h-18-trim-whitespace"></a>
18. Trim Whitespace
Remove leading and trailing whitespace from a column using trim().
# Trim Whitespace
df.withColumn("trimmed", trim(df._1)).show()
Yields below the output.
# Output:
+---------+---+--------+
| _1 | _2| trimmed|
+---------+---+--------+
| Finance | 10| Finance|
|Marketing| 20|Marketing|
| Sales | 30| Sales |
| IT | 40| IT |
+---------+---+--------+
<a id="h-19-check-if-column-value-is-null"></a>
19. Check if Column Value is Null
Check whether a column has null values.
# Check if Column Value is Null
df.withColumn("is_null", df._1.isNull()).show()
Yields below the output.
# Output:
+---------+---+--------+
| _1 | _2|is_null |
+---------+---+--------+
| Finance | 10| False|
|Marketing| 20| False|
| Sales | 30| False|
| IT | 40| False|
+---------+---+--------+
<a id="h-20-coalesce-column-value"></a>
20. Coalesce Column Value
Replace null values in a column with a default using coalesce().
# Coalesce Column Value
df.withColumn("safe_val", coalesce(df._1, lit("Unknown"))).show()
Yields below the output.
# Output:
+---------+---+---------+
| _1 | _2| safe_val|
+---------+---+---------+
| Finance | 10| Finance |
|Marketing| 20|Marketing|
| Sales | 30| Sales |
| IT | 40| IT |
+---------+---+---------+
<a id="string-functions"></a>
String Functions
String functions are used to clean, manipulate, and analyze text data. They help with tasks such as changing case, extracting substrings, replacing text, and performing pattern matching using regular expressions.
<a id="h-21-substring"></a>
21. Substring
Extracts a part of the string from a specified column using start position and length.
# Extracte Substring
df.withColumn("sub", substring(df._1, 1, 3)).show()
Yields below the output.
# Output:
+---------+---+---+
| _1 | _2|sub|
+---------+---+---+
| Finance | 10|Fin|
|Marketing| 20|Mar|
| Sales | 30| Sa|
| IT | 40| IT|
+---------+---+---+
<a id="h-22-length-of-string"></a>
22. Length of String
Returns the number of characters in the string column.
# Length of String
df.withColumn("length", length(df._1)).show()
Yields below the output.
# Output:
+---------+---+------+
| _1 | _2|length|
+---------+---+------+
| Finance | 10| 7|
|Marketing| 20| 9|
| Sales | 30| 6|
| IT | 40| 2|
+---------+---+------+
<a id="h-23-lowercase-string"></a>
23. Lowercase String
Converts all characters in the string column to lowercase.
# onvert lower string
df.withColumn("lower", lower(df._1)).show()
Yields below the output.
# Output:
+---------+---+---------+
| _1 | _2| lower|
+---------+---+---------+
| Finance | 10| finance|
|Marketing| 20|marketing|
| Sales | 30| sales |
| IT | 40| it |
+---------+---+---------+
<a id="h-24-convert-first-letter-to-upper"></a>
24. Convert First Letter to Upper
Converts the first letter of each word in a string column to uppercase (title case)
# Convert First Letter to Upper
df.withColumn("initcap", initcap(df._1)).show()
Yields below the output.
# Output:
+---------+---+-----------+
| _1 | _2| initcap |
+---------+---+-----------+
| Finance | 10| Finance |
|Marketing| 20| Marketing |
| Sales | 30| Sales |
| IT | 40| It |
+---------+---+-----------+
<a id="h-25-remove-leading-spaces-left-trim"></a>
25. Remove Leading Spaces (Left Trim)
Removes spaces from the beginning (left side) of the string.
# Remove Leading Spaces (Left Trim)
df.withColumn("ltrim", ltrim(df._1)).show()
Yields below the output.
# Output:
+---------+---+--------+
| _1 | _2| ltrim|
+---------+---+--------+
| Finance | 10| Finance|
|Marketing| 20|Marketing|
| Sales | 30|Sales |
| IT | 40|IT |
+---------+---+--------+
<a id="h-26-remove-trailing-spaces-right-trim"></a>
26. Remove Trailing Spaces (Right Trim)
Removes spaces from the end (right side) of the string.
# Remove Trailing Spaces (Right Trim)
df.withColumn("rtrim", rtrim(df._1)).show()
Yields below the output.
# Output:
+---------+---+--------+
| _1 | _2| rtrim|
+---------+---+--------+
| Finance | 10| Finance|
|Marketing| 20|Marketing|
| Sales | 30| Sales |
| IT | 40| IT |
+---------+---+--------+
<a id="h-27-replace-substring"></a>
27. Replace Substring
Replaces all occurrences of a pattern in a string with another using regex.
# Replace Substring
df.withColumn("replaced", regexp_replace(df._1, "a", "@")).show()
Yields below the output.
# Output:
+---------+---+---------+
| _1 | _2| replaced|
+---------+---+---------+
| Finance | 10|Fin@nce |
|Marketing| 20|M@rketing|
| Sales | 30|S@les |
| IT | 40|IT |
+---------+---+---------+
<a id="h-28-check-string-contains-substring"></a>
28. Check String Contains Substring
Checks if the string column contains a specific substring.
# Check String Contains Substring
df.withColumn("contains_i", df._1.contains("i")).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2|contains_i|
+---------+---+----------+
| Finance | 10| True|
|Marketing| 20| True|
| Sales | 30| False|
| IT | 40| True|
+---------+---+----------+
<a id="h-29-pad-string-from-left"></a>
29. Pad String from Left
Pads the string from the left with a specified character to a given length.
# Pad String from Left
df.withColumn("lpad", lpad(df._1, 10, "*")).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2| lpad |
+---------+---+----------+
| Finance | 10|***Finance|
|Marketing| 20|Marketing |
| Sales | 30|**** Sales|
| IT | 40|********IT|
+---------+---+----------+
<a id="h-30-pad-string-from-right"></a>
30. Pad String from Right
Pads the string from the right with a specified character to a given length.
# Pad String from Right
df.withColumn("rpad", rpad(df._1, 10, "#")).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2| rpad |
+---------+---+----------+
| Finance | 10|Finance###|
|Marketing| 20|Marketing#|
| Sales | 30| Sales####|
| IT | 40|IT########|
+---------+---+----------+
<a id="date-and-time-functions"></a>
Date and Time Functions
Date and time functions handle operations on timestamp and date columns, including extracting specific components, formatting, performing date arithmetic, and converting between formats for time-based analysis.
<a id="h-31-current-date"></a>
31. Current Date
Returns the current system date.
# Get the current system date
df.withColumn("today", current_date()).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2| today|
+---------+---+----------+
| Finance | 10|2025-08-11|
|Marketing| 20|2025-08-11|
| Sales | 30|2025-08-11|
| IT | 40|2025-08-11|
+---------+---+----------+
<a id="h-32-current-timestamp"></a>
32. Current Timestamp
Returns the current timestamp, including date and time.
# Get the current timestamp
df.withColumn("now", current_timestamp()).show()
Yields below the output.
# Output:
+---------+---+-----------------------+
| _1 | _2| now |
+---------+---+-----------------------+
| Finance | 10|2025-08-11 14:32:18.123|
|Marketing| 20|2025-08-11 14:32:18.123|
| Sales | 30|2025-08-11 14:32:18.123|
| IT | 40|2025-08-11 14:32:18.123|
+---------+---+-----------------------+
<a id="h-33-format-date"></a>
33. Format Date
It formats a date column into a specified string pattern.
# Get the specified date format
df.withColumn("formatted", date_format(current_date(), "yyyy/MM/dd")).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2| formatted|
+---------+---+----------+
| Finance | 10|2025/08/11|
|Marketing| 20|2025/08/11|
| Sales | 30|2025/08/11|
| IT | 40|2025/08/11|
+---------+---+----------+
<a id="h-34-date-difference"></a>
34. Date Difference
Returns the number of days between two dates.
# Get the difference between two dates
df.withColumn("days_diff", datediff(current_date(), lit("2024-01-01"))).show()
Yields below the output.
# Output:
+---------+---+---------+
| _1 | _2|days_diff|
+---------+---+---------+
| Finance | 10| 589|
|Marketing| 20| 589|
| Sales | 30| 589|
| IT | 40| 589|
+---------+---+---------+
<a id="h-35-add-months"></a>
35. Add Months
Adds a specified number of months to a date.
# Add specified number of months to a date
df.withColumn("next_month", add_months(current_date(), 1)).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2|next_month|
+---------+---+----------+
| Finance | 10|2025-09-11|
|Marketing| 20|2025-09-11|
| Sales | 30|2025-09-11|
| IT | 40|2025-09-11|
+---------+---+----------+
<a id="h-36-months-between"></a>
36. Months Between
Returns the number of months between two dates.
# Get the number of months between two dates
df.withColumn("months_diff", months_between(current_date(), lit("2024-01-01"))).show()
Yields below the output.
# Output:
+---------+---+-----------+
| _1 | _2|months_diff|
+---------+---+-----------+
| Finance | 10| 19.354838|
|Marketing| 20| 19.354838|
| Sales | 30| 19.354838|
| IT | 40| 19.354838|
+---------+---+-----------+
<a id="h-37-extract-year"></a>
37. Extract Year
Extracts the year from a date.
# Extracts the year from a date
df.withColumn("year", year(current_date())).show()
Yields below the output.
# Output:
+---------+---+----+
| _1 | _2|year|
+---------+---+----+
| Finance | 10|2025|
|Marketing| 20|2025|
| Sales | 30|2025|
| IT | 40|2025|
+---------+---+----+
<a id="h-38-extract-month"></a>
38. Extract Month
Extracts the month from a date.
# Extracts the month from a date
df.withColumn("month", month(current_date())).show()
Yields below the output.
# Output:
+---------+---+-----+
| _1 | _2|month|
+---------+---+-----+
| Finance | 10| 8|
|Marketing| 20| 8|
| Sales | 30| 8|
| IT | 40| 8|
+---------+---+-----+
<a id="h-39-string-to-date"></a>
39. String to Date
Converts a string into a DateType column.
# Converts a string into a DateType column
df.withColumn("parsed", to_date(lit("2025-08-07"), "yyyy-MM-dd")).show()
Yields below the output.
# Output:
+---------+---+----------+
| _1 | _2| parsed|
+---------+---+----------+
| Finance | 10|2025-08-07|
|Marketing| 20|2025-08-07|
| Sales | 30|2025-08-07|
| IT | 40|2025-08-07|
+---------+---+----------+
<a id="h-40-day-of-week"></a>
40. Day of Week
Returns the day of the week as a number (1 = Sunday, 7 = Saturday).
# Returns the day of the week as a number
df.withColumn("dow", dayofweek(current_date())).show()
Yields below the output.
# Output:
+---------+---+---+
| _1 | _2|dow|
+---------+---+---+
| Finance | 10| 2|
|Marketing| 20| 2|
| Sales | 30| 2|
| IT | 40| 2|
+---------+---+---+
<a id="aggregation-and-grouping-functions-in-pyspark"></a>
Aggregation and Grouping Functions in PySpark
Aggregation and grouping functions help summarize data by combining multiple rows into a single row based on specified criteria. These functions are often used in data analysis, reporting, and statistical operations.
Below are themost commonly used aggregation and grouping functions in PySpark.
<a id="h-41-groupby"></a>
41. groupBy()
Groups DataFrame rows based on one or more columns for aggregation.
# Group by "department" and count the number of employees in each department
from pyspark.sql import SparkSession
from pyspark.sql.functions import count
spark = SparkSession.builder.appName("AggGroupExample").getOrCreate()
data = [("John", "Sales", 3000),
("Mike", "Sales", 4000),
("Sara", "Finance", 3500),
("Kate", "Finance", 2800),
("Paul", "IT", 4500)]
df = spark.createDataFrame(data, ["name", "department", "salary"])
df.groupBy("department").count().show()
Yields below the output.
# Output:
+----------+-----+
|department|count|
+----------+-----+
| Sales| 2|
| Finance| 2|
| IT| 1|
+----------+-----+
<a id="h-42-agg"></a>
42. agg()
Performs multiple aggregate functions on grouped data.
# Group by department and calculate average and max salary
from pyspark.sql.functions import avg, max
df.groupBy("department").agg(avg("salary"), max("salary")).show()
Yields below the output.
# Output:
+----------+-----------+-----------+
|department|avg(salary)|max(salary)|
+----------+-----------+-----------+
| Sales| 3500.0| 4000|
| Finance| 3150.0| 3500|
| IT| 4500.0| 4500|
+----------+-----------+-----------+
<a id="h-43-count"></a>
43. count()
Counts the number of rows or non-null values in a column.
# Count total rows in the DataFrame
df.select(count("*").alias("total_rows")).show()
Yields below the output.
# Output:
+-----------+
|total_rows |
+-----------+
| 5|
+-----------+
<a id="h-44-min"></a>
44. min()
Returns the smallest value in a column.
# Find the minimum salary
from pyspark.sql import functions as F
data = [("John", 3000), ("Jane", 4000), ("Jake", 2500)]
df = spark.createDataFrame(data, ["Name", "Salary"])
df.select(F.min("Salary").alias("Min_Salary")).show()
Yields below the output.
# Output:
+----------+
|Min_Salary|
+----------+
| 2500|
+----------+
<a id="h-45-avg"></a>
45. avg()
Returns the average value of a column.
# Calculate average salary
df.select(F.avg("Salary").alias("Avg_Salary")).show()
Yields below the output.
# Output:
+----------+
|Avg_Salary|
+----------+
| 3166.6667|
+----------+
<a id="h-46-approx-count-distinct"></a>
46. approx_count_distinct()
It provides an Approximate distinct count for large datasets.
# Approx distinct count of departments
df.select(F.approx_count_distinct("Dept").alias("Approx_Distinct_Dept")).show()
Yields below the output.
# Output:
+-------------------+
|Approx_Distinct_Dept|
+-------------------+
| 2|
+-------------------+
<a id="window-functions"></a>
Window Functions
Window functions perform calculations across a set of rows related to the current row, without collapsing the result into a single output row. They are commonly used for ranking, running totals, and moving averages.
<a id="h-47-row-number"></a>
47. row_number()
Assigns a unique sequential number to each row within a partition.
# Assign a unique sequential number to each row within a partition
from pyspark.sql import SparkSession
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number, col
spark = SparkSession.builder.appName("WindowFunctions").getOrCreate()
data = [("John", "HR", 3000),
("Mike", "HR", 4000),
("Sara", "IT", 5000),
("Kate", "IT", 4500)]
df = spark.createDataFrame(data, ["Name", "Dept", "Salary"])
windowSpec = Window.partitionBy("Dept").orderBy(col("Salary").desc())
df.withColumn("row_number", row_number().over(windowSpec)).show()
Yields below the output.
# Output:
+----+---+------+----------+
|Name|Dept|Salary|row_number|
+----+---+------+----------+
|Mike| HR | 4000 | 1 |
|John| HR | 3000 | 2 |
|Sara| IT | 5000 | 1 |
|Kate| IT | 4500 | 2 |
+----+---+------+----------+
<a id="h-48-rank"></a>
48. rank()
Assigns ranks to rows, leaving gaps when there are ties.
# Rank employees by salary within each department
from pyspark.sql.functions import rank
df.withColumn("rank", rank().over(windowSpec)).show()
Yields below the output.
# Output:
+----+---+------+----+
|Name|Dept|Salary|rank|
+----+---+------+----+
|Mike| HR | 4000 | 1 |
|John| HR | 3000 | 2 |
|Sara| IT | 5000 | 1 |
|Kate| IT | 4500 | 2 |
+----+---+------+----+
<a id="h-49-dense-rank"></a>
49. dense_rank()
Assigns ranks to rows without leaving gaps for ties.
# Dense rank employees by salary within each department
from pyspark.sql.functions import dense_rank
df.withColumn("dense_rank", dense_rank().over(windowSpec)).show()
Yields below the output.
# Output:
+----+---+------+----------+
|Name|Dept|Salary|dense_rank|
+----+---+------+----------+
|Mike| HR | 4000 | 1 |
|John| HR | 3000 | 2 |
|Sara| IT | 5000 | 1 |
|Kate| IT | 4500 | 2 |
+----+---+------+----------+
<a id="h-50-lag"></a>
50. lag()
Returns the value of a column from a previous row.
# Get the previous salary in each department
from pyspark.sql.functions import lag
df.withColumn("prev_salary", lag("Salary", 1).over(windowSpec)).show()
Yields below the output.
# Output:
+----+---+------+-----------+
|Name|Dept|Salary|prev_salary|
+----+---+------+-----------+
|Mike| HR | 4000 | null |
|John| HR | 3000 | 4000 |
|Sara| IT | 5000 | null |
|Kate| IT | 4500 | 5000 |
+----+---+------+-----------+
<a id="h-51-lead"></a>
51. lead()
Returns the value of a column from the next row.
# Get the next salary in each department
from pyspark.sql.functions import lead
df.withColumn("next_salary", lead("Salary", 1).over(windowSpec)).show()
Yields below the output.
# Output:
+----+---+------+-----------+
|Name|Dept|Salary|next_salary|
+----+---+------+-----------+
|Mike| HR | 4000 | 3000 |
|John| HR | 3000 | null |
|Sara| IT | 5000 | 4500 |
|Kate| IT | 4500 | null |
+----+---+------+-----------+
<a id="h-52-first-value"></a>
52. first_value()
Returns the first value in an ordered partition.
# Get the highest salary in each department
from pyspark.sql.functions import first
df.withColumn("first_salary", first("Salary").over(windowSpec)).show()
Yields below the output.
# Output:
+----+---+------+------------+
|Name|Dept|Salary|first_salary|
+----+---+------+------------+
|Mike| HR | 4000 | 4000 |
|John| HR | 3000 | 4000 |
|Sara| IT | 5000 | 5000 |
|Kate| IT | 4500 | 5000 |
+----+---+------+------------+
<a id="h-53-last-value"></a>
53. last_value()
Returns the last value in an ordered partition.
# Get the lowest salary in each department
from pyspark.sql.functions import last
df.withColumn("last_salary", last("Salary").over(windowSpec)).show()
Yields below the output.
# Output:
+----+---+------+-----------+
|Name|Dept|Salary|last_salary|
+----+---+------+-----------+
|Mike| HR | 4000 | 3000 |
|John| HR | 3000 | 3000 |
|Sara| IT | 5000 | 4500 |
|Kate| IT | 4500 | 4500 |
+----+---+------+-----------+
<a id="null-handling-functions"></a>
Null Handling Functions
Null handling functions detect, replace, and remove missing values from datasets. They ensure data quality by preventing null-related errors during processing and analysis.
<a id="h-54-fillna"></a>
54. fillna()
Replaces null values with a specified value in one or more columns.
# Replaces null values with a specified value
# Imports
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.master("local[1]") \
.appName("AIStudyLabs.com") \
.getOrCreate()
data = [
("James",None,"M"),
("Anna","NY","F"),
("Julia",None,None)
]
columns = ["name","state","gender"]
df = spark.createDataFrame(data,columns)
df.fillna("Unknown").show()
Yields below the output.
# Output:
+-----+-------+-------+
| name| state| gender|
+-----+-------+-------+
|James|Unknown| M |
| Anna| NY | F |
|Julia|Unknown|Unknown|
+-----+-------+-------+
<a id="h-55-dropna"></a>
55. dropna()
Removes rows containing null values.
# Remove rows with null values
df.dropna().show()
Yields below the output.
# Output:
+----+-----+------+
|name|state|gender|
+----+-----+------+
|Anna| NY| F|
+----+-----+------+
<a id="h-56-replace"></a>
56. replace()
Replaces specific values, including null, with another value.
# Replace null value with specifiec value
df.replace({None: "Not Specified"}).show()
Yields below the output.
# Output:
+-----+-------------+-------------+
| name| state| gender|
+-----+-------------+-------------+
|James|Not Specified| M|
| Anna| NY | F|
|Julia|Not Specified|Not Specified|
+-----+-------------+-------------+
<a id="h-57-na-fill"></a>
57. na.fill()
Fills null values (same as fillna()) but accessed via .na namespace.
# Fill null value using na.fill()
df.na.fill({"state": "Unknown", "gender": "Not Given"}).show()
Yields below the output.
# Output:
+-----+-------+---------+
| name| state| gender|
+-----+-------+---------+
|James|Unknown| M|
| Anna| NY | F|
|Julia|Unknown|Not Given|
+-----+-------+---------+
<a id="h-58-na-drop"></a>
58.na.drop()
Drops rows with null values (same as dropna()).
# Drop rows with null value
df.na.drop().show()
Yields below the output.
# Output:
+----+-----+------+
|name|state|gender|
+----+-----+------+
|Anna| NY| F|
+----+-----+------+
<a id="h-59-na-replace"></a>
59. na.replace()
Replaces null or other specific values using .na.replace().
# Replaces null or other specific values
df.na.replace("NY", "New York").show()
Yields below the output.
# Output:
+-----+--------+------+
| name| state|gender|
+-----+--------+------+
|James| null| M|
| Anna|New York| F|
|Julia| null| null|
+-----+--------+------+
<a id="array-and-map-functions"></a>
Array and Map functions
Array and map functionsmanipulate complex data types like arrays and maps. They allow creation, transformation, flattening, and element extraction, making them useful for working with nested or hierarchical data.
<a id="h-array-functions"></a>
Array Functions
<a id="h-60-explode"></a>
60. explode()
Converts array elements into multiple rows (one row per element)
# Convert array elements into multiple rows
# Create SparkSession and Prepare sample Data
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('pyspark-by-examples').getOrCreate()
from pyspark.sql.functions import explode
arrayData = [
('James',['Java','Scala'],{'hair':'black','eye':'brown'}),
('Michael',['Spark','Java',None],{'hair':'brown','eye':None}),
('Robert',['CSharp',''],{'hair':'red','eye':''}),
('Washington',None,None),
('Jefferson',['1','2'],{})
df = spark.createDataFrame(data=arrayData, schema = ['name','languages','properties'])
df.select(df.name,explode(df.languages)).show()
Yields below the output.
# Output:
+----------+--------+
| name|language|
+----------+--------+
| James| Java|
| James| Scala|
| Michael| Spark|
| Michael| Java|
| Michael| null|
| Robert| CSharp|
| Robert| |
| Jefferson| 1|
| Jefferson| 2|
+----------+--------+
<a id="h-61-explode-outer"></a>
61. explode_outer()
Similar to explode() but keeps rows even if the array is null or empty.
# explode_outer() on array
from pyspark.sql.functions import explode_outer
df.select(df.name,explode_outer(df.languages)).show()
Yields below the output.
# Output:
+----------+--------+
| name|language|
+----------+--------+
| James| Java|
| James| Scala|
| Michael| Spark|
| Michael| Java|
| Michael| null|
| Robert| CSharp|
| Robert| |
|Washington| null|
| Jefferson| 1|
| Jefferson| 2|
+----------+--------+
<a id="h-62-posexplode"></a>
62. posexplode()
Like explode(), but adds the element position (index) in the array.
# posexplode() on array
from pyspark.sql.functions import posexplode
df.select(df.name,posexplode(df.languages)).show()
Yields below the output.
# Output:
+----------+---+--------+
| name|pos|language|
+----------+---+--------+
| James| 0| Java|
| James| 1| Scala|
| Michael| 0| Spark|
| Michael| 1| Java|
| Michael| 2| null|
| Robert| 0| CSharp|
| Robert| 1| |
| Jefferson| 0| 1|
| Jefferson| 1| 2|
+----------+---+--------+
<a id="h-63-posexplode-outer"></a>
63. posexplode_outer()
Like posexplode() but keeps rows when the array is null.
# posexplode_outer() on array
from pyspark.sql.functions import posexplode_outer
df.select(df.name,explode_outer(df.languages)).show()
Yields below the output.
# Output:
+----------+----+--------+
| name| pos|language|
+----------+----+--------+
| James| 0| Java|
| James| 1| Scala|
| Michael| 0| Spark|
| Michael| 1| Java|
| Michael| 2| null|
| Robert| 0| CSharp|
| Robert| 1| |
|Washington|null| null|
| Jefferson| 0| 1|
| Jefferson| 1| 2|
+----------+----+--------+
<a id="h-map-functions"></a>
Map Functions
<a id="h-64-map-keys"></a>
64. map_keys()
Returns an array containing all keys from a map column.
# Return all keys using map_keys
from pyspark.sql.functions import map_keys
df.select(df.name,map_keys(df.properties)).show()
Yields below the output.
# Output:
+----------+--------------+
| name| keys|
+----------+--------------+
| James| [hair, eye] |
| Michael| [hair, eye] |
| Robert| [hair, eye] |
|Washington| null|
| Jefferson| []|
+----------+--------------+
<a id="h-65-map-values"></a>
65. map_values()
Returns an array containing all values from a map column.
# Return all keys using map_values
from pyspark.sql.functions import map_values
df.select(df.name,map_values(df.properties)).show()
Yields below the output.
# Output:
+----------+--------------+
| name| values|
+----------+--------------+
| James|[black, brown]|
| Michael|[brown, null] |
| Robert|[red, ] |
|Washington| null|
| Jefferson| []|
+----------+--------------+
<a id="h-66-element-at"></a>
66. element_at()
Returns the value for the given key in a map or index in an array (1-based index).
# Returns the value for the given key in a map
df.select("df.name", df.element_at("properties", "hair").show()
Yields below the output.
# Output:
+----------+----------+
| name| hair |
+----------+-----------+
| James| black|
| Michael| brown|
| Robert| red|
|Washington| null|
| Jefferson| null|
+----------+-----------+
<a id="joins-and-merging-functions"></a>
Joins and Merging Functions
Joins and merging functions combine two or more DataFrames based on matching keys or conditions. They support inner, outer, left, right, semi, and cross joins to integrate multiple datasets.
<a id="h-67-join"></a>
67. join()
Combine DataFrames using a given column and join type
# Join two DataFrames using join()
data1 = [("James", "Sales", 3000), ("Michael", "Sales", 4600), ("Robert", "IT", 4100)]
df1 = spark.createDataFrame(data1, ["name", "dept", "salary"])
data2 = [("Sales", "New York"), ("IT", "San Francisco")]
df2 = spark.createDataFrame(data2, ["dept", "location"])
df1.join(df2, on="dept", how="inner").show()
Yields below the output.
# Output:
+-----+-------+------+-------------+
| dept| name|salary| location|
+-----+-------+------+-------------+
|Sales| James| 3000| New York|
|Sales|Michael| 4600| New York|
| IT| Robert| 4100|San Francisco|
+-----+-------+------+-------------+
<a id="h-68-broadcast"></a>
68. broadcast()
Optimize join by broadcasting a small DataFrame.
# Optimize join by broadcasting
from pyspark.sql.functions import broadcast
df1.join(broadcast(df2), "dept").show()
Yields below the output.
# Output:
+-----+-------+------+-------------+
| dept| name|salary| location|
+-----+-------+------+-------------+
|Sales| James| 3000| New York|
|Sales|Michael| 4600| New York|
| IT| Robert| 4100|San Francisco|
+-----+-------+------+-------------+
<a id="h-69-union"></a>
69. union()
Combine two DataFrames with the same schema.
# Combine two DataFrames with the same schema
df1.union(df1).show()
Yields below the output.
# Output:
+-------+-----+------+
| name| dept|salary|
+-------+-----+------+
| James|Sales| 3000|
|Michael|Sales| 4600|
| Robert| IT| 4100|
| James|Sales| 3000|
|Michael|Sales| 4600|
| Robert| IT| 4100|
+-------+-----+------+
<a id="h-70-unionbyname"></a>
70. unionByName()
Combine DataFrames by matching column names.
# Combine DataFrames by matching column names.
df1.unionByName(df1).show()
Yields below the output.
# Output:
+-------+-----+------+
| name| dept|salary|
+-------+-----+------+
| James|Sales| 3000|
|Michael|Sales| 4600|
| Robert| IT| 4100|
| James|Sales| 3000|
|Michael|Sales| 4600|
| Robert| IT| 4100|
+-------+-----+------+
<a id="h-71-crossjoin"></a>
71. crossJoin()
Cartesian product of two DataFrames.
# Product of two DataFrames
df1.crossJoin(df2).show()
Yields below the output.
# Output:
+-------+-----+------+-------------+
| name| dept|salary| location|
+-------+-----+------+-------------+
| James|Sales| 3000| New York|
| James|Sales| 3000|San Francisco|
|Michael|Sales| 4600| New York|
|Michael|Sales| 4600|San Francisco|
| Robert| IT| 4100| New York|
| Robert| IT| 4100|San Francisco|
+-------+-----+------+-------------+
<a id="h-72-joinwith"></a>
72. joinWith()
Join a returning tuple of matching rows.
# Join a returning tuple of matching rows.
df1.joinWith(df2, df1.dept == df2.dept).show(truncate=False)
Yields below the output.
# Output:
+-------------------+------------------+
|_1 | _2 |
+-------------------+------------------+
|[James, Sales, 3000]|[Sales, New York] |
|[Michael, Sales, 4600]|[Sales, New York]|
|[Robert, IT, 4100] |[IT, San Francisco]|
+-------------------+------------------+
<a id="file-i-o-functions-in-pyspark"></a>
File I/O functions in PySpark
File I/O functions read and write data in different formats such as CSV, JSON, Parquet, ORC, and Delta. They enable smooth data exchange between PySpark and external storage systems.
<a id="h-73-write-csv"></a>
73. write.csv()
Saves DataFrame as CSV files.
# Write DataFrame to csv file
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("FileIOFunctions") \
.getOrCreate()
data = [("James", "Sales", 3000),
("Michael", "Sales", 4600),
("Robert", "IT", 4100)]
columns = ["Name", "Department", "Salary"]
df = spark.createDataFrame(data, columns)
df.write.csv("output/csv_data", header=True)
Yields below the output.
# Output:
part-00000-*.csv
Name,Department,Salary
James,Sales,3000
Michael,Sales,4600
Robert,IT,4100
<a id="h-74-read-csv"></a>
74. read.csv()
Reads CSV file into DataFrame.
# Read CSV file into DataFrame
df_csv = spark.read.csv("output/csv_data", header=True, inferSchema=True)
df_csv.show()
Yields below the output.
# Output:
+-------+----------+------+
| Name|Department|Salary|
+-------+----------+------+
| James| Sales| 3000|
|Michael| Sales| 4600|
| Robert| IT| 4100|
+-------+----------+------+
<a id="h-75-write-json"></a>
75. write.json()
Saves DataFrame as JSON files.
# Write DataFrame to JSON
df.write.json("output/json_data")
Yields below the output.
# Output:
part-00000-*.json
{"Name":"James","Department":"Sales","Salary":3000}
{"Name":"Michael","Department":"Sales","Salary":4600}
{"Name":"Robert","Department":"IT","Salary":4100}
<a id="h-76-read-json"></a>
76. read.json()
Reads JSON file into DataFrame.
# Reads JSON file into DataFrame
df_json = spark.read.json("output/json_data")
df_json.show()
Yields below the output.
# Output:
+----------+-------+------+
|Department| Name|Salary|
+----------+-------+------+
| Sales| James| 3000|
| Sales|Michael| 4600|
| IT| Robert| 4100|
+----------+-------+------+
<a id="h-77-write-parquet"></a>
77. write.parquet()
Saves DataFrame as Parquet files (columnar format).
# Write DataFrame to Paraquet file
df.write.parquet("output/parquet_data")
Yields below the output.
# Output:
part-00000-*.parquet
(Binary Parquet file)
<a id="h-78-read-parquet"></a>
78. read.parquet()
Reads Parquet file into DataFrame.
# Read Paraquet file into DataFrame
df_parquet = spark.read.parquet("output/parquet_data")
df_parquet.show()
Yields below the output.
# Output:
+-------+----------+------+
| Name|Department|Salary|
+-------+----------+------+
| James| Sales| 3000|
|Michael| Sales| 4600|
| Robert| IT| 4100|
+-------+----------+------+
<a id="h-79-write-format"></a>
79. write.format()
Saves DataFrame in a specified format (e.g., "orc").
# Saves DataFrame in a specified format
df.write.format("orc").save("output/orc_data")
Yields below the output.
# Output:
Folder output/orc_data with ORC files.
<a id="h-80-read-format"></a>
80. read.format()
Reads file in specified format.
# Reads file in specified format.
df_orc = spark.read.format("orc").load("output/orc_data")
df_orc.show()
Yields below the output.
# Output:
+-------+----------+------+
| Name|Department|Salary|
+-------+----------+------+
| James| Sales| 3000|
|Michael| Sales| 4600|
| Robert| IT| 4100|
+-------+----------+------+
<a id="h-81-saveastable"></a>
81. saveAsTable()
Saves DataFrame as a Hive table.
# Write DataFrame to a Hive table
df.write.saveAsTable("employee_table", mode="overwrite")
Yields below the output.
# Output:
Table employee_table created in Spark SQL metastore.
<a id="h-82-spark-sql"></a>
82. spark.sql()
Reads data using SQL query.
# Read data using SQL query
df_sql = spark.sql("SELECT * FROM employee_table")
df_sql.show()
Yields below the output.
# Output:
+-------+----------+------+
| Name|Department|Salary|
+-------+----------+------+
| James| Sales| 3000|
|Michael| Sales| 4600|
| Robert| IT| 4100|
+-------+----------+------+
<a id="udfs-and-performance-functions"></a>
UDFs and Performance Functions
UDF(User-Defined Function) and performance-related functions allow you to define custom transformations while optimizing PySpark execution. They are essential for implementing business-specific logic and improving job speed.
<a id="h-83-udf"></a>
83. udf()
Register a Python function as a User-Defined Function.
# Define user defined function
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import IntegerType, StringType
spark = SparkSession.builder.appName("PySparkTop100").getOrCreate()
data = [
("James", "Sales", 3000),
("Michael", "Sales", 4600),
("Robert", "IT", 4100),
("Maria", "IT", 3000),
("Jen", "Finance", 3900)
]
columns = ["Name", "Department", "Salary"]
df = spark.createDataFrame(data, columns)
df.show()
def add_bonus(salary):
return salary + 500
bonus_udf = F.udf(add_bonus, IntegerType())
df.withColumn("BonusSalary", bonus_udf("Salary")).show()
Yields below the output.
# Output:
+-------+----------+------+-----------+
| Name|Department|Salary|BonusSalary|
+-------+----------+------+-----------+
| James| Sales| 3000| 3500|
|Michael| Sales| 4600| 5100|
| Robert| IT| 4100| 4600|
| Maria| IT| 3000| 3500|
| Jen| Finance| 3900| 4400|
+-------+----------+------+-----------+
<a id="h-84-pandas-udf"></a>
84. pandas_udf()
Vectorized UDF for better performance using Pandas.
import pandas as pd
@F.pandas_udf(IntegerType())
def bonus_pandas(s: pd.Series) -> pd.Series:
return s + 1000
df.withColumn("BonusSalary", bonus_pandas("Salary")).show()
Yields below the output.
# Output:
+-------+----------+------+-----------+
| Name|Department|Salary|BonusSalary|
+-------+----------+------+-----------+
| James| Sales| 3000| 4000|
|Michael| Sales| 4600| 5600|
| Robert| IT| 4100| 5100|
| Maria| IT| 3000| 4000|
| Jen| Finance| 3900| 4900|
+-------+----------+------+-----------+
<a id="h-85-checkpoint"></a>
85. checkpoint()
Use .checkpoint() to save a DataFrame to a reliable storage location (like HDFS) and truncate the execution plan lineage, which helps in long/complex jobs.
from pyspark.sql import SparkSession
# Create SparkSession with checkpoint directory
spark = SparkSession.builder.appName("CheckpointExample").getOrCreate()
spark.sparkContext.setCheckpointDir("checkpoint_dir")
# Sample Data
data = [("Finance", 10), ("Marketing", 20), ("Sales", 30), ("IT", 40)]
df = spark.createDataFrame(data, ["Department", "ID"])
# Apply checkpoint
df_cp = df.checkpoint()
# Show checkpointed DataFrame
df_cp.show()
Yields below the output.
# Output:
+----------+---+
|Department| ID|
+----------+---+
| Finance| 10|
| Marketing| 20|
| Sales| 30|
| IT| 40|
+----------+---+
<a id="h-86-cache"></a>
86. cache()
Store DataFrame in memory for faster reuse.
df.cache()
print(df.count())
Yields below the output.
# Output:
5
<a id="h-87-persist"></a>
87. persist()
Store DataFrame with a specific storage level.
from pyspark import StorageLevel
df.persist(StorageLevel.MEMORY_AND_DISK)
print(df.count())
Yields below the output.
# Output:
5
<a id="h-88-unpersist"></a>
88. unpersist()
Remove DataFrame from cache.
df.unpersist()
Yields below the output.
# Output:
(No output – just releases memory.)
<a id="miscellaneous-utility-functions"></a>
Miscellaneous Utility Functions
Miscellaneous utility functions provide extra capabilities that don’t fit into other categories, such as generating random numbers, creating unique IDs, or obtaining descriptive statistics.
<a id="h-89-monotonically-increasing-id"></a>
89. monotonically_increasing_id()
Generates unique IDs for each row.
df.withColumn("ID", F.monotonically_increasing_id()).show()
Yields below the output.
# Output:
+-------+----------+------+-----------+
| Name|Department|Salary| ID|
+-------+----------+------+-----------+
| James| Sales| 3000| 0|
|Michael| Sales| 4600| 8589934592|
| Robert| IT| 4100| 17179869184|
| Maria| IT| 3000| 25769803776|
| Jen| Finance| 3900| 34359738368|
+-------+----------+------+-----------+
<a id="h-90-input-file-name"></a>
90. input_file_name()
Shows the file path of the source data.
file_df = spark.read.option("header", True).csv("employees.csv")
file_df.withColumn("FileName", F.input_file_name()).show()
Yields below the output.
# Output:
+-------+----------+------+----------------------+
| Name|Department|Salary| FileName |
+-------+----------+------+----------------------+
| James| Sales| 3000|file:/path/employees.csv|
|Michael| Sales| 4600|file:/path/employees.csv|
| Robert| IT| 4100|file:/path/employees.csv|
+-------+----------+------+----------------------+
<a id="h-91-current-date"></a>
91. current_date()
Get the current system date.
df.withColumn("Today", F.current_date()).show()
Yields below the output.
# Output:
+-------+----------+------+----------+
| Name|Department|Salary| Today|
+-------+----------+------+----------+
| James| Sales| 3000|2025-08-13|
|Michael| Sales| 4600|2025-08-13|
| Robert| IT| 4100|2025-08-13|
| Maria| IT| 3000|2025-08-13|
| Jen| Finance| 3900|2025-08-13|
+-------+----------+------+----------+
<a id="h-92-current-timestamp"></a>
92. current_timestamp()
Get current system timestamp.
df.withColumn("Now", F.current_timestamp()).show(truncate=False)
Yields below the output.
# Output:
+-------+----------+------+-----------------------+
|Name |Department|Salary|Now |
+-------+----------+------+-----------------------+
|James |Sales |3000 |2025-08-13 16:52:12.123 |
|Michael|Sales |4600 |2025-08-13 16:52:12.123 |
|Robert |IT |4100 |2025-08-13 16:52:12.123 |
|Maria |IT |3000 |2025-08-13 16:52:12.123 |
|Jen |Finance |3900 |2025-08-13 16:52:12.123 |
+-------+----------+------+-----------------------+
<a id="h-93-lit"></a>
93. lit()
Add a constant column.
df.withColumn("Country", F.lit("USA")).show()
Yields below the output.
# Output:
+-------+----------+------+-------+
| Name|Department|Salary|Country|
+-------+----------+------+-------+
| James| Sales| 3000| USA|
|Michael| Sales| 4600| USA|
| Robert| IT| 4100| USA|
| Maria| IT| 3000| USA|
| Jen| Finance| 3900| USA|
+-------+----------+------+-------+
<a id="h-94-rand"></a>
94. rand()
Generate a random number column.
df.withColumn("Random", F.rand()).show()
Yields below the output.
# Output:
+-------+----------+------+-------------------+
| Name|Department|Salary| Random|
+-------+----------+------+-------------------+
| James| Sales| 3000| 0.7273891240039923 |
|Michael| Sales| 4600|0.08921341169330965 |
| Robert| IT| 4100|0.6675100482888047 |
| Maria| IT| 3000| 0.3145293719292226 |
| Jen| Finance| 3900| 0.9821121846487522 |
+-------+----------+------+-------------------+
<a id="h-95-isempty"></a>
95. isEmpty()
Checks if a DataFrame contains no rows.
data = []
df = spark.createDataFrame(data, "name STRING, age INT")
print(df.rdd.isEmpty())
Yields below the output.
# Output:
True
<a id="h-96-randomsplit"></a>
96. randomSplit()
Splits a DataFrame into multiple parts based on given weights.
data = [("A",), ("B",), ("C",), ("D",)]
df = spark.createDataFrame(data, ["Letter"])
train, test = df.randomSplit([0.75, 0.25], seed=42)
print("Train count:", train.count())
print("Test count:", test.count())
Yields below the output.
# Output:
Train count: 3
Test count: 1
<a id="h-97-sample-take-random-sample"></a>
97. sample() – Take Random Sample
Randomly selects a fraction of rows.
data = [(i,) for i in range(1, 11)]
df = spark.createDataFrame(data, ["Number"])
df.sample(withReplacement=False, fraction=0.3, seed=1).show()
Yields below the output.
# Output:
+------+
|Number|
+------+
| 3|
| 8|
| 10|
+------+
<a id="h-98-approxquantile-approximate-quantiles"></a>
98. approxQuantile() – Approximate Quantiles
Computes approximate quantiles for a column.
data = [(10,), (20,), (30,), (40,), (50,)]
df = spark.createDataFrame(data, ["Value"])
result = df.approxQuantile("Value", [0.5], 0.01)
print(result)
Yields below the output.
# Output:
[30.0]
<a id="h-99-distinct-remove-duplicate-rows"></a>
99. distinct() – Remove Duplicate Rows
Returns a DataFrame with distinct rows.
data = [("A", 1), ("A", 1), ("B", 2)]
df = spark.createDataFrame(data, ["Col1", "Col2"])
df.distinct().show()
Yields below the output.
# Output:
+----+----+
|Col1|Col2|
+----+----+
| A| 1|
| B| 2|
+----+----+
<a id="h-100-printschema-display-schema"></a>
100. printSchema() – Display Schema
Displays the schema of a DataFrame in a tree format.
df = spark.createDataFrame([("Alice", 25)], ["name", "age"])
df.printSchema()
Yields below the output.
# Output:
root
|-- name: string (nullable = true)
|-- age: integer (nullable = true)
Conclusion
Mastering PySpark is not just about knowing how to write code—it’s about understanding the right function to use at the right time. This curated list of 100 essential PySpark functions gives you a strong foundation to tackle both interview questions and real-world data engineering challenges.
From basic DataFrame operations to complex window functions, from string cleaning to schema manipulation, these functions cover the most common—and most powerful—tools available in PySpark.
Keep practicing these functions with real datasets, experiment with different parameters, and explore PySpark’s official documentation to go even deeper. The more you use these functions in hands-on projects, the more confident and efficient you’ll become as a data engineer.
Remember: Interviews test not just your memory, but your ability to apply the right PySpark function to solve problems efficiently. With these 100 functions at your fingertips, you’ll be well-prepared to impress interviewers and excel in your career.
Happy Learning!!