In PySpark, the split() function is commonly used to split string columns into multiple parts based on a delimiter or a regular expression. It is available in pyspark.sql.functions and and is widely used for text processing.
What makes PySpark split() powerful is that it converts a string column into an array column, making it easy to extract specific elements or expand them into multiple columns for further analysis.
In this article, we’ll explore a step-by-step guide to split string columns in PySpark DataFrame using the split() function with the delimiter, regex, and limit parameters.
Key Points-
- PySpark split() is used to split string columns into arrays or multiple columns.
- It is available in the
pyspark.sql.functionsmodule and must be imported before use. - The function returns a column of array type, where each element represents a part of the split string.
- Supports both delimiters and regex.
- Use array indexing to extract individual parts.
- The limit parameter helps control the number of splits.
- Individual elements from the split array can be accessed using indexing with
getItem(index). - The split array can be expanded into multiple structured columns using
withColumn(). - It is commonly used for parsing names, addresses, dates, or CSV-like string columns into structured fields.
- The function is optimized for distributed processing, making it suitable for very large datasets.
split()works seamlessly with other PySpark functions, such assize(),explode(), orconcat_ws(), to further manipulate split results.
<a id="h-pyspark-split-function"></a>
PySpark split() Function
The split() function is used to split a string column into an array of substrings based on a given delimiter or regular expression and return an array column. You can use the limit parameter to handle the number of splits.
<a id="h-syntax"></a>
Syntax
Following is the syntax of the split() function.
# Syntax of the split()
pyspark.sql.functions.split(str, pattern, limit=-1)
<a id="h-parameters"></a>
Parameters
str:Column or string expression to split.pattern:Delimiter or regex pattern (e.g.," ",",","-","[.]").limit:(optional) Maximum number of splits:-1:No limit (default).n:Split into at mostnparts.
<a id="h-return-value"></a>
Return Value
Returns a Column of type Array(StringType), where each element is a substring.
<a id="h-pyspark-split-dataframe-string-column-by-delimiter"></a>
PySpark Split DataFrame String Column by Delimiter
You can use split() to split a string column into an array of substrings using a specified delimiter. To create a new column with these substrings, pass the desired column name and the existing string column (along with the delimiter) into the withColumn() function, which generates a new column of array type containing the split elements.
# Split DataFrame column by delimiter "-"
from pyspark.sql import SparkSession
from pyspark.sql.functions import split, size, concat_ws, col
# Create SparkSession
spark = SparkSession.builder.appName("aistudylabs").getOrCreate()
# Sample Data
data = [
('James','','Smith','1991-04-01'),
('Michael','Rose','','2000-05-19'),
('Robert','','Williams','1978-09-05'),
('Maria','Anne','Jones','1967-12-01'),
('Jen','Mary','Brown','1980-02-17')
]
columns = ["firstname","middlename","lastname","dob"]
df = spark.createDataFrame(data, columns)
print("Original DataFrame:")
df.show(truncate=False)
# Split dob by delimiter "-"
df1 = df.withColumn("dob_array", split(df["dob"], "-"))
print("Split dob into array by '-' :")
df1.show(truncate=False)
Yields below the output.
Here, the string was split into an array column.
<a id="h-pyspark-split-column-into-multiple-columns"></a>
PySpark Split Column into Multiple Columns
You can also use split() to split a string column into multiple columns by specifying the index of each element in the split array. With the help of withColumn(), you can extract specific parts of the split string and create individual structured columns, effectively converting one string column into multiple meaningful columns.
# Split dob into multiple columns (year, month, day)
df2 = df.withColumn("year", split(df["dob"], "-")[0]) \
.withColumn("month", split(df["dob"], "-")[1]) \
.withColumn("day", split(df["dob"], "-")[2])
print("Split dob into year, month, day :")
df2.show(truncate=False)
Yields below the output.
# Output:
Split dob into year, month, day :
+---------+----------+--------+----------+----+-----+---+
|firstname|middlename|lastname|dob |year|month|day|
+---------+----------+--------+----------+----+-----+---+
|James | |Smith |1991-04-01|1991|04 |01 |
|Michael |Rose | |2000-05-19|2000|05 |19 |
|Robert | |Williams|1978-09-05|1978|09 |05 |
|Maria |Anne |Jones |1967-12-01|1967|12 |01 |
|Jen |Mary |Brown |1980-02-17|1980|02 |17 |
+---------+----------+--------+----------+----+-----+---+
This is how you can split a column into multiple columns in PySpark.
<a id="h-pyspark-split-with-limit-parameter"></a>
PySpark split with limit parameter
Optionally use the limit parameter to control the number of splits the limit parameter in split() to divide a string column by a delimiter with a specified maximum number of splits.
# Split dob with limit = 2
df_limit2 = df.withColumn("dob_split_limit2", split(df["dob"], "-", 2))
print("Split dob with limit = 2:")
df_limit2.show(truncate=False)
# Split dob with limit = -1 (no limit)
df_limit_neg = df.withColumn("dob_split_limit_neg", split(df["dob"], "-", -1))
print("Split dob with limit = -1 (no limit):")
df_limit_neg.show(truncate=False)
Yields below the output.
# Output:
Split dob with limit = 2:
+--------+----------+--------+----------+------------------+
|firstname|middlename|lastname|dob |dob_split_limit2 |
+--------+----------+--------+----------+------------------+
|James | |Smith |1991-04-01|[1991, 04-01] |
|Michael |Rose | |2000-05-19|[2000, 05-19] |
|Robert | |Williams|1978-09-05|[1978, 09-05] |
|Maria |Anne |Jones |1967-12-01|[1967, 12-01] |
|Jen |Mary |Brown |1980-02-17|[1980, 02-17] |
+--------+----------+--------+----------+------------------+
Split dob with limit = -1 (no limit):
+--------+----------+--------+----------+-------------------------+
|firstname|middlename|lastname|dob |dob_split_limit_neg |
+--------+----------+--------+----------+-------------------------+
|James | |Smith |1991-04-01|[1991, 04, 01] |
|Michael |Rose | |2000-05-19|[2000, 05, 19] |
|Robert | |Williams|1978-09-05|[1978, 09, 05] |
|Maria |Anne |Jones |1967-12-01|[1967, 12, 01] |
|Jen |Mary |Brown |1980-02-17|[1980, 02, 17] |
+--------+----------+--------+----------+-------------------------+
Here, the string is split only once, meaning the rest of the string stays intact.
<a id="h-pyspark-split-string-into-character"></a>
PySpark Split String into Character
To split a string into individual characters by passing an empty string "" as the delimiter of this function.
# Split string by character (example: split firstname into characters)
df3 = df.withColumn("fname_chars", split(df["firstname"], ""))
print("Split firstname into characters:")
df3.show(truncate=False)
Yields below the output.
# Output:
Split firstname into characters:
+---------+----------+--------+----------+---------------------+
|firstname|middlename|lastname|dob |fname_chars |
+---------+----------+--------+----------+---------------------+
|James | |Smith |1991-04-01|[J, a, m, e, s] |
|Michael |Rose | |2000-05-19|[M, i, c, h, a, e, l]|
|Robert | |Williams|1978-09-05|[R, o, b, e, r, t] |
|Maria |Anne |Jones |1967-12-01|[M, a, r, i, a] |
|Jen |Mary |Brown |1980-02-17|[J, e, n] |
+---------+----------+--------+----------+---------------------+
<a id="h-pyspark-split-get-first-element-from-array"></a>
PySpark Split Get First Element from Array
By indexing with [0], you can quickly extract the first item from the split array. Commonly used for extracting the first name or year from a string.
# Split dob and get first item (year)
df4 = df.withColumn("year", split(df["dob"], "-")[0])
print("Get first item (year) from dob:")
df4.show(truncate=False)
Yields below the output.
# Output:
Get first item (year) from dob:
+---------+----------+--------+----------+----+
|firstname|middlename|lastname|dob |year|
+---------+----------+--------+----------+----+
|James | |Smith |1991-04-01|1991|
|Michael |Rose | |2000-05-19|2000|
|Robert | |Williams|1978-09-05|1978|
|Maria |Anne |Jones |1967-12-01|1967|
|Jen |Mary |Brown |1980-02-17|1980|
+---------+----------+--------+----------+----+
<a id="h-pyspark-split-get-last-element-from-array"></a>
PySpark Split Get Last Element from Array
Using size(split()) - 1 gives you the last element of the split array. This is often used to extract the day from a date, or the last name in a full name string.
# Split dob and get last item (day)
df5 = df.withColumn("day", split(df["dob"], "-")[size(split(df["dob"], "-")) - 1])
print("Get last item (day) from dob:")
df5.show(truncate=False)
Yields below the output.
# Output:
Get last item (day) from dob:
+---------+----------+--------+----------+---+
|firstname|middlename|lastname|dob |day|
+---------+----------+--------+----------+---+
|James | |Smith |1991-04-01|01 |
|Michael |Rose | |2000-05-19|19 |
|Robert | |Williams|1978-09-05|05 |
|Maria |Anne |Jones |1967-12-01|01 |
|Jen |Mary |Brown |1980-02-17|17 |
+---------+----------+--------+----------+---+
<a id="h-pyspark-split-string-into-array"></a>
PySpark Split String into Array
You can first create a combined string (for example, a full_name) and then apply split() to break it into an array of words. If you need to concat the split elements back into a single string, you can use the concat_ws() function.
# Split String into Array
df7 = df.withColumn("full_name", concat_ws(" ", "firstname", "middlename", "lastname")) \
.withColumn("name_parts", split(concat_ws(" ", "firstname", "middlename", "lastname"), " "))
print("Split full_name into parts:")
df7.show(truncate=False)
Yields below the output.
# Output:
Split full_name into parts:
+---------+----------+--------+----------+----------------+--------------------+
|firstname|middlename|lastname|dob |full_name |name_parts |
+---------+----------+--------+----------+----------------+--------------------+
|James | |Smith |1991-04-01|James Smith |[James, , Smith] |
|Michael |Rose | |2000-05-19|Michael Rose |[Michael, Rose, ] |
|Robert | |Williams|1978-09-05|Robert Williams|[Robert, , Williams]|
|Maria |Anne |Jones |1967-12-01|Maria Anne Jones|[Maria, Anne, Jones]|
|Jen |Mary |Brown |1980-02-17|Jen Mary Brown |[Jen, Mary, Brown] |
+---------+----------+--------+----------+----------------+--------------------+
<a id="h-pyspark-split-dataframe-into-chunks-row-based-not-split"></a>
PySpark Split DataFrame into Chunks (Row-based, not split())
To handle splitting a DataFrame into chunks, you can divide a DataFrame into smaller row-based chunks in two ways:
- By using
zipWithIndex()and assigning chunk IDs for equal splits. - By using randomSplit() when you want random partitions based on ratios.This is useful for parallel processing or sampling data in smaller sets.
# PySpark Split DataFrame into Chunks
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, floor
# Create Spark session
spark = SparkSession.builder.appName("aistudylabs").getOrCreate()
# Sample DataFrame
data = [
('James','','Smith','1991-04-01'),
('Michael','Rose','','2000-05-19'),
('Robert','','Williams','1978-09-05'),
('Maria','Anne','Jones','1967-12-01'),
('Jen','Mary','Brown','1980-02-17')
]
columns = ["firstname","middlename","lastname","dob"]
df = spark.createDataFrame(data, columns)
print("Original DataFrame:")
df.show(truncate=False)
# -----------------------
# Method 1: Split into equal chunks using zipWithIndex()
# -----------------------
# Add row index
df_indexed = df.rdd.zipWithIndex().toDF(["data", "row_id"])
# Flatten back to columns + row_id
for col_name in df.columns:
df_indexed = df_indexed.withColumn(col_name, col("data").getItem(col_name))
df_indexed = df_indexed.drop("data")
# Number of chunks
num_chunks = 2
chunk_size = (df_indexed.count() + num_chunks - 1) // num_chunks
# Assign chunk number
df_chunks = df_indexed.withColumn("chunk_id", floor(col("row_id") / chunk_size))
print("DataFrame with chunk IDs:")
df_chunks.show(truncate=False)
# Collect chunks separately
chunk_dfs = [df_chunks.filter(col("chunk_id") == i).drop("row_id", "chunk_id") for i in range(num_chunks)]
print("Chunk 1:")
chunk_dfs[0].show(truncate=False)
print("Chunk 2:")
chunk_dfs[1].show(truncate=False)
# -----------------------
# Method 2: Random Split into chunks
# -----------------------
df1, df2 = df.randomSplit([0.5, 0.5], seed=42)
print("Random Split - Chunk 1:")
df1.show(truncate=False)
print("Random Split - Chunk 2:")
df2.show(truncate=False)
Yields below the output.
# Output:
Original DataFrame:
+---------+----------+--------+----------+
|firstname|middlename|lastname|dob |
+---------+----------+--------+----------+
|James | |Smith |1991-04-01|
|Michael |Rose | |2000-05-19|
|Robert | |Williams|1978-09-05|
|Maria |Anne |Jones |1967-12-01|
|Jen |Mary |Brown |1980-02-17|
+---------+----------+--------+----------+
DataFrame with chunk IDs:
+------+---------+----------+--------+----------+--------+
|row_id|firstname|middlename|lastname|dob |chunk_id|
+------+---------+----------+--------+----------+--------+
|0 |James | |Smith |1991-04-01|0 |
|1 |Michael |Rose | |2000-05-19|0 |
|2 |Robert | |Williams|1978-09-05|0 |
|3 |Maria |Anne |Jones |1967-12-01|1 |
|4 |Jen |Mary |Brown |1980-02-17|1 |
+------+---------+----------+--------+----------+--------+
Chunk 1:
+---------+----------+--------+----------+
|firstname|middlename|lastname|dob |
+---------+----------+--------+----------+
|James | |Smith |1991-04-01|
|Michael |Rose | |2000-05-19|
|Robert | |Williams|1978-09-05|
+---------+----------+--------+----------+
Chunk 2:
+---------+----------+--------+----------+
|firstname|middlename|lastname|dob |
+---------+----------+--------+----------+
|Maria |Anne |Jones |1967-12-01|
|Jen |Mary |Brown |1980-02-17|
+---------+----------+--------+----------+
Random Split - Chunk 1:
+---------+----------+--------+----------+
|firstname|middlename|lastname|dob |
+---------+----------+--------+----------+
|Maria |Anne |Jones |1967-12-01|
+---------+----------+--------+----------+
Random Split - Chunk 2:
+---------+----------+--------+----------+
|firstname|middlename|lastname|dob |
+---------+----------+--------+----------+
|James | |Smith |1991-04-01|
|Michael |Rose | |2000-05-19|
|Robert | |Williams|1978-09-05|
|Jen |Mary |Brown |1980-02-17|
+---------+----------+--------+----------+
<a id="h-frequently-asked-questions-of-pyspark-split"></a>
Frequently Asked Questions of PySpark split()
The split() function is used to split a string column into an array of substrings based on a given delimiter or regular expression.
It takes a column containing string values, applies the specified delimiter or regex, and returns an array column.
The main parameters are the column to split, the delimiter or regex pattern, and an optional limit to control the number of splits.
It returns an array column where each element is a substring derived from the original string.
You can access array elements by their index, such as the first or last element, depending on your requirement.
You use split() when you need structured parts of a string as separate elements rather than a single transformed string.
It supports both fixed delimiters (like a comma or dash) and regular expressions for flexible splitting.
<a id="h-conclusion"></a>
Conclusion
In this article, we explored the PySpark split() function, which is widely used to process string columns.
- It converts strings into arrays using a delimiter or regex.
- You can then extract specific items, expand arrays into multiple columns, or even split into characters.
- Common use cases include splitting by delimiters, getting the first/last element, and transforming strings into arrays for advanced processing.
With these techniques, you can easily handle complex string operations in PySpark DataFrames.
Happy Learning!!
<a id="h-reference"></a>