Spark with AWS Glue: Getting Started Guide

Apache Spark and AWS Glue make a practical pair for data processing and analytics. This tutorial is a walkthrough for newcomers to AWS who want to use Spark with AWS Glue. You will set up the AWS pieces, create a Glue job with PySpark, and run the code against sample sales data. The data is dummy data.

By the end, you will have a working glue job with pyspark that reads raw CSV from S3, cleans and aggregates it, and writes the results back to S3 as Parquet. The article also covers the parts that usually cause trouble: IAM permissions, schema handling, output overwrites and cost.

How AWS Glue runs Spark

AWS Glue is a serverless data integration service. When you run a glue job with pyspark, the service provisions the workers, runs your script, and releases the resources when the job finishes. You do not manage EC2 instances or install Spark yourself. Glue charges for the resources a glue job with pyspark uses, measured in Data Processing Units (DPUs), rather than for idle capacity.

Your code is ordinary PySpark, so a glue job with pyspark looks much like a PySpark job you would run on your own cluster. The SparkSession, DataFrame operations and SQL functions in this tutorial work the same way when you use Spark with AWS Glue. What Glue adds is the layer around the code: a data catalog of table definitions, connectors to S3 and databases, job bookmarks, scheduling through triggers, and the job monitoring in the console.

This matters when something fails in Spark with AWS Glue. The cause is often in the Glue layer rather than in your transformations. A missing IAM permission, a wrong bucket path or a job role without S3 access stops the job before your Spark code runs. The troubleshooting section below covers those cases for Spark with AWS Glue.

Prerequisites:

To follow this tutorial, you will need the following to run Spark with AWS Glue:

An AWS account: Sign up for an AWS account at https://aws.amazon.com/.
AWS Glue permissions: Ensure that your AWS account has the necessary permissions to create and run AWS Glue jobs. If you are using an existing account, make sure you have appropriate permissions or consult your AWS account administrator.
An IAM role for Glue: The glue job with pyspark needs a role that can read the input folder and write the output folder. Create a role with the AWSGlueServiceRole managed policy, and add S3 access to your bucket.
Sample dataset: You can use any dataset of your choice, but for this tutorial, we will use this SalesData.csv (here is how the data looks like):

Date,Salesperson,Lead Name,Segment,Region,Target Close,Forecasted Monthly Revenue,Opportunity Stage,Weighted Revenue,Closed Opportunity,Active Opportunity,Latest Status Entry
1/2/2011,Gerri Hinds,US_SMB_1317,SMB,US,2/2/2011,103699,Lead,10370,FALSE,FALSE,FALSE
1/3/2011,David King,EMEA_Enterprise_405,Enterprise,EMEA,4/9/2011,393841,Lead,39384,FALSE,FALSE,FALSE
1/6/2011,James Swanger,US_Enterprise_1466,Enterprise,US,5/4/2011,326384,Lead,32638,FALSE,FALSE,FALSE
1/11/2011,Gerri Hinds,US_SMB_2291,SMB,US,2/14/2011,76316,Lead,7632,FALSE,FALSE,FALSE

Spark with AWS Glue step-by-step Tutorial

Step 1: Set up an S3 Bucket:

Create an S3 bucket to store your sample data and the artifacts for your Spark with AWS Glue job. Navigate to the S3 service in the AWS Management Console and click on "Create bucket." Provide a unique name and select your desired region.
Once the bucket is created, create two sub-folders named:
cleaned_data
raw_data

Step 2: Prepare the Sample Data:

Upload the SalesData.csv file to your S3 bucket by clicking on your bucket name under the raw_data sub-folder, then "Upload." The Spark with AWS Glue job reads from this folder, so the file must be here.

Step 3: Create a Glue Job:

Log in to the AWS Management Console and navigate to the AWS Glue service.
In the AWS Glue console, select "ETL Jobs" in the left-hand menu, then select "Spark script editor" and click on "Create".
Provide a name for the glue job with pyspark at the top.
Write the Spark (PySpark) code for your Spark with AWS Glue job. In this application, we perform the following operations:
Create a Spark session
Read the data from the SalesData.csv file, which is stored in our S3 bucket under the raw_data sub-folder, and perform the following transformations:
Conversion of Date: The "Date" column is converted from string type to date type using the to_date function. This enables easier date-based analysis and operations.
Total Sales by Salesperson: The code calculates the total sales by salesperson by grouping the DataFrame by "Salesperson" and calculating the sum of the "Forecasted_Monthly_Revenue" column.
Average Revenue by Opportunity Stage: The code calculates the average revenue per opportunity stage by grouping the DataFrame by "Opportunity_Stage" and calculating the average of the "Weighted_Revenue" column.
Filtering Closed Opportunities: The code filters the DataFrame to include only closed opportunities by selecting rows where the "Closed_Opportunity" column is set to True.
Selection of Specific Columns: The code selects specific columns from the DataFrame to include in the cleaned dataset. Adjust column names as needed.
Once everything is done, the glue job with pyspark saves all the results to our S3 bucket under the cleaned_data sub-folder.

Step 4: Example of Spark with AWS Glue

The example below is a complete Spark with AWS Glue job. Remember to replace YOUR_BUCKET_LOCATION_OF_RAW_DATA and YOUR_BUCKET_LOCATION_OF_CLEANED_DATA with your S3 bucket locations. Feel free to modify the transformations or add additional ones based on your specific needs.

# Examples of Spark with AWS Glue

# Imports
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import DateType

S3_INPUT_DATA = 's3://mybuckect-spark-by-example/raw_data/'
S3_OUTPUT_DATA = 's3://mybuckect-spark-by-example/cleaned_data/'

def main():

    # Creating the SparkSession
    spark = SparkSession.builder.appName("My Demo ETL App").getOrCreate()
    spark.sparkContext.setLogLevel('ERROR')

    # Spark DataFrame (Raw) - Transformation
    df = (
        spark.read.option("Header", True)
        .option("InferSchema", True)
        .csv(S3_INPUT_DATA)
    )

    # Map column names that contain spaces to underscore names
    replacements = {c:c.replace(' ','_') for c in df.columns if ' ' in c}

    # Rename the columns that contain spaces
    df = df.select(
        [F.col(c).alias(replacements.get(c, c)) for c in df.columns]
    )

    # Convert the "Date" column from string type to date type
    df = df.withColumn("Date", F.to_date(F.col("Date"), "M/d/yyyy"))

    # Calculate the total sales by salesperson
    sales_by_salesperson = df.groupBy("Salesperson").agg(
        F.sum("Forecasted_Monthly_Revenue").alias("Total_Sales")
    )

    # Calculate the average revenue per opportunity stage
    avg_revenue_by_stage = df.groupBy("Opportunity_Stage").agg(
        F.avg("Weighted_Revenue").alias("Avg_Revenue")
    )

    # Filter the dataset to include only closed opportunities
    closed_opportunities = df.filter(F.col("Closed_Opportunity") == True)

    # Select specific columns for the cleaned dataset
    cleaned_df = df.select(
        "Date",
        "Salesperson",
        "Segment",
        "Region",
        "Opportunity_Stage",
        "Weighted_Revenue",
    )

    # Print the total number of records in the cleaned dataset
    print(f"Total no. of records in the cleaned dataset is: {cleaned_df.count()}")

    try:
        # Save the DataFrames under different folders within the S3_OUTPUT_DATA bucket
        sales_by_salesperson.write.mode("overwrite").parquet(
            S3_OUTPUT_DATA + "/sales_by_salesperson"
        )
        avg_revenue_by_stage.write.mode("overwrite").parquet(
            S3_OUTPUT_DATA + "/avg_revenue_by_stage"
        )
        closed_opportunities.write.mode("overwrite").parquet(
            S3_OUTPUT_DATA + "/closed_opportunities"
        )
        cleaned_df.write.mode("overwrite").parquet(
            S3_OUTPUT_DATA + "/cleaned_df"
        )
        print('The cleaned data is uploaded')
    except:
        print('Something went wrong, please check the logs :P')

if __name__ == '__main__':
    main()

Save the Spark with AWS Glue job and click "Run".
Monitor the glue job with pyspark in the AWS Glue console. You can check the job logs and progress to ensure the data processing is successful.

The glue job with pyspark prints the record count first, then confirms the upload. For the four sample rows above, the log looks like this. Your count depends on the file you load, so treat this as an example.

Total no. of records in the cleaned dataset is: 4
The cleaned data is uploaded

Step 5: Verify the Spark with AWS Glue Results

Navigate to the S3 bucket where you specified the job output path. You should see four folders under cleaned_data, one for each Parquet output: sales_by_salesperson, avg_revenue_by_stage, closed_opportunities and cleaned_df. The record count in the Spark with AWS Glue job log should match the number of rows you loaded.

Congratulations! You have successfully processed data using Spark with AWS Glue.

Validating the output

Checking the job log of your Spark with AWS Glue run is a start, but it does not show whether the values are right. Open a notebook or a second glue job with pyspark, and read one of the Parquet folders back:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
totals = spark.read.parquet(
    "s3://mybuckect-spark-by-example/cleaned_data/sales_by_salesperson"
)
totals.printSchema()
totals.show(truncate=False)

The schema from your Spark with AWS Glue output should list Salesperson as a string and Total_Sales as a number. Then compare one total by hand against the CSV. Pick a salesperson who appears in two or three rows, add their Forecasted Monthly Revenue values yourself, and check the figure matches. A mismatch usually means a column was read as a string, so the sum skipped it, or the file was read twice. Fixing either one early saves a lot of confusion later.

Keep one more habit in mind as you extend this Spark with AWS Glue job. Every time you add a transformation, check the schema again with printSchema(). A column that changes type silently, such as a date that becomes a string after a join, is the most common reason a Spark with AWS Glue job produces plausible but wrong numbers.

Troubleshooting common failures

AccessDenied when reading the input

The glue job with pyspark role needs s3:GetObject on the raw_data folder and s3:ListBucket on the bucket. Check the role in the glue job with pyspark details, then compare its policy with the bucket path in S3_INPUT_DATA. A typo in the bucket name, or a missing trailing slash on the folder, can produce the same error. A glue job with pyspark that fails this way has not even started reading your data, so check the permissions before you touch the code.

Column names with spaces

Headers such as "Lead Name" and "Forecasted Monthly Revenue" contain spaces. Spark can read them if you wrap each name in backticks, but the names get awkward in every later expression. The replacements dictionary in Step 4 renames them to underscores before any transformation runs, which is why the rest of the code uses names like Forecasted_Monthly_Revenue.

InferSchema slows large jobs

InferSchema reads the data once to guess the column types. That is convenient on a small file, but on a large dataset it adds a full pass before your transformations start. For production jobs, define the schema with StructType and pass it to the reader, so Spark skips the guessing step. The same change speeds up any glue job with pyspark that runs on a schedule.

The output folder already exists

The mode('overwrite') setting replaces the Parquet files in each output folder on every run, so a rerun does not fail. Be careful pointing the output at a folder that holds data you want to keep, because the overwrite removes what was there.

Choosing between AWS Glue and Amazon EMR

Spark with AWS Glue suits scheduled ETL where you want less infrastructure to manage and a data catalog that other services can share. Amazon EMR suits long-running clusters, custom Spark versions and jobs that need fine control over the cluster itself. If your job starts quickly, runs on a schedule and writes to S3, Glue is usually the simpler choice. If many analysts run notebooks against the same cluster all day, EMR may fit better. This tutorial uses Spark with AWS Glue, but the PySpark logic would run on EMR with changes to the job setup and the paths.

Cost and performance tips

Glue bills for the resources a glue job with pyspark uses, so small habits make a difference:

Filter early. Apply filters and column selections before joins and aggregations, so Spark moves less data between workers.
Write Parquet, not CSV. Parquet is columnar and compressed, so later queries read fewer bytes than they would from CSV, which helps every Spark with AWS Glue output you write.
Partition large outputs. Partition by a column you filter on, such as Region, so later queries can skip folders.
Avoid collect() on large DataFrames. It pulls every row into the driver. Use show(), limit() or a write to S3 instead.
Set a job timeout. A glue job with pyspark that is stuck on bad input should stop instead of running for hours.

Reading the job logs

The Glue console shows two log streams for each glue job with pyspark run: the driver output and the Spark executor logs. The line that reports the record count is useful even when nothing has failed, since a count of zero usually means the input path is wrong or the filter removed everything.

When the glue job with pyspark fails, search the driver log for the first Python exception rather than the last one. Spark often prints a long chain of Java errors after the real cause, and the first traceback names the line in your script that broke. In the code above, the try block catches every exception and prints a generic message, so the traceback is hidden. While you debug, remove the bare except, or print the exception, so the real error is visible in the log.

Executor logs help when the glue job with pyspark is slow or runs out of memory. A stage that keeps retrying on a few tasks often points to skewed data, where one key holds most of the rows. Grouping by a column with very few distinct values, such as Region in this dataset, is a common cause.

What the output folders are for

Each folder under cleaned_data answers a different question for the Spark with AWS Glue job, which is why it writes four outputs rather than one:

sales_by_salesperson gives one row per salesperson with their total forecasted revenue. It is the table a sales manager would open first.
avg_revenue_by_stage shows the average weighted revenue at each stage of the pipeline, which helps compare how much a typical lead is worth at each step.
closed_opportunities keeps only the rows that closed, so you can measure the revenue that was actually won.
cleaned_df is the full cleaned dataset with the selected columns, the base for any further analysis.

Keeping them separate makes reruns of your Spark with AWS Glue job cheaper to reason about.

Conclusion

This tutorial walked through getting started with Spark with AWS Glue. You created the S3 layout, wrote a glue job with pyspark, ran the transformations, and wrote the results to S3 as Parquet. You also checked one result by hand and saw the common failures before they happened to you.

For a next step, read the AWS Glue PySpark extensions reference and learn about job bookmarks, which stop a Glue job from reprocessing files it has already read, and about scheduled triggers, which run the job without manual clicks. The same structure works for any CSV you load: swap in your own bucket, column names and aggregations, and keep the read, clean, aggregate and write steps in that order. The glue job with pyspark pattern here carries over to larger files without a rewrite, although you will want to revisit the schema and partitioning as the data grows.

Spark with AWS Glue - Getting Started with Data Processing and Analytics