Introduction to PySpark for Data Engineering — SparkSession & SparkContext

Introduction to PySpark

• What is PySpark?
• PySpark Features & Advantages
• PySpark Architecture
• PySpark Ecosystem
• PySpark Use Cases and Applications
• PySpark vs MapReduce
• Quiz - Introduction

Setup PySpark Environment

• Install PySpark on MacOS
• Install PySpark on Windows
• Install PySpark in Anaconda

SparkSession & SparkContext

• What is SparkSession
• Creating SparkSession
• SparkSession Methods
• What is Spark Context?
• What does SparkContext do?
• SparkContext Methods
• Quiz - PySpark Session & Context

Resilient Distributed Datasets (RDDs)

• RDD - Introduction
• RDD - Advantages & Limitations
• RDD - Create from Parallelize
• RDD - Read Text Files
• RDD - Parallelize
• RDD - Collect Data
• RDD - Transformations
• RDD - Actions
• RDD - Word Count Example
• RDD - Working with Key-Value
• RDD - Performance Considerations
• RDD - Quiz

Apache Spark Concepts

• Spark - DAG
• Spark - Lineage
• Spark - Lazy Evaluation
• Spark - Driver
• Spark - Executor
• Spark - Worker
• Spark - Job
• Spark - Stage
• Spark - Task

Cache, Persistence and Shared Variables

• Cache and Persistence
• Spark Persistence Levels
• Broadcast Shared Variable
• Accumulator Shared Variable
• Quiz - RDD Cache, Persistence & Shared Variables

DataFrame Introduction

• DataFrame - Introduction
• DataFrame vs RDD
• DataFrame - Way to Create
• DataFrame - Transformations
• DataFrame - Actions
• DataFrame - Schema
• DataFrame - Quiz

DataFrame Narrow Transformation

• DataFrame - Select Columns
• DataFrame - Filter Rows
• DataFrame - Rename Column
• DataFrame - Add & Update Columns
• DataFrame - Drop Column
• DataFrame - map() and mapPartitions()
• DataFrame - flatMap()
• DataFrame - Union and UnionAll
• DataFrame - Quiz Narrow Transformations

DataFrame Wider Transformation

• DataFrame - SQL Join Types
• DataFrame - Groupby and Aggregations
• DataFrame - Remove Duplicate Rows
• DataFrame - Pivot and Unpivot
• DataFrame - Quiz Wider Transformations

DataFrame Actions

• DataFrame - Collect Data
• DataFrame - Basic Actions
• DataFrame - foreach and foreachPartition
• DataFrame - reduce()
• DataFrame - Quiz Actions

PySpark SQL Functions

• String Functions
• Date Functions
• Timestamp Functions
• Aggregate Functions
• Array Functions
• Map Functions
• Sort Functions
• Window Functions
• JSON Functions
• Quiz - SQL Functions

Advanced DataFrame Features

• Handling Missing Data in DataFrames
• Explain StructType & StructField
• UDF (user Defined Functions)
• DataFrame Cache and Persist
• Creating Views & Tables
• Quiz - Advanced DataFrame Techniques

PySpark SQL

• SQL - Introduction
• SQL - Create Views
• SQL - Query Execution
• SQL - Group & Aggregate
• SQL - Joins
• SQL - Built-in Functions
• SQL - Combining with DataFrame
• SQL - Quiz

Spark Read & Write CSV Files

• Spark Reading CSV Files
• Using Options in Reading CSV
• Using Custom Schema
• Handing Quotes and Special Characters
• Handling Date and Timestamp
• Write CSV File
• Quiz - Read & Write CSV Files

Working with JSON Files

• Spark Read JSON Files
• Spark Write JSON FIles

Working with Parquet Files

• Introduction to Parquet in Spark
• Write DataFrame to Parquet FIle
• Reading Parquet Files in Spark
• Schema Management and Evolution in Parquet
• Performance Optimization with Parquet
• Quiz - Spark with Parquet Files

What Next?

• Advance PySpark Concepts