Apache Spark Tutorial with Scala Examples — Using SQL Queries

Introduction to Apache Spark

• What is Apache Spark?
• Apache Spark Features
• Apache Spark Architecture
• Apache Spark Ecosystem
• Apache Spark Use Cases & Applications
• Apache Spark vs MapReduce
• Apache Spark Quiz Introduction

Setup Apache Spark Environment

• Install Spark on Mac
• Install Spark on Windows
• Run Spark from IntelliJ
• Run Spark First Program

SparkSession & SparkContext

• What is SparkSession
• Creating SparkSession
• SparkSession Methods
• What is Spark Context?
• What does SparkContext do?
• SparkContext Methods
• Quiz - Spark Session & Context

Resilient Distributed Datasets (RDDs)

• RDD - Introduction
• RDD - Advantages & Limitations
• RDD - Create from Parallelize
• RDD - Process Text Files
• RDD - How to Parallelize
• RDD - Collect Data
• RDD - Transformations
• RDD - Actions
• RDD - Word Count Example
• RDD - Key-Value Pair
• RDD - Fault Tolerance
• RDD - Performance
• RDD - Quiz

Apache Spark Concepts

• Spark - DAG
• Spark - Lineage
• Spark - Lazy Evaluation
• Spark - Driver
• Spark - Executor
• Spark - Worker
• Spark - Job
• Spark - Stage
• Spark - Task

Cache, Persistence and Shared Variables

• Cache and Persistence
• Spark Persistence Levels
• Broadcast Shared Variable
• Accumulator Shared Variable
• RDD Examples to Explore
• Quiz - RDD Cache, Persistence & Shared Variables

DataFrame Introduction

• DataFrame - Introduction
• DataFrame vs RDD
• DataFrame - Create Multiple Ways
• DataFrame - Transformations
• DataFrame - Actions
• DataFrame - Schema
• DataFrame - Quiz

DataFrame Narrow Transformation

• DataFrame - Select Columns
• DataFrame - Filter Rows
• DataFrame - Rename Column
• DataFrame - Add & Update Columns
• DataFrame - Drop Column
• DataFrame - map() and mapPartitions()
• DataFrame - flatMap()
• DataFrame - Union and UnionAll
• DataFrame - Quiz Narrow Transformations

DataFrame Wider Transformation

• DataFrame - Joins
• DataFrame - Groupby and Aggregations
• DataFrame - Remove Duplicate Rows
• DataFrame - Pivot and Unpivot
• DataFrame - Quiz Wider Transformations

DataFrame Actions

• DataFrame - Collect Data
• DataFrame - Basic Actions
• DataFrame - foreach and foreachPartition
• DataFrame - reduce() Action
• DataFrame - Quiz Actions

Spark SQL Functions

• String Functions
• Date Functions
• Timestamp Functions
• Aggregate Functions
• Array Functions
• Map Functions
• Sort Functions
• Window Functions
• JSON Functions
• Quiz - SQL Functions

Advanced DataFrame Features

• Handling Missing Data in DataFrames
• Explain StructType & StructField
• UDF (user Defined Functions)
• DataFrame Cache and Persist
• Creating Views & Tables
• Quiz - Advanced DataFrame Techniques

Using SQL Queries

• SQL - Introduction
• SQL - Creating and Managing Views
• SQL - Executing Queries
• SQL - Grouping & Aggregation
• SQL - Working with Joins
• SQL - Using Built-in Functions
• SQL - Using SQL with DataFrame
• SQL - Quiz

Spark Read & Write CSV Files

• Spark Reading CSV Files
• Using Options in Reading CSV
• Using Custom Schema
• Handing Quotes and Special Characters
• Handling Date and Timestamp
• Write CSV File
• Quiz - Read & Write CSV Files

Working with JSON Files

Working with Parquet Files

• Introduction to Parquet in Spark
• Write DataFrame to Parquet FIle
• Reading Parquet Files in Spark
• Schema Management and Evolution in Parquet
• Performance Optimization with Parquet
• Quiz - Spark with Parquet Files

Spark Dataset

• Introduction to Datasets
• Creating Datasets
• Encoders and Type Safety
• Dataset Transformations
• Actions on Datasets
• Working with Typed Aggregations
• Quiz - Spark Dataset

What Next?

• Advance Spark Concepts