Apache Spark Tutorial with Java Examples — Working with Parquet Files

Introduction to Apache Spark

• What is Apache Spark?
• Apache Spark Features
• Apache Spark Architecture
• Apache Spark Ecosystem
• Use Cases and Application of Spark
• Apache Spark vs MapReduce
• Quiz - Introduction

Setup Apache Spark Environment

• Install Spark on Mac
• Install Spark on Windows
• Run Spark from IntelliJ
• Run Spark First Program

SparkSession & SparkContext

• What is SparkSession
• Creating SparkSession
• SparkSession Methods
• What is JavaSparkContext?
• What does SparkContext do?
• JavaSparkContext Methods
• Quiz - Spark Session & Context

Resilient Distributed Datasets (RDDs)

• RDD Introduction
• RDD Advantages & Limitations
• Creating RDD from Parallelize
• Read Text Files into RDD
• How to Parallelize RDD
• Collect Data from RDD
• RDD Transformations
• RDD Actions
• RDD - Word Count Example
• Working with Key-Value RDDs
• Fault Tolerance in RDDs
• RDD Performance
• Quiz - Java Spark RDD

Cache, Persistence and Shared Variables

• Cache and Persistence
• Spark Persistence Levels
• Broadcast Shared Variable
• Accumulator Shared Variable
• Quiz - RDD Cache, Persistence & Shared Variables

DataFrame Introduction

• What is a DataFrame
• How DataFrame differ from RDD
• Create DataFrame
• DataFrame Transformations
• DataFrame Actions
• Spark Schema
• Quiz - DataFrame Introduction

DataFrame Narrow Transformation

• Select DataFrame Columns
• Filter DataFrame Rows
• Rename DataFrame Column
• Add & Update Columns of DataFrame
• Drop Column
• Spark map() and mapPartitions()
• Spark flatMap()
• Spark Union and UnionAll
• Quiz - DataFrame Narrow Transformations

DataFrame Wider Transformation

• Spark SQL Join Types
• Groupby and Aggregations
• Remove Duplicate Rows
• Pivot and Unpivot DataFrame
• Quiz - Wider Transformations

DataFrame Actions

• Spark Collect Data from DataFrame
• show, count, take, first, head
• Spark foreach and foreachPartition
• DataFrame reduce() Action
• Quiz - DataFrame Actions

Spark SQL Functions

• String Functions
• Date Functions
• Timestamp Functions
• Aggregate Functions
• Array Functions
• Map Functions
• Sort Functions
• Window Functions
• JSON Functions
• Quiz - SQL Functions

Advanced DataFrame Features

• Handling Missing Data in DataFrames
• Explain StructType & StructField
• UDF (user Defined Functions)
• DataFrame Cache and Persist
• Creating Views & Tables
• Quiz - Advanced DataFrame Techniques

Using SQL Queries

• Introduction to Spark SQL
• Creating and Managing Views
• Executing SQL Queries in Spark
• Aggregation, and Grouping
• Working with Joins in Spark SQL
• Using Built-in Functions in Spark SQL
• Combining SQL with DataFrame API
• Quiz - Using SQL Queries in Spark

Spark Read & Write CSV Files

• Spark Reading CSV Files
• Using Options in Reading CSV
• Using Custom Schema
• Handing Quotes and Special Characters
• Handling Date and Timestamp
• Write CSV File
• Quiz - Read & Write CSV Files

Working with JSON Files

Working with Parquet Files

• Introduction to Parquet in Spark
• Write DataFrame to Parquet FIle
• Reading Parquet Files in Spark
• Schema Management and Evolution in Parquet
• Performance Optimization with Parquet
• Quiz - Spark with Parquet Files

Spark Dataset

• Introduction to Dataset
• Creating Datasets
• Encoders and Type Safety
• Dataset Transformations
• Actions on Datasets
• Working with Typed Aggregations
• Quiz - Spark Dataset

What Next?

• Advance Spark Concepts