What is PySpark?

PySpark is an Apache Spark library written in Python to run Python applications using Apache Spark capabilities. Using PySpark we can run applications parallelly on the distributed cluster (multiple nodes).

In other words, PySpark is a Python API which is an analytical processing engine for large-scale powerful distributed data processing and machine learning applications.

<img height="85" width="531" src="https://yrltjyzsmzdylqvoonwf.supabase.co/storage/v1/object/public/lesson-images/100/183/What-is-PySpark.png" />

source: https://databricks.com/

Apache Spark was basically written in Scala and later, due to its industry adaptation, PySpark was released for Python using Py4J.  Py4J is a Java library that is integrated within PySpark and allows Python to dynamically interface with JVM objects, hence to run PySpark you also need Java to be installed along with Python, and Apache Spark.

In real-time, PySpark has been used a lot in the machine learning & data scientists community; thanks to vast Python machine learning libraries. 

What is Apache Spark?

Apache Spark is an open-source unified analytics engine used for large-scale data processing, hereafter referred it as Spark. Spark is designed to be fast, flexible, and easy to use, making it a popular choice for processing large-scale data sets. Spark runs operations on billions and trillions of data on distributed clusters 100 times faster than traditional applications. 

Apache Spark is a distributed computing system designed for big data processing and analytics. It was originally developed at the University of California, Berkeley's AMPLab, and later donated to the Apache Software Foundation, where it became an Apache project.

Spark is designed to be fast, flexible, and easy to use, making it a popular choice for processing large-scale data sets. It provides a common framework for various data processing tasks, including batch processing, real-time stream processing, machine learning, and graph processing.

Spark can run on single-node machines or multi-node machines(Cluster). It was created to address the limitations of MapReduce, by doing in-memory processing. Spark reuses data by using an in-memory cache to speed up machine learning algorithms that repeatedly call a function on the same dataset.  This lowers the latency making Spark multiple times faster than MapReduce, especially when doing machine learning, and interactive analytics.  Apache Spark can also process real-time streaming. 

It is also a multi-language engine, that provides APIs (Application Programming Interfaces) and libraries for several programming languages like Java, Scala, Python, and R, allowing developers to work with Spark using the language they are most comfortable with.

1. Scala: Spark's primary and native language is Scala. Many of Spark's core components are written in Scala, and it provides the most extensive API for Spark.
2. Java: Spark provides a Java API that allows developers to use Spark within Java applications. Java developers can access most of Spark's functionality through this API.
3. Python: Spark offers a Python API, called PySpark, which is popular among data scientists and developers who prefer Python for data analysis and machine learning tasks. PySpark provides a Pythonic way to interact with Spark.
4. R: Spark also offers an R API, enabling R users to work with Spark data and perform distributed data analysis using their familiar R language.

Apache Spark History

• Apache Spark was started by Matei Zaharia at UC Berkeley's AMPLab in 2009. 
• In 2010 It was open-sourced under BSD Licence.  
• In 2013 Spark grew into a broad community and moved to Apache Software Foundation. 
• Spark 1.x is a significant release with various libraries and APIs for Data processing. 
• In July 2016 Spark 2.0 was released with Structured Streaming, which simplifies real-time data processing by bringing batch and stream processing under a unified API. 
• Spark 3.0 is a major release with AQE(Adaptive Query Execution) and improved Python API changes.