What is a DataFrame?

Spark DataFrame is a distributed collection of data organized into named columns that computes the data in a distributed environment. Each column in a DataFrame has a name and an associated type. Similar to traditional database tables. 

DataFrame is a high-level API that provides unified access to multiple Spark libraries, including Spark SQL, Spark Streaming, MLib, and GraphX. 

DataFrame is part of the Spark SQL module, providing an easy way to work with SQL-like queries. Spark SQL provides a high-level, structured API for distributed data processing in Spark. DataFrames are designed to combine the benefits of both RDDs (Resilient Distributed Datasets) and traditional relational databases, offering a more convenient and efficient way to work with structured data in distributed environments.

DataFrame Benefits:

• Like RDDs, DataFrames use lazy evaluation, meaning that transformations on DataFrames are not executed immediately. Instead, a logical execution plan is built, and operations are performed when an action is triggered.
• Capable of  inferring  the schema of the data  automatically  while loading from the sources. Also if needed one can define the schema explicitly.
• Provides structured data handling through a schematic view of data. 
• DataFrames have named columns with well-defined data types, which makes them suitable for working with structured and semi-structured data.
• Ability to scale from kilobytes of data on a single laptop to petabytes on a large cluster.
• They perform in an optimized way by taking the advantage of Spark's  Catalyst Query Optimizer and Tungsten execution engine.
• Support for a wide array of data formats and storage systems.
• State-of-the-art optimization and code generation through the Spark SQL Catalyst optimizer.
• Seamless integration with all big data tooling and infrastructure via Spark.
• APIs for Python, Java, Scala, and R (in development via SparkR).