Introduction
DataFrame transformations are operations that allow us to manipulate, filter, aggregate, and modify the data. Like RDD transformations, DataFrame transformations are lazily evaluated, meaning they compute only after an action is called.
Below are some of the most used DataFrame transformations.
Narrow Transformations
A narrow transformation in Apache Spark refers to an operation on a DataFrame where each input partition contributes only to a single output partition. Narrow transformations can be computed independently on each partition of the DataFrame without the need to shuffle or exchange data between partitions.
These transformations are more efficient because they can be executed in parallel on each partition of the data without requiring communication between partitions.
| Transformation | Description | | --------------- | -------------------------------------------------------------------------------------------------- | | filter() | Filters out rows from the DataFrame based on a condition. | | select() | Selects specific columns from the DataFrame. | | withColumn() | Adds a new column or replaces an existing one with the transformed values. | | drop() | Drops specified columns from the DataFrame. | | explode() | Creates new rows for each element in an array or map column. | | sample() | Samples a fraction of rows from the DataFrame. | | randomSplit() | Splits the DataFrame into multiple DataFrames randomly. | | withWatermark() | Specifies the watermark for event-time processing. | | map() | Applies a function to each row in the DataFrame. | | flatMap() | Applies a function that returns an iterator to each row in the DataFrame and flattens the results. | | mapPartitions() | Applies a function to each partition of the DataFrame. |
Wide Transformations
A wide transformation, also known as a shuffle operation, refers to an operation on a DataFrame where data from multiple partitions needs to be combined, shuffled, or aggregated. These operations involve exchanging data between partitions and can impact performance as they may require network communication and data movement across the cluster.
Wide transformations are necessary when computations depend on data that is distributed across multiple partitions and needs to be aggregated or grouped. These operations often involve a shuffle phase, where data is redistributed across partitions based on keys or grouping criteria.
| Transformation | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | groupBy() | Groups the DataFrame using specified columns, creating a GroupedData object. This is often followed by aggregation operations. | | agg() | Performs aggregation operations on grouped data. It allows you to specify multiple aggregation expressions at once. | | rollup() | Creates multi-dimensional rollups for aggregation. It's similar to groupBy but allows you to specify multiple levels of aggregation. | | cube() | Creates multi-dimensional cubes for aggregation. It's similar to rollup but generates all possible combinations of grouping columns. | | pivot() | Pivots a column of the DataFrame and performs aggregation based on the unique values in that column. | | window() | Defines a window specification for use in window functions. Window functions allow you to perform calculations across a set of rows related to the current row. | | join() | Performs a join operation between two DataFrames. | | reduceBy() | Reduces the elements of a DataFrame using a user-defined function. It's similar to groupBy and aggregate functions, but it allows you to specify a custom aggregation logic. |
These are just some of the commonly used transformations in Apache Spark DataFrame API. There are more transformations available depending on specific use cases and requirements.
These transformations are performed in a distributed and parallel manner, leveraging the parallel processing capabilities of Spark's underlying execution engine. It's important to note that transformations in Spark are immutable, meaning they do not modify the original DataFrame but instead return a new DataFrame with the transformation applied.