Introduction

DataFrame transformations allow us to manipulate, filter, aggregate, and modify data within a Spark application using Java. Like RDD transformations, DataFrame transformations in Spark are lazily evaluated, meaning they compute only after an action is called.

Below are some of the most commonly used DataFrame transformations using Java.

Narrow Transformations

A narrow transformation in Apache Spark using Java refers to an operation on a DataFrame where each input partition contributes only to a single output partition. These transformations are efficient because they can be executed in parallel on each partition without requiring communication between partitions.

Common narrow transformations include:

| Transformation | Description | | --------------- | -------------------------------------------------------------------------------------------------- | | filter() | Filters out rows from the DataFrame based on a condition. | | select() | Selects specific columns from the DataFrame. | | withColumn() | Adds a new column or replaces an existing one with transformed values. | | drop() | Drops specified columns from the DataFrame. | | explode() | Creates new rows for each element in an array or map column. | | sample() | Samples a fraction of rows from the DataFrame. | | randomSplit() | Splits the DataFrame into multiple DataFrames randomly. | | withWatermark() | Specifies the watermark for event-time processing. | | map() | Applies a function to each row in the DataFrame. | | flatMap() | Applies a function that returns an iterator to each row in the DataFrame and flattens the results. | | mapPartitions() | Applies a function to each partition of the DataFrame. |

Example using filter and select transformations in Java Spark:

Dataset<Row> df = ...; // Assume df is an existing DataFrame
// Apply filter and select transformations
Dataset<Row> selectedData = df.filter(col("age").geq(18)).select("name", "age");

Wide Transformations

A wide transformation, also known as a shuffle operation, refers to an operation on a DataFrame where data from multiple partitions must be combined, shuffled, or aggregated. These transformations can affect performance, as they often require network communication and data movement across the cluster.

Common wide transformations include:

| Transformation | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | groupBy() | Groups the DataFrame using specified columns and creates a GroupedData object, often followed by aggregation operations. | | agg() | Performs aggregation operations on grouped data, allowing you to specify multiple aggregation expressions. | | rollup() | Creates multi-dimensional rollups for aggregation, similar to groupBy, but allowing multiple levels of aggregation. | | cube() | Creates multi-dimensional cubes for aggregation, generating all possible combinations of grouping columns. | | pivot() | Pivots a column of the DataFrame and performs aggregation based on the unique values in that column. | | window() | Defines a window specification for use in window functions, which allow calculations across sets of rows related to the current row. | | join() | Performs a join operation between two DataFrames. | | reduceBy() | Reduces the elements of a DataFrame using a user-defined function, allowing for custom aggregation logic, similar to groupBy and aggregate functions. |

Example using groupBy and agg transformations in Java Spark:

Dataset<Row> aggData = df.groupBy("department")
    .agg(
        functions.avg("salary").as("avg_salary"),
        functions.max("salary").as("max_salary")
    );

These transformations are performed in a distributed and parallel manner, leveraging the parallel processing capabilities of Spark's underlying execution engine. It's important to note that transformations in Spark are immutable, meaning they do not modify the original DataFrame but instead return a new DataFrame with the transformation applied.