Introduction
In PySpark, DataFrame actions are operations that trigger the execution of the computation plan established using DataFrame transformations. Unlike transformations, which are lazy and simply define what computation needs to be done, actions prompt Spark to execute these computations and either return results back to the driver program or write them to an external storage system like HDFS, S3, or databases.
Actions in PySpark DataFrame typically involve reading data from external sources, collecting data to the driver program, writing data to external systems, or performing computations that yield non-DataFrame results.
Here are some common DataFrame actions in PySpark:
| Action | Description |
| ---------- | ------------------------------------------------------------------------------------------------ |
| show() | Displays the first few rows of the DataFrame in a tabular format. |
| collect()| Retrieves all rows from the DataFrame and brings them to the driver program as a list of Row objects. |
| take(n) | Retrieves the first N rows from the DataFrame and brings them to the driver program as a list of Row objects. |
| count() | Returns the number of rows in the DataFrame. |
| first() | Returns the first row of the DataFrame as a Row object. |
| head(n) | Returns the first N rows of the DataFrame as a list of Row objects. |
| foreach(f)| Applies a function f to each element (row) of the DataFrame. |
| write() | Writes the DataFrame to external systems such as HDFS, S3, or databases. Use methods like write.format(‘parquet’).save(‘path’). |
| toPandas()| Converts the DataFrame to a Pandas DataFrame. |
These actions initiate the execution of the computation plan and return results or save data as required. Notably, actions are usually the final steps in a PySpark script, as they execute the defined computations and yield tangible results.