Built with Tutify.io

    AIStudyLabs

    Spark Get DataType & Column Names of DataFrame

    Spark Get DataType & Column Names of DataFrame

    In Spark you can get all DataFrame column names and types (DataType) by using df.dttypes and df.schema where df is an object of DataFrame. Let's see some examples of how to get data type and column name of all columns and data type of selected column by name using Scala examples.

    Related: Convert Column Data Type in Spark DataFrame

    <a id="h-1-spark-get-all-datatype-amp-column-names"></a>

    1. Spark Get All DataType & Column Names

    First, let's see how to get all data types (DataType) & column names using df.dttypes; where dttypes returns all Spark DataFrame columns as Array[(String,Stirng)]. The first value in the tuple is the column name and the second value is a data type.

    
import spark.sqlContext.implicits._
val df = Seq((1,"Robert"), (2,"Julia")).toDF("id","name")

// Get All column names and it's types
df.dtypes.foreach(f=>println(f._1+","+f._2))

// Prints below column name & data type
// Id,IntegerType
// Name,StringType

    Similarly, you can also get the data type & name of all columns using df.schema, schema returns a StructType which is an array of StructField, and by using its methods you can get the column name and its type. df.schema.fields returns Array[StructField].

    
// Get All column names and it's types
df.schema.fields.foreach(f=>f.name +","+f.dataType)

    This yields the same output as above.

    <a id="h-2-get-datatype-of-a-specific-column-name"></a>

    2. Get DataType of a Specific Column Name

    If you want to get the data type of a specific DataFrame column by name then use the below example.

    
// Get data type of a specific column
println(df.schema("name").dataType)

// Prints data type of a "name" column
// StringType

    <a id="h-3-get-all-column-names"></a>

    3. Get All Column Names

    You can get the all columns of a Spark DataFrame by using df.columns, it returns an array of column names as Array[Stirng].

    
// Get All column names from DataFrame
val allColumnNames=df.columns
println(allColumnNames.mkString(","))

// Print all column names in comma separated string
// Id,name

    <a id="h-4-get-dataframe-schema"></a>

    4. Get DataFrame Schema

    As you would already know, use df.printSchama() to display column name and types to console, similarly df.schema.printTreeString() also prints schema to console.

    
df.printSchema()

// root
// |-- id: integer (nullable = false)
// |-- name: string (nullable = true)

    <a id="h-5-get-column-nullable-property-amp-metadata"></a>

    5. Get Column Nullable Property & Metadata

    Let's see how to get if a column is accepts null values (Nullable) and Metadata of the column.

    
// Get Column Nullable Property & Metadata
print(df.schema("name").nullable)
val metaData=df.schema("name").metadata

    <a id="h-6-other-ways-to-get-dataframe-schema"></a>

    6. Other ways to get DataFrame Schema

    
print(df.schema.toDDL)
// `id` INT,`name` STRING

print(df.schema.prettyJson)
//{
//  "type" : "struct",
//  "fields" : [ {
//    "name" : "id",
//    "type" : "integer",
//    "nullable" : false,
//    "metadata" : { }
//  }, {
//    "name" : "name",
//    "type" : "string",
//    "nullable" : true,
//    "metadata" : { }
//  } ]
// }

    <a id="h-7-conclusion"></a>

    7. Conclusion

    In summary, you can get the names and data type's (DataType) of all DataFrame column's by using df.dttypes and df.schema and also you can use several StructFeild methods to get the additional details of the Spark DataFrame column.

    Happy Learning !!

    <a id="h-related-articles"></a>

    Related Articles