Use from_dict(), from_records(), json_normalize() methods to convert list of dictionaries (dict) to pandas DataFrame. Dict is a type in Python to hold key-value pairs. Key is used as a column name and value is used for column value when we convert dict to DataFrame. When a key is not found for some dicts and it exists on other dicts, it creates a DataFrame with NaN for non-existing keys.
In this article, we will see how to convert a list of dictionaries (dict) to a pandas DataFrame using pd.DataFrame(), pd.DataFrame.from_dict(), and pd.DataFrame.from_records(). Each dictionary in the list can have similar or different keys but different values.
Key Points –
- Convert a list of dictionaries to a DataFrame directly with
pd.DataFrame(list_of_dicts). - Pandas arranges columns based on the order of keys in the first dictionary by default.
- If some dictionaries lack certain keys, Pandas will insert
NaNfor missing values in those columns. - Use the
columnsparameter to control which columns appear in the DataFrame or their order. - Specify a custom index by using the
indexparameter during DataFrame creation or set the index later. - For better performance, especially with large datasets, consider using
pd.DataFrame.from_records(). - If dictionaries contain nested structures, use
json_normalize()to flatten them into a DataFrame format.
<a id="h-quick-examples-of-convert-a-list-of-dictionaries-to-a-dataframe"></a>
Quick Examples of Convert a List of Dictionaries to a DataFrame
If you are in a hurry, below are some quick examples of how to convert a list of dictionaries (dict) to a Pandas DataFrame.
# Quick examples of convert a list of dictionaries to a dataframe
# Example 1: Convert a list of dictionaries
# Using from_records() method
df = pd.DataFrame.from_records(technologies)
# Example 2: Set Custom index by using index parameter
df = pd.DataFrame.from_records(technologies, index=['1', '2', '3'])
# Example 3: Convert a list of dictionaries
# By from_dict method
df = pd.DataFrame.from_dict(data)
# Example 4: Dictionary orientations of column
df=pd.DataFrame.from_dict(technologies, orient='columns')
# Example 5: Convert a list of dictionaries
# Using json_normalize()
df=pd.json_normalize(technologies)
<a id="h-create-a-dataframe-from-list-of-dict"></a>
Create a DataFrame from List of Dict
If you have a list of dictionaries (dict), it is easy to create a DataFrame by using the DataFrame constructor. For more examples refer to how to create a pandas DataFrame with examples.
# Create a DataFrame from list
# Of dictionaries with default indexes
import pandas as pd
# List of dict object
technologies=[{'Courses':'Spark','Duration':'30days','Discount':1000},
{'Courses':'python','Fee':25000,'Courses_Fee':'Spark'},
{'Fee':30000,'Duration':'35days','Duration_Discount':'10days'}]
# Create DataFrame from list of dic object
df=pd.DataFrame(technologies)
print(df)
Yields below output. Note that when a key is not found for some dicts and it exists on other dicts, it creates a DataFrame with NaN for non-existing keys. In case you would like to change the NaN values refer to How to replace NaN/None values with empty String.
<a id="h-using-from-records-method-convert-a-list-of-dictionaries"></a>
Using from_records() Method Convert a List of Dictionaries
The from_records() method is used to convert a list of dictionaries to DataFrame. It can also be used to convert structured or record ndarray to DataFrame and is used to create a DataFrame from a structured ndarray, sequence of tuples or dicts, or from another DataFrame.
# Convert a list of dictionaries
# Using from_records() method
df = pd.DataFrame.from_records(technologies)
print(df)
Yields the same output as above.
<a id="h-set-custom-index-by-using-index-parameter"></a>
Set Custom Index by Using Index Parameter
To set a custom index while converting a list of dictionaries to a Pandas DataFrame, you can use the index parameter of the pd.DataFrame() constructor or any of its related methods (from_records(), from_dict()). For example, the index parameter is set to the custom_index list, assigning custom index values to the DataFrame. Y
In the second example, let's see how to set a custom index by using index parameter. If you already have a DataFrame, you can set the Index to the DataFrame by using df.index.
# Convert list of dictionaries
# To DataFrame with custom index
custom_index = ['1', '2', '3']
df = pd.DataFrame(technologies, index=custom_index)
print(df)
# Set Custom index by using index parameter
df = pd.DataFrame.from_records(technologies,index=['1', '2', '3'])
print(df)
Yields below output.
# Output:
Courses Duration Discount Fee Courses_Fee Duration_Discount
1 Spark 30days 1000.0 NaN NaN NaN
2 python NaN NaN 25000.0 Spark NaN
3 NaN 35days NaN 30000.0 NaN 10days
<a id="h-convert-a-list-of-dictionaries-by-using-from-dict-method"></a>
Convert a List of Dictionaries by Using from_dict() Method
The pd.DataFrame.from_dict() method can be used to convert a list of dictionaries to a Pandas DataFrame. This method creates a DataFrame from a dictionary or a list of dictionaries. Each dictionary in the list will represent a row in the DataFrame, and the keys of the dictionaries will be used as column names.
Use pd.DataFrame.from_dict() to transform a list of dictionaries to pandas DatFrame. This function is used to construct DataFrame from dict of array-like or dicts.
# Convert a List of Dictionaries by from_dict method
df = pd.DataFrame.from_dict(technologies)
print(df)
Yields below output.
# Output:
Courses Duration Discount Fee Courses_Fee Duration_Discount
0 Spark 30days 1000.0 NaN NaN NaN
1 python NaN NaN 25000.0 Spark NaN
2 NaN 35days NaN 30000.0 NaN 10days
<a id="h-dictionary-orientations-of-columns-and-index"></a>
Dictionary Orientations of Columns and Index
There are two primary types of dictionary orientations which are called Columns and index. It is to make the distinction between the different types of dictionary orientations with the orient='columns'.
# Dictionary orientations of column
df=pd.DataFrame.from_dict(technologies, orient='columns')
print(df)
Yields the same output as above.
<a id="h-convert-a-list-of-dictionaries-by-using-json-normalize"></a>
Convert a List of Dictionaries by Using json_normalize()
If we want to convert an object to a JSON string, we have to note that NaN’s and None will be converted to null and datetime objects will be converted to UNIX timestamps. json_normalize() function works with lists of dictionaries (dict).
# Convert a list of dictionaries
# Using json_normalize
df=pd.json_normalize(technologies)
print(df)
Yields the same output as above.
<a id="h-complete-examples-to-convert-a-list-of-dictionaries-of-pandas-dataframe"></a>
Complete Examples to Convert a List of Dictionaries of Pandas DataFrame
# Create a DataFrame from list of Dictionaries with default indexes.
import pandas as pd
technologies=[{'Courses':'Spark','Duration':'30days','Discount':1000},{'Courses':'python','Fee':25000,'Courses_Fee':'Spark'},{'Fee':30000,'Duration':'35days','Duration_Discount':'10days'}]
df=pd.DataFrame(technologies)
print(df)
# Convert a List of dictionaries using from_records() method
df = pd.DataFrame.from_records(technologies)
print(df)
# Set Custom index by using index parameter
df = pd.DataFrame.from_records(technologies, index=['1', '2', '3'])
print(df)
# Convert a List of Dictionaries by from_dict method
df = pd.DataFrame.from_dict(data)
print(df)
# Dictionary orientations of column
df=pd.DataFrame.from_dict(technologies, orient='columns')
print(df)
# Convert a list of dictionaries using json_normalize
df=pd.json_normalize(technologies)
print(df)
<a id="h-frequently-asked-questions-on-convert-list-of-dictionaries-to-dataframe"></a>
Frequently Asked Questions on Convert List of Dictionaries to DataFrame
You can convert a list of dictionaries to a DataFrame in Pandas using the pd.DataFrame() constructor. Each dictionary in the list will become a row in the DataFrame, and the keys of the dictionaries will become the column names
You can set a custom index while converting a list of dictionaries to a DataFrame in Pandas. You can use the index parameter of the pd.DataFrame() constructor or its related methods such as pd.DataFrame.from_records() or pd.DataFrame.from_dict(). Here's an example using the pd.DataFrame() constructor:
The keys in the dictionaries should be consistent to create a DataFrame properly. If the keys are inconsistent, you might face issues. Make sure that each dictionary in the list has the same set of keys. Adjust the data accordingly before creating the DataFrame.
Apart from the pd.DataFrame() constructor, you can use pd.DataFrame.from_records() or pd.DataFrame.from_dict() methods. These methods offer additional flexibility in handling the data and specifying the orientation (columns or index).
You can customize the order of columns in a Pandas DataFrame by specifying the columns parameter when using the pd.DataFrame() constructor or its related methods such as pd.DataFrame.from_records() or pd.DataFrame.from_dict(). Here's an example using the pd.DataFrame() constructor
<a id="h-conclusion"></a>
Conclusion
In this article, you have learned about how to convert a list of dictionaries to pandas DataFrame by from_record(), from_dict(), json_normalize() with the examples.
<a id="h-related-articles"></a>
Related Articles
- Pandas Series.isin() Function
- Pandas.Series.combine()
- Convert PySpark DataFrame to Pandas
- convert pandas to pyspark dataframe
- Rename Specific Columns in Pandas
- Pandas Create Empty DataFrame
- Pandas Convert Column to Int in DataFrame
- Pandas Convert JSON to DataFrame
- Pandas Convert Integer to Datetime Type
- Pandas Convert Datetime to Date Column
- pandas convert column to numpy array
- Convert Pandas Column to Lowercase
- Pandas Convert String to Integer
- How to Convert Pandas DataFrame to List?
- Pandas – Convert DataFrame to Dictionary (Dict)
- Pandas Empty DataFrame with Specific Column Types
- Retrieve Number of Columns From Pandas DataFrame
- Create Pandas DataFrame With Working Examples
<a id="h-references"></a>