Convert Polars Casting a Column to Decimal

In Polars, casting a column to Decimal involves converting the column’s data type to a high-precision decimal format. This is crucial in scenarios where exactness and control over decimal places are essential, such as financial calculations, measurements, or any data analysis that demands precision.

By casting a column to Decimal, you control the total number of digits (precision) and the number of digits after the decimal point (scale), helping you maintain accuracy in your data. In this article, I will explain how to convert polars casting a column to decimal.

Key Points –

• Decimal Type in Polars is used for high-precision numeric operations, especially useful in financial or scientific computations.
• Casting to Decimal is done using cast(pl.Decimal(precision, scale)) where precision is the total number of digits and scale is digits after the decimal point.
• Float64 and Int64 columns can be cast directly to Decimal without needing conversion to string.
• Null values are preserved during casting; they remain null in the resulting Decimal column.
• You can cast from Float64, Int64, or similar numeric types to Decimal.
• Negative numbers are correctly handled and retain their sign in the Decimal format.
• The cast(pl.Decimal(precision, scale)) method is used within with_columns() to apply changes.
• After casting, the column’s data type becomes decimal, providing precise numeric control.

<a id="h-usage-of-nbsp-polars-casting-a-column-to-decimal"></a>

Usage of Polars Casting a Column to Decimal

Casting a column to Decimal in Polars means changing the column’s data type to Decimal, which allows for high-precision numeric operations. This is useful when working with data that requires exact decimal values, such as financial calculations or scientific measurements.

First, let’s create a Polars DataFrame.

import polars as pl

# Creating a DataFrame 
technologies = {
   'Courses': ["Spark", "PySpark", "Pandas"],
   'Fee': [22000, 25000, 26000], 
   'Discount': [100.50, 250.20, 140.40] 
}

df = pl.DataFrame(technologies)
print("Original DataFrame:\n", df)

Yields below output.

To cast a Float64 column to a Decimal in Polars, you can use the cast() method. However, Polars doesn't directly support a Decimal type, so you can convert it to a Decimal using pl.datatypes.Decimal by specifying the precision and scale.

# Casting the 'Discount' column from Float64 to Decimal
df2 = df.with_columns(
    pl.col('Discount').cast(pl.Decimal(10, 2)).alias('Discount_Decimal'))
print("DataFrame after casting 'Discount' to Decimal:\n", df2)

Here,

• pl.Decimal(10, 2) specifies a decimal type with a total precision of 10 and a scale of 2 (i.e., up to two decimal places).
• The cast() method is applied to the 'Discount' column to convert it to the Decimal type.
• The with_columns() method is used to add the newly casted column to the DataFrame.

Alternatively, to cast a Float64 column to Decimal in Polars, you can use the cast() method.

# Cast 'Discount' column from Float64 to Decimal
df2 = df.with_columns(
    pl.col("Discount").cast(pl.Decimal))
print("DataFrame with 'Discount' as Decimal:\n", df2)

# Output:
# ataFrame with 'Discount' as Decimal:
3 shape: (3, 3)
┌─────────┬───────┬───────────────┐
│ Courses ┆ Fee   ┆ Discount      │
│ ---     ┆ ---   ┆ ---           │
│ str     ┆ i64   ┆ decimal[38,0] │
╞═════════╪═══════╪═══════════════╡
│ Spark   ┆ 22000 ┆ 100           │
│ PySpark ┆ 25000 ┆ 250           │
│ Pandas  ┆ 26000 ┆ 140           │
└─────────┴───────┴───────────────┘

Here,

• The Discount column is originally a Float64 type.
• Using cast(pl.Decimal), we convert the Discount column into a Decimal type, which ensures precision and avoids floating-point rounding errors.
• The with_columns() method applies the transformation to the Discount column while keeping the rest of the DataFrame unchanged.

<a id="h-casting-an-integer-column-to-decimal"></a>

Casting an Integer Column to Decimal

To cast an integer column to a Decimal in Polars, you can use the same approach as you would for a Float64 column, but instead, cast the integer column to Decimal. By using the cast() method, you can convert the integer column to Decimal for greater precision. In this example, we'll cast the Fee column (currently an integer) to Decimal to ensure higher precision.

# Casting the 'Fee' column from Integer to Decimal
df2 = df.with_columns(
    pl.col('Fee').cast(pl.Decimal(10, 2)).alias('Fee_Decimal'))
print("\nDataFrame after casting 'Fee' to Decimal:\n", df2)

# Output:
# DataFrame after casting 'Fee' to Decimal:
# shape: (3, 4)
┌─────────┬───────┬──────────┬───────────────┐
│ Courses ┆ Fee   ┆ Discount ┆ Fee_Decimal   │
│ ---     ┆ ---   ┆ ---      ┆ ---           │
│ str     ┆ i64   ┆ f64      ┆ decimal[10,2] │
╞═════════╪═══════╪══════════╪═══════════════╡
│ Spark   ┆ 22000 ┆ 100.5    ┆ 22000.00      │
│ PySpark ┆ 25000 ┆ 250.2    ┆ 25000.00      │
│ Pandas  ┆ 26000 ┆ 140.4    ┆ 26000.00      │
└─────────┴───────┴──────────┴───────────────┘

Here,

• The pl.Decimal(10, 2) specifies the decimal type, where 10 is the total number of digits and 2 is the number of digits after the decimal point.
• The cast() method is used on the 'Fee' column to convert it from Integer to Decimal.
• with_columns() adds the newly casted column to the DataFrame.

<a id="h-casting-a-column-to-decimal-with-specific-precision"></a>

Casting a Column to Decimal with Specific Precision

To cast a column to Decimal with specific precision in Polars, you can use the cast() method with the desired precision and scale. The precision represents the total number of significant digits, while the scale represents the number of digits after the decimal point.

# Cast 'Discount' column from Float64 
# To Decimal with precision 10 and scale 3
df_decimal = df.with_columns(
    pl.col("Discount").cast(pl.Decimal(10, 3)))
print("DataFrame with 'Discount' as Decimal (precision 10, scale 3):\n", df_decimal)

# Output:
# DataFrame with 'Discount' as Decimal (precision 10, scale 3):
# shape: (3, 3)
┌─────────┬───────┬───────────────┐
│ Courses ┆ Fee   ┆ Discount      │
│ ---     ┆ ---   ┆ ---           │
│ str     ┆ i64   ┆ decimal[10,3] │
╞═════════╪═══════╪═══════════════╡
│ Spark   ┆ 22000 ┆ 100.500       │
│ PySpark ┆ 25000 ┆ 250.200       │
│ Pandas  ┆ 26000 ┆ 140.400       │
└─────────┴───────┴───────────────┘

Here,

• Precision 10: This means the decimal number can have up to 10 total digits.
• Scale3: This means there will be exactly 3 digits after the decimal point, ensuring that the values are stored with two decimal places.

<a id="h-casting-a-column-with-null-values-to-decimal"></a>

Casting a Column with Null Values to Decimal

When you have a column with null values and want to cast it to Decimal, Polars will handle the null values gracefully. The null values will remain null after casting to Decimal. This behavior is consistent with Polars' handling of null values across data types.

import polars as pl

# Creating a DataFrame with null values
technologies = {
   'Courses': ["Spark", "PySpark", "Pandas"],
   'Fee': [22000, 25000, 26000], 
   'Discount': ["100.50", None, "140.40"]  # Contains string numbers and a null value
}
df = pl.DataFrame(technologies)

# Cast 'Price' column from String to Decimal
df2 = df.with_columns(
    pl.col("Discount").cast(pl.Decimal(10, 2)))
print("DataFrame with 'Discount' as Decimal (with nulls):\n", df2)

# Output:
# DataFrame with 'Discount' as Decimal (with nulls):
# shape: (3, 3)
┌─────────┬───────┬───────────────┐
│ Courses ┆ Fee   ┆ Discount      │
│ ---     ┆ ---   ┆ ---           │
│ str     ┆ i64   ┆ decimal[10,2] │
╞═════════╪═══════╪═══════════════╡
│ Spark   ┆ 22000 ┆ 100.50        │
│ PySpark ┆ 25000 ┆ null          │
│ Pandas  ┆ 26000 ┆ 140.40        │
└─────────┴───────┴───────────────┘

Here,

• The Discount column contains numeric values represented as strings (like "100.50", "140.40") and one null value.
• Using cast(pl.Decimal(10, 2)), we cast the Discount column to a Decimal type with precision 10 and scale 2.
• The null value remains null after casting because null is a special value that is not affected by casting operations.

<a id="h-casting-a-string-column-to-decimal"></a>

Casting a String Column to Decimal

To cast a string column to Decimal in Polars, you can use the cast() method with the pl.Decimal type. When casting from a string, Polars will attempt to interpret the string as a valid number (e.g., a numeric string such as "100.50" or "250.75").

import polars as pl

# Creating a DataFrame with null values
technologies = {
   'Courses': ["Spark", "PySpark", "Pandas"],
   'Fee': [22000, 25000, 26000], 
   'price_str': ["100.50", "250.20", "140.40"]  # String representation of prices
}

df = pl.DataFrame(technologies)

# Cast 'price_str' from String to Decimal
df2 = df.with_columns(
    pl.col("price_str").cast(pl.Decimal(10, 2))  # Precision 10, scale 2
)
print("DataFrame with 'price_str' as Decimal:\n", df2)

# Output:
# DataFrame with 'price_str' as Decimal:
# shape: (3, 3)
┌─────────┬───────┬───────────────┐
│ Courses ┆ Fee   ┆ price_str     │
│ ---     ┆ ---   ┆ ---           │
│ str     ┆ i64   ┆ decimal[10,2] │
╞═════════╪═══════╪═══════════════╡
│ Spark   ┆ 22000 ┆ 100.50        │
│ PySpark ┆ 25000 ┆ 250.20        │
│ Pandas  ┆ 26000 ┆ 140.40        │
└─────────┴───────┴───────────────┘

Here,

• The price_str column contains values as strings like "100.50", "250.20", and "140.40".
• Using cast(pl.Decimal(10, 2)), we convert the string values to Decimal with a precision of 10 and scale of 2 (2 digits after the decimal point).
• This conversion ensures that the data is handled with more precision compared to floating-point types.

<a id="h-casting-a-column-with-negative-values-to-decimal"></a>

Casting a Column with Negative Values to Decimal

Casting a column with negative values to Decimal in Polars works similarly to casting columns with positive values. Negative values are preserved during the casting process, and the resulting Decimal will have the appropriate sign, precision, and scale.

import polars as pl

# Creating a DataFrame with null values
technologies = {
   'Courses': ["Spark", "PySpark", "Pandas"],
   'Fee': [22000, 25000, 26000], 
  'Discount': ["-100.50", "250.20", "-140.40"]  # String representation of negative prices
}

df = pl.DataFrame(technologies)

# Cast 'Discount' column from String to Decimal (precision 10, scale 2)
df2 = df.with_columns(
    pl.col("Discount").cast(pl.Decimal(10, 2))  
)
print("DataFrame with 'Discount' as Decimal (negative values):\n", df2)

# Output:
# DataFrame with 'Price' as Decimal (negative values):
# shape: (3, 3)
┌─────────┬───────┬───────────────┐
│ Courses ┆ Fee   ┆ Discount      │
│ ---     ┆ ---   ┆ ---           │
│ str     ┆ i64   ┆ decimal[10,2] │
╞═════════╪═══════╪═══════════════╡
│ Spark   ┆ 22000 ┆ -100.50       │
│ PySpark ┆ 25000 ┆ 250.20        │
│ Pandas  ┆ 26000 ┆ -140.40       │
└─────────┴───────┴───────────────┘

Here,

• The Discount column contains negative values represented as strings, like "-100.50" and "-140.40".
• We use cast(pl.Decimal(10, 2)) to convert the column to Decimal with a precision of 10 and a scale of 2.
• The negative sign (-) is preserved during the conversion to Decimal.

<a id="h-conclusion"></a>

Conclusion

In summary, converting casting columns to Decimal in Polars is an effective and simple method to maintain precision, particularly useful for financial data or other scenarios requiring high numerical accuracy. Whether the data type is integer or float, applying the cast(pl.Decimal(precision, scale)) method provides better control over number formatting and helps prevent floating-point rounding errors.

Happy Learning!!

<a id="h-related-articles"></a>

Related Articles

• Polars Looping Through the Rows in a Dataset
• Make a Constant Column in Polars
• Extract Value of Polars Literal
• Convert Polars String to Integer
• Polars Rename Columns to Lowercase
• How to Select Last Column of Polars DataFrame
• How to Change Position of a Column in Polars
• How to Add a Column with Numerical Value in Polars
• Get First N Characters from a String Column in Polars
• Removing Null Values on Selected Columns only in Polars DataFrame

<a id="h-references"></a>

References

• https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.clear.html