Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Handling decimal values in spark scala

Handling decimal values in spark scala


I have the data in the data file as shown below


This data should be converted to decimal values and should be in a position of 8.7 where 8 are the digits before decimal and 7 are the digits after decimal.

I am trying to read the data file like below.

val readDataFile ="com.databricks.spark.csv").option("header", "true").option("delimiter", "|").schema(***SCHEMA*****).load(****DATA FILE PATH******)

I have tried this

val changed = dataFileWithSchema.withColumn("COLUMN NAME ", dataFileWithSchema.col("COLUMN NAME ").cast(new DecimalType(38,3)))


but it gives me only the zeros like this


But I want digits How can I achieve this ?

can you please help me?

Don't have an account?
Coming from Hortonworks? Activate your account here