Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Convert String column into date & timestamp Spark dataframes

Highlighted

Convert String column into date & timestamp Spark dataframes

Explorer

I am trying to covert string column in dataframe to date/time. I am loading dataframe from hive tables and i have tried below mentioned function in converting string to date/time. But it is not giving me the correct output as it is converting all values to null.

(unix_timestamp($"BIRTHDT","MM-dd-yyyy").cast("date")) &&

(to_date(($"BIRTHDT","MM-dd-yyyy").cast("date"))

Values in my birthdt columns are as follows

20061202 20061203 20061205 20061206 20061208

Am i missing something?

Thanks

Rahul

1 REPLY 1
Highlighted

Re: Convert String column into date & timestamp Spark dataframes

@rahul gulati

The format you are specifying for the date doesn't match what is in your columns. You are saying to use MM-dd-yyyy, but your dates are yyyyMMdd.

Don't have an account?
Coming from Hortonworks? Activate your account here