Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Reading StringType columns with Int value as partition columns breaks down

Reading StringType columns with Int value as partition columns breaks down

New Contributor

I have a DataFrame with integer values but I store it as a StringType as I need to keep leading zeros. Something like following (sample):

field
01

I save this table as:

sqlContext.write.format("orc").mode(SaveMode.Overwrite).save("sometable/field=01/")

It's all fine up until this step.

The problem starts when I try to read it back. I use the following code to read it back (notice the base folder):

sqlContext.read.format("orc").load("sometable")

What I get back is:

field

(it is blank at best and sometimes it is littered with junk)

Spark Version Used: 1.6.2

Don't have an account?
Coming from Hortonworks? Activate your account here