Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Load Hive table with partitions with date

Highlighted

Load Hive table with partitions with date

New Contributor


Hello All, 

I have a scenario and would like to get an expert opinion on that.

I have to load a Hive table in partitions from a relational DB via spark (python). I cannot create the hive table as I am not sure how many columns are there in source and they might change in future so I have to fetch data by using select * from tablename.

However I am sure of the partition column and know that will not change. This column is of "date" datatype in source db.

I am using SaveAsTable with partitionBy options and I am able to properly create folders as per the partition column and hive table is also getting created.

The issue I am facing is that since the partition column is of "date" data type and same is not supported in hive for partitions. Due to this I am unable to read data via hive or impala queries as it says date is not supported as partitioned column.

Please note that I cannot typecast the column at the time of issuing select statement as I have to do a select * from tablename and not select a,b,cast(c) as varchar from table.

Don't have an account?
Coming from Hortonworks? Activate your account here