I have a scenario and would like to get an expert opinion on that.
I have to load a Hive table in partitions from a relational DB via spark (python). I cannot create the hive table as I am not sure how many columns are there in source and they might change in future so I have to fetch data by using select * from tablename.
However I am sure of the partition column and know that will not change. This column is of "date" datatype in source db.
I am using SaveAsTable with partitionBy options and I am able to properly create folders as per the partition column and hive table is also getting created.
The issue I am facing is that since the partition column is of "date" data type and same is not supported in hive for partitions. Due to this I am unable to read data via hive or impala queries as it says date is not supported as partitioned column.
Please note that I cannot typecast the column at the time of issuing select statement as I have to do a select * from tablename and not select a,b,cast(c) as varchar from table.
... View more