Member since
11-07-2021
1
Post
0
Kudos Received
0
Solutions
11-07-2021
04:23 AM
I'm loading the pyspark(spark-2.4 & python-3.7.4) data frame to Hive using Hivewarehouseconnector. Hive tables are partitioned and orc formatted. Data frame can contain multiple partition values. I need to load those partition values in single load and overwrite if that partition was already exists. df.select(columns).write\
.format(HiveWarehouseSession().HIVE_WAREHOUSE_CONNECTOR)\
.mode('overwrite')\
.option('inferSchema', 'true')\
.option('table','tablename')\
.option('partition','partition_column')\
.save() Got the below error. I'm passing right partition column name. Caused by: java.lang.IllegalArgumentException: Invalid partition spec: partition_column. Even in append mode, it's not working. I'm getting same error. How to accomplish the load? Kindly help.
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)