Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while exchanging partition in hive tables

Error while exchanging partition in hive tables

Contributor

I am trying to merge the incremental data with an existing hive table.

For testing I created a dummy table from the base table as below:

create base.dummytable like base.fact_table

The table: base.fact_table is partition based on 

dbsource String

When I checked the dummy table's DDL, I could see that the partition column is correctly defined.

PARTITIONED BY (                                                 |
|   `dbsource` string)

Then I tried to exchange one of the partition from the dummy table by dropping it first.

spark.sql("alter table base.dummy drop partition(dbsource='NEO4J')")

The partition: NEO4J has dropped successfully and I ran the exchange statement as below:

 

spark.sql("ALTER TABLE base.dummy EXCHANGE PARTITION (dbsource = 'NEO4J') WITH TABLE stg.inc_labels_neo4jdata")

The exchange statement is giving an error:

Error: Error while compiling statement: FAILED: ValidationFailureSemanticException table is not partitioned but partition spec exists: {dbsource=NEO4J}

The table I am trying to push the incremental data is partitioned by 

dbsource

and I have dropped it successfully. I am running this from spark code and the config is given below:

 

 val conf = new SparkConf().setAppName("MERGER").set("spark.executor.heartbeatInterval", "120s")
      .set("spark.network.timeout", "12000s")
      .set("spark.sql.inMemoryColumnarStorage.compressed", "true")
      .set("spark.shuffle.compress", "true")
      .set("spark.shuffle.spill.compress", "true")
      .set("spark.sql.orc.filterPushdown", "true")
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .set("spark.kryoserializer.buffer.max", "512m")
      .set("spark.serializer", classOf[org.apache.spark.serializer.KryoSerializer].getName)
      .set("spark.streaming.stopGracefullyOnShutdown", "true")
      .set("spark.dynamicAllocation.enabled", "false")
      .set("spark.shuffle.service.enabled", "true")
      .set("spark.executor.instances", "4")
      .set("spark.executor.memory", "4g")
      .set("spark.executor.cores", "5")
      .set("hive.merge.sparkfiles","true")
      .set("hive.merge.mapfiles","true")
      .set("hive.merge.mapredfiles","true")

Could anyone let me know what the mistake I am doing here & what should I change inorder to successfully exchange the partition ?