Support Questions
Find answers, ask questions, and share your expertise

Error while exchanging partition in hive tables

Error while exchanging partition in hive tables

Contributor

I am trying to merge the incremental data with an existing hive table.

For testing I created a dummy table from the base table as below:

create base.dummytable like base.fact_table

The table: base.fact_table is partition based on 

dbsource String

When I checked the dummy table's DDL, I could see that the partition column is correctly defined.

PARTITIONED BY (                                                 |
|   `dbsource` string)

Then I tried to exchange one of the partition from the dummy table by dropping it first.

spark.sql("alter table base.dummy drop partition(dbsource='NEO4J')")

The partition: NEO4J has dropped successfully and I ran the exchange statement as below:

 

spark.sql("ALTER TABLE base.dummy EXCHANGE PARTITION (dbsource = 'NEO4J') WITH TABLE stg.inc_labels_neo4jdata")

The exchange statement is giving an error:

Error: Error while compiling statement: FAILED: ValidationFailureSemanticException table is not partitioned but partition spec exists: {dbsource=NEO4J}

The table I am trying to push the incremental data is partitioned by 

dbsource

and I have dropped it successfully. I am running this from spark code and the config is given below:

 

 val conf = new SparkConf().setAppName("MERGER").set("spark.executor.heartbeatInterval", "120s")
      .set("spark.network.timeout", "12000s")
      .set("spark.sql.inMemoryColumnarStorage.compressed", "true")
      .set("spark.shuffle.compress", "true")
      .set("spark.shuffle.spill.compress", "true")
      .set("spark.sql.orc.filterPushdown", "true")
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .set("spark.kryoserializer.buffer.max", "512m")
      .set("spark.serializer", classOf[org.apache.spark.serializer.KryoSerializer].getName)
      .set("spark.streaming.stopGracefullyOnShutdown", "true")
      .set("spark.dynamicAllocation.enabled", "false")
      .set("spark.shuffle.service.enabled", "true")
      .set("spark.executor.instances", "4")
      .set("spark.executor.memory", "4g")
      .set("spark.executor.cores", "5")
      .set("hive.merge.sparkfiles","true")
      .set("hive.merge.mapfiles","true")
      .set("hive.merge.mapredfiles","true")

Could anyone let me know what the mistake I am doing here & what should I change inorder to successfully exchange the partition ?