question Re: Unable to drop Hive table due to corrupt partition. in Archives of Support Questions (Read Only)

Unable to drop Hive table due to corrupt partition.

bsaini — Wed, 14 Oct 2015 09:26:37 GMT

When running a Hive CTAS query that was using wrong serde (accidently) the query was killed in the middle which caused a few partitions to get created but the partition looks corrupted.. Notice the non-ascii character in the partition name.

/apps/hive/warehouse/mydb.db/mytbl
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__/equip_init_f1=ϧ
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__/equip_init_f1=ϧ/equip_nbr_l1=__HIVE_DEFAULT_PARTITION__
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__/equip_init_f1=ϧ/equip_nbr_l1=__HIVE_DEFAULT_PARTITION__/000004_0
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__/equip_init_f1=?
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__/equip_init_f1=?/equip_nbr_l1=__HIVE_DEFAULT_PARTITION__
/apps/hive/warehouse/mydb.db/mytbl/created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__/equip_init_f1=?/equip_nbr_l1=__HIVE_DEFAULT_PARTITION__/000083_0

When running a DROP table statement is run, following exception appears in the metastore.log

2015-10-13 17:55:50,660 ERROR [pool-3-thread-35]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(151)) - Error happens in method drop_table_with_environment_context: MetaException(message:Timeout when executing method: drop_table_with_environment_context)
	at org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
	at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
	at org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
	at org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1820)
	at org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1807)
	at org.apache.hadoop.hive.metastore.ObjectStore.access$200(ObjectStore.java:160)
	at org.apache.hadoop.hive.metastore.ObjectStore$2.getJdoResult(ObjectStore.java:1734)
	at org.apache.hadoop.hive.metastore.ObjectStore$2.getJdoResult(ObjectStore.java:1725)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2391)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1725)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1719)
	at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
	at com.sun.proxy.$Proxy0.getPartitions(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.dropPartitionsAndGetLocations(HiveMetaStore.java:1693)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1532)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1737)
	at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
	at com.sun.proxy.$Proxy5.drop_table_with_environment_context(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9256)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9240)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when executing method: drop_table_with_environment_context
	at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
	... 35 more

Re: Unable to drop Hive table due to corrupt partition.

bsaini — Wed, 14 Oct 2015 09:32:11 GMT

Here is what I see in the `PARTITIONS` table in metastore (special

(404,1444064420,0,'created_yr=__HIVE_DEFAULT_PARTITION__/created_mo=__HIVE_DEFAULT_PARTITION__/equip_init_f1=?/equip_nbr_l1=__HIVE_DEFAULT_PARTITION__',452,48,NULL)

Re: Unable to drop Hive table due to corrupt partition.

bsaini — Wed, 14 Oct 2015 12:42:36 GMT

Also, Since the DROP table command just hangs, what will be a good work around to completely removing this table and partitions?

Re: Unable to drop Hive table due to corrupt partition.

dstreev — Wed, 14 Oct 2015 16:14:55 GMT

Try creating a temporary database and move the table 'as is' into the new database.

CREATE DATABASE if not exists Junk;

USE targetDB;

ALTER TABLE MyCorruptTable RENAME TO Junk.MyMovedCorruptTable;

DROP DATABASE JUNK Cascade;

Re: Unable to drop Hive table due to corrupt partition.

bsaini — Wed, 14 Oct 2015 19:40:52 GMT

Thanks David, will try that and let you know how it goes.

Re: Unable to drop Hive table due to corrupt partition.

bsaini — Sat, 17 Oct 2015 20:03:16 GMT

Thanks David, for some reason the 3rd statement to alter the table to rename it fails with following error message -

metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) - InvalidOperationException(message:Unable to change partition or table. Database th_ti_hit_src does not exist Check metastore logs for detailed stack.junk_th)

  at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:253)

Looks like we will have to clean the table up manually from metastore and then delete the partitions manually.

Re: Unable to drop Hive table due to corrupt partition.

nsabharwal — Mon, 19 Oct 2015 19:19:14 GMT

@bsaini@hortonworks.com Can you verify this ? th_ti_hit_src does not exist

Re: Unable to drop Hive table due to corrupt partition.

bsaini — Mon, 19 Oct 2015 22:41:10 GMT

That database exists. User is able to use that database and also manually verified that in the metastore.

Re: Unable to drop Hive table due to corrupt partition.

deepesh1 — Tue, 20 Oct 2015 07:46:50 GMT

Is the DROP TABLE command hanging or is just outright fails? If its hanging it would be good to see the stack dump from the cli to determine where it hangs. The other option is to manually delete the HDFS directory (recursive remove) of the offending table followed by a drop table if exists <tablename>.

Re: Unable to drop Hive table due to corrupt partition.

raj_sivanesan — Thu, 06 Oct 2016 22:44:25 GMT

I have similar issue with one of my table which has about 7000 partitions and they are in the form as follows:

id_copy=íí?@%00%00%1CS

id_copy=ñ%10o%14%00%00%1CS

id_copy=ñ%17ç%0C%00%00%1CS

I have tried the above suggestions above , none had resolved the issue. If you have any other suggestion , it would be appreciated.

I have a table with partitions as below in total 7000 and I am not able to neither drop the table nor the database.

any suggestion is really appreciated.

id_copy=íí?@%00%00%1CS

id_copy=ñ%10o%14%00%00%1CS

id_copy=ñ%17ç%0C%00%00%1CS

Re: Unable to drop Hive table due to corrupt partition.

shuracka — Thu, 19 Oct 2017 03:29:08 GMT

@Raj Sivanesan

Here's what I did in a lab environment. I wound up with over 20k partitions on a table (d'oh) and was ok with blowing out the table/database. I can't confirm that this should be done on a production cluster - use with caution. Feedback is welcome.

Backup Hive Metastore:

mysqldump -u root -p hivedb >> hivedb.bak

Hive Metastore:

--SELECT TBL_ID FROM TBLS WHERE TBL_NAME = 'myTable';
DELETE FROM PARTITION_KEY_VALS WHERE PART_ID IN (SELECT PART_ID FROM PARTITIONS WHERE TBL_ID = 54);
DELETE FROM PARTITION_PARAMS WHERE PART_ID IN (SELECT PART_ID FROM PARTITIONS WHERE TBL_ID = 54);
DELETE FROM PARTITIONS WHERE TBL_ID = 54;

Hive:

DROP DATABASE IF EXISTS myDatabase CASCADE;