question Re: Hive external table MSCK and DROP not working.. in Support Questions

Hive external table MSCK and DROP not working..

hanumanth — Tue, 15 Nov 2022 04:10:03 GMT

Hi Team,

We have 1 table which has more than 1 lakh partitions. Support team moved data from HDFS to other path and try to do MSCK but it is not working even drop table also. Getting below errors..

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

While drooping partitions also getting same error after 10min

can one help me on this..

Re: Hive external table MSCK and DROP not working..

asish — Tue, 15 Nov 2022 05:39:20 GMT

As there are lot many partitions,so DROP command would time out.

You must remove the partitions using below command:

You need to drop partition manually https://stackoverflow.com/questions/13815179/how-to-update-drop-a-hive-partition

Then invoke drop table.

Drop command times out owing to lot many partitions.

Re: Hive external table MSCK and DROP not working..

smruti — Tue, 15 Nov 2022 12:34:19 GMT

@hanumanth Review the logs to get a better idea on what it is failing on. If we have that many partitions, do check what's the value of hive.msck.repair.batch.size and if it is set to 0 at present, you may set this to a value such as 3000 and retry.

Re: Hive external table MSCK and DROP not working..

VidyaSargur — Mon, 21 Nov 2022 04:55:01 GMT

@hanumanth, Have the replies helped resolve your issue? If so, can you please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future?