MSCK REPAIR TABLE hangs when hdfs directories of the target table has more than certain number of sub-directories

[#1]

Hello,

I installed HDP-2.5.3 recently and have an issue with MSCK REPAIR TABLE. I tried to migrate data from another cluster and created an external hive table on it. The command was successfully done with until 170 data directories and it was even very quick like 3 seconds. However, when I tried it with 190 or more data directories, it was hanging somewhere until I killed it in a few hours.

I looked at hivemetastore.log and found it didn't proceed any more after having called 'get_partitions'.

I tested it with data with much more directories like thousands in another cluster where HDP-2.4.X is installed and it worked without any problem. And 'get_partitions_with_auth' was called instead of 'get_partitions'. I compared their configs one by one in ambari but don't see any difference.

Does anyone have any ideas?

[#2]

@Donghoon Kang

add condition and give a try, I hope that will help you.

Ex:

alter table abc DROP PARTITION (date=>='2015-09-01');

[#3]

Yeah, I tried it and it worked even though still didn't try it with more than 170 directories. However, I really need to use MSCK REPAIR TABLE instead of ALTER TABLE to add thousands of partitions at a time.

[#4]

Can you take the "jstack" output of the hive cli from when it is stuck and share it here? Would be helpful if you could share hive.log as well.

[#5]

Hi,

Facing the same issue with MSCK REPAIR TABLE after migrating to HDP-2.5.3. Anyone had luck with a workaround?

Thanks in advance!

[#6]

@Donghoon Kang

I am facing same issue after upgrading to 2.5.3 from 2.4.2..did anybody find a solution?

[#7]

@Madan Gudi

I am facing same issue after upgrading to 2.5.3 from 2.4.2..did anybody find a solution?

[#8 · Accepted solution]

We checked out the hive code, removed the patch which was causing this issue ( top two commits affecting org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java). This issue was resolved after that.

[#9]

Can you please suggest the change that you have done?

[#10]

Can you please check if "hive.mv.files.thread=0" helps in this case?