Hive msck ERROR : Failed with exception java.util.ConcurrentModificationException

[#1]

SYMPTOM Concurrency issues hit during multi-threaded moveFile issued when processing queries such as "INSERT OVERWRITE TABLE ... SELECT .." The following pattern is displayed in stack trace:

Loading data to table testdb.test_table from hdfs://xyz/ra_hadoop/.hive-staging_hive_2017-01-31_14-09-52_561_8101886747064006778-4/-ext-10000 ERROR : Failed with exception java.util.ConcurrentModificationException org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2883) at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3140) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1727) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:353) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151)

ROOT CAUSE

This issue occurs because of issue described in HIVE-15355.

WORKAROUND

• Set the following property at the client side:

set hive.mv.files.thread=0;

• Re-run the hive query

Note: The fix for HIVE-15355 is expected to be included in the next major release of HDP.