after upgrade to hdp-2.5 insert query is failing with following exception,the same query was working in hdp-2.3.4

[#1]

ERROR [main]: exec.Task (SessionState.java:printError(989)) - Failed with exception java.util.ConcurrentModificationException org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2654) at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2974) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1732) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:353) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:217) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:315) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:429) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:718) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

[#2 · Accepted solution]

this seems a problem with move task parallelism introduced in hdp-2.5, could you please try run this query after setting following param

set hive.mv.files.thread=0

[#3]

Can you explain what does set hive.mv.thread=0 does? I assume it removes parallelism option,does that reduce performance of the insert query.