Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive

Problem description :

When upgrading to HDP 2.5.3 , some queries to Hive were failing Especially when executing Insert Overwrite queries. with the following error :

ERROR : Failed with exception java.util.ConcurrentModificationException org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2883) at ..... etc

Caused by: java.util.ConcurrentModificationException

Workaround:

"Add SORT BY 0" to your query.

INSERT OVERWRITE TABLE AAA SELECT A , B , C FROM BBBB SORT BY 0 ;

More investigations to follow up ...