Hello Experts,
We are using HDP 3.1.5 and since short time we have one specific table with several partitions created in it, where the compaction for one of these partitions fails repeatly. I'm not able to find the exact error why it is failing.
from yarn I see the following error in the application log:
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1657267798389_0085_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
and in Metastore Log I only find this:
ERROR [thread-71]: compactor.Worker (Worker.java:run(198)) - Caught exception while trying to compact
id:697908,dbname:standard,tableName:<table-name>,partName:<partitionname>,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highe
stWriteId:0. Marking failed to avoid repeated failures, java.io.IOException: Major compactor job failed for thread-71-compactor-<dbname>.<tablename>.<partitionname>! Hadoop JobId: job_1657267798389_0085
at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.launchCompactionJob(CompactorMR.java:596)
at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:322)
at org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:182)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:179)
I'm unable to find any more information on what going wrong here.
Best Regards
