Support Questions

Find answers, ask questions, and share your expertise

Hive Compaction error

avatar
Explorer

I have ORC tables created in Hive for ACID transactions and I am able to perform insert/update/delete. But the transaction files created delta* are not getting merged. When I run show compactions, it shows all the compactions were either attempted or failed and none of them were successful. I modified all the properties as per the recommendation from other posts and there is no change in status. Hive metastore logs shows compaction initiation log. Where can I find the actual logs generated by each compaction run? I can see the job name listed in show compactions command but not sure where to look at it. Any suggestions?

1 ACCEPTED SOLUTION

avatar
Super Collaborator

When you do SHOW COMPACTIONS, if compaction MR job was submitted, it will show Hadoop Job ID, which can be used to get more info if the problem is with the job in the Resource Manager UI. If it failed even before submitting the job to the cluster, the errors would be in the log of the standalone Hive Metastore running the compactor processes.

View solution in original post

2 REPLIES 2

avatar
Super Collaborator

When you do SHOW COMPACTIONS, if compaction MR job was submitted, it will show Hadoop Job ID, which can be used to get more info if the problem is with the job in the Resource Manager UI. If it failed even before submitting the job to the cluster, the errors would be in the log of the standalone Hive Metastore running the compactor processes.

avatar
Explorer

Thanks @Eugene Koifman I was not able to find the old logs but i did submit compactions manually and it succeeded and the same was logged in both metastore and mapreduce job logs. Not sure what went wrong with automatic compactions. I will try again and check the logs for more detail.