Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Hive Compaction error

Explorer

I have ORC tables created in Hive for ACID transactions and I am able to perform insert/update/delete. But the transaction files created delta* are not getting merged. When I run show compactions, it shows all the compactions were either attempted or failed and none of them were successful. I modified all the properties as per the recommendation from other posts and there is no change in status. Hive metastore logs shows compaction initiation log. Where can I find the actual logs generated by each compaction run? I can see the job name listed in show compactions command but not sure where to look at it. Any suggestions?

1 ACCEPTED SOLUTION

Expert Contributor

When you do SHOW COMPACTIONS, if compaction MR job was submitted, it will show Hadoop Job ID, which can be used to get more info if the problem is with the job in the Resource Manager UI. If it failed even before submitting the job to the cluster, the errors would be in the log of the standalone Hive Metastore running the compactor processes.

View solution in original post

2 REPLIES 2

Expert Contributor

When you do SHOW COMPACTIONS, if compaction MR job was submitted, it will show Hadoop Job ID, which can be used to get more info if the problem is with the job in the Resource Manager UI. If it failed even before submitting the job to the cluster, the errors would be in the log of the standalone Hive Metastore running the compactor processes.

Explorer

Thanks @Eugene Koifman I was not able to find the old logs but i did submit compactions manually and it succeeded and the same was logged in both metastore and mapreduce job logs. Not sure what went wrong with automatic compactions. I will try again and check the logs for more detail.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.