Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive Compaction error

Solved Go to solution
Highlighted

Hive Compaction error

Explorer

I have ORC tables created in Hive for ACID transactions and I am able to perform insert/update/delete. But the transaction files created delta* are not getting merged. When I run show compactions, it shows all the compactions were either attempted or failed and none of them were successful. I modified all the properties as per the recommendation from other posts and there is no change in status. Hive metastore logs shows compaction initiation log. Where can I find the actual logs generated by each compaction run? I can see the job name listed in show compactions command but not sure where to look at it. Any suggestions?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hive Compaction error

Expert Contributor

When you do SHOW COMPACTIONS, if compaction MR job was submitted, it will show Hadoop Job ID, which can be used to get more info if the problem is with the job in the Resource Manager UI. If it failed even before submitting the job to the cluster, the errors would be in the log of the standalone Hive Metastore running the compactor processes.

View solution in original post

2 REPLIES 2
Highlighted

Re: Hive Compaction error

Expert Contributor

When you do SHOW COMPACTIONS, if compaction MR job was submitted, it will show Hadoop Job ID, which can be used to get more info if the problem is with the job in the Resource Manager UI. If it failed even before submitting the job to the cluster, the errors would be in the log of the standalone Hive Metastore running the compactor processes.

View solution in original post

Re: Hive Compaction error

Explorer

Thanks @Eugene Koifman I was not able to find the old logs but i did submit compactions manually and it succeeded and the same was logged in both metastore and mapreduce job logs. Not sure what went wrong with automatic compactions. I will try again and check the logs for more detail.

Don't have an account?
Coming from Hortonworks? Activate your account here