Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Compactor MR task is failing with file not found error

Solved Go to solution

Compactor MR task is failing with file not found error

Rising Star

In my lab environment I have a transactional hive table setup that is receiving data from NiFi. Everything works except the number of delta directories is blowing up because the compactor is not successfully running. It appears to be looking for a _tmp file that does not exist. This is the error that I am getting:

No of maps and reduces are 0 job_1478131263487_0009
Job commit failed: java.io.FileNotFoundException: File hdfs://node2.localdomain:8020/apps/hive/warehouse/stock.db/hdp/_tmp_80e50691-c9fe-485a-8e1e-ba20331c7d97 does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Compactor MR task is failing with file not found error

Rising Star

Discovered the problem was the hive user account did not have write permissions to the directory containing the hive database. After fixing the permissions the compactor ran succesfully.

1 REPLY 1
Highlighted

Re: Compactor MR task is failing with file not found error

Rising Star

Discovered the problem was the hive user account did not have write permissions to the directory containing the hive database. After fixing the permissions the compactor ran succesfully.