Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Compactor MR task is failing with file not found error

avatar
Expert Contributor

In my lab environment I have a transactional hive table setup that is receiving data from NiFi. Everything works except the number of delta directories is blowing up because the compactor is not successfully running. It appears to be looking for a _tmp file that does not exist. This is the error that I am getting:

No of maps and reduces are 0 job_1478131263487_0009
Job commit failed: java.io.FileNotFoundException: File hdfs://node2.localdomain:8020/apps/hive/warehouse/stock.db/hdp/_tmp_80e50691-c9fe-485a-8e1e-ba20331c7d97 does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
1 ACCEPTED SOLUTION

avatar
Expert Contributor

Discovered the problem was the hive user account did not have write permissions to the directory containing the hive database. After fixing the permissions the compactor ran succesfully.

View solution in original post

1 REPLY 1

avatar
Expert Contributor

Discovered the problem was the hive user account did not have write permissions to the directory containing the hive database. After fixing the permissions the compactor ran succesfully.