Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive transaction table compacte delta file error, tmp file does not exist.

avatar
New Contributor

No of maps and reduces are 0 job_1632722322222_0037 Job commit failed: java.io.FileNotFoundException: File hdfs://xxx/user/hive/warehouse/xxx.db/table/dt=2021-09-25/hr=00/_tmp_d5b133c1-68a1-4e98-af45-308ffbdc9f42 does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:986) at org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:122) at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1046) at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1043) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1053) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:781) at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:286) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:238) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

 

I had configed the MapReduce excutor root as hadoop proxyuser like this:

hadoop.proxyuser.root.hosts=*
hadoop.proxyuser.root.groups=*

I changed the file owner to root and chmod 777 of the file

but still got error "_tmp_d5b133c1-68a1-4e98-af45-308ffbdc9f44 does not exist."

please help me, thanks.

4 REPLIES 4

avatar
Guru

@David2 Is this a managed table?

I don't see delta/base dir hdfs://xxx/user/hive/warehouse/xxx.db/table/dt=2021-09-25/hr=00/_tmp_d5b133c1-68a1-4e98-af45-308ffbdc9f42

 

Can you try a  MAJOR compaction and run the job,if this is a MANAGED table????

avatar
New Contributor

yes, it is a MANAGED table, there is no base dir only delta dir.

i will try MAJOR compaction.

thanks.

avatar
New Contributor

compacte major got the same error

avatar
Guru

@David2  does the file exist??

What kind of file is this? 

Please share the DDL for the table.

Please also share the application logs of the failed job

Share application logs for the failed compaction job too.