Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HistoryFileManager - Error while trying to move a job to done

Highlighted

HistoryFileManager - Error while trying to move a job to done

Expert Contributor

Hi All,

Our oozie workflow jobs are getting suspended. After search in logs i have found some error in logs

Error in oozie logs:

2019-07-01 14:53:34,740 WARN  ipc.Server - IPC Server handler 3 on 10020, call org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB.getJobReport from 172.20.100.146:58166 Call#137071 Retry#0
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load history file /mr-history/tmp/sshuser/job_1561807385110_19761-1562133916809-sshuser-oozie%3Alauncher%3AT%3Djava%3AW%3DE%2DAgg-1562133931559-1-0-SUCCEEDED-default-1562133922065.jhist

Error in History server:

2019-07-03 06:05:34,723 INFO  hs.HistoryFileManager - Moving /mr-history/tmp/sshuser/job_1561807385110_19761-1562133916809-sshuser-oozie%3Alauncher%3AT%3Djav
a%3AW%3DE%2DAgg-1562133931559-1-0-SUCCEEDED-default-1562133922065.jhist to /mr-history/done/2019/07/03/000019/job_1561807385110_19761-1562133916809-sshuser-oozie%3Alauncher%3AT%3Djava%3AW%3DE%2DAgg-1562133931559-1-0-SUCCEEDED-default-1562133922065.jhist

2019-07-03 06:05:34,780 ERROR hs.HistoryFileManager - Error while trying to move a job to done
java.io.IOException: rename from /mr-history/tmp/sshuser/job_1561807385110_19761-1562133916809-sshuser-oozie%3Alauncher%3AT%3Djava%3AW%3DE%2DAgg-1562133931559-1-0-SUCCEEDED-default-1562133922065.jhist to /mr-history/done/2019/07/03/000019/job_1561807385110_19761-1562133916809-sshuser-oozie%3Alauncher%3AT
%3Djava%3AW%3DE%2DAgg-1562133931559-1-0-SUCCEEDED-default-1562133922065.jhist failed.
at org.apache.hadoop.fs.FileSystem.rename(FileSystem.java:1342)
        at org.apache.hadoop.fs.DelegateToFileSystem.renameInternal(DelegateToFileSystem.java:197)
        at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:749)
        at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:679)
        at org.apache.hadoop.fs.FileContext.rename(FileContext.java:960)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.moveToDoneNow(HistoryFileManager.java:1022)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.access$1000(HistoryFileManager.java:82)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$HistoryFileInfo.moveToDone(HistoryFileManager.java:440)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$1.run(HistoryFileManager.java:917)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748


Kindly help me to resolve the issue.

Don't have an account?
Coming from Hortonworks? Activate your account here