Support Questions
Find answers, ask questions, and share your expertise

oozie workflow suspended and YarnRuntimeException: Could not load history file

oozie workflow suspended and YarnRuntimeException: Could not load history file

Expert Contributor

Hi All,

We are getting oozie workflow suspended automatically, as mentioned below

User          : sshuser
Group         : -
Created       : 2019-07-01 14:50 GMT
Started       : 2019-07-01 14:50 GMT
Last Modified : 2019-07-02 04:06 GMT
Ended         : 2019-07-02 04:06 GMT
CoordAction ID: 0173513-190628063928782-oozie-oozi-C@174

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0178959-190629112435711-oozie-oozi-W@:start:                                  OK        -                      OK         -
------------------------------------------------------------------------------------------------------------------------------------
0178959-190629112435711-oozie-oozi-W@virtual                   KILLED    0178960-190629112435711-oozie-oozi-WSUSPENDED

After dig into logs, i have found the error:

2019-07-01 14:53:35,067  WARN ActionCheckXCommand:523 - SERVER[hn1.cloudapp.net] USER[sshuser] GROUP[-] TOKEN[] APP[E_virtualWf] JOB[0178960-190629112435711-oozie-oozi-W] ACTION[0178960-190629112435711-oozie-oozi-W@hbaseDelete] Exception while executing check(). Error Code [JA009], Message[JA009: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load history file wasbs://hbase@hdpsa.blob.core.windows.net/mr-history/tmp/sshuser/job_1561807385110_11674-1561992748186-sshuser-oozie%3Alauncher%3AT%3Djava%3AW%3DE_virtualWf-1561992812433-1-0-SUCCEEDED-default-1561992804743.jhist

I have checked the permission

rwxrwxrwx - /mr-history and

rwxrwxrwxt - /tmp

Sometime job working fine and sometime i'm getting above issue.

Could someone help me to find the solution?

Will be thankful to you.