Oozie - unable to kill workflow.
I am trying to kill an existing coordinator job and re-deploy it on oozie. I am running the command -
oozie job -run --config siftcuration.properties.
Here is my oozie properties file -
nameNode=hdfs://ip-<ip>:8020
jobTracker=ip-<ip>:8032
master=yarn
mode=client
queueName=default
#oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.use.system.libpath=true
oozie.wf.rerun.failnodes=true
runtimeEnvironment=aws_prod
appName=SiftCuration
appBaseDir=${nameNode}/user/${user.name}/oozieJobs/${appName}
oozie.coord.application.path=${appBaseDir}/workflow/coordinator.xml
etl.OverwriteOrAppend=Overwrite
spark.driver.memory=4g
spark.executor.memory=4g
spark.num.executors=3
spark.executor.cores=1
etl.default.date.format=yyyy-MM-dd
etl.run.date=yesterday
coordinatorKickDate=2024-03-14
When I run this command, it gives me a coordinator job ID. But when I check the coordinator status, I see that is is still in PREP state. It has been in PREP state for over 4 hours.
0000002-240312195311040-oozie-oozi-C SiftCuration PREP 1 DAY 2024-03-14 03:00 GMT
I tried checking the logs and I am seeing this issue -
2024-03-13 11:59:34,030 WARN KillXCommand:523 - SERVER[ip-<ip>.ec2.internal] USER[hadoop] GROUP[-] TOKEN[] APP[SiftCuration] JOB[0043858-220322202819429-oozie-oozi-W] ACTION[] E0725: Workflow instance can not be killed, 0043858-220322202819429-oozie-oozi-W, Error Code: E0725
This workflow ID belongs to the previous coordinator job that was already killed.
On checking the yarn logs for this workflow, I see this
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0043858-220322202819429-oozie-oozi-W@:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0043858-220322202819429-oozie-oozi-W@SnowflakeJobStartStoredProcedure OK application_1647980654756_172599SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0043858-220322202819429-oozie-oozi-W@ParameterGenerator OK application_1647980654756_172603SUCCEEDED -
------------------------------------------------------------------------------------------------------------------------------------
0043858-220322202819429-oozie-oozi-W@SiftCurationSparkETL KILLED application_1647980654756_172607RUNNING JA009
------------------------------------------------------------------------------------------------------------------------------------
It says application_1647980654756_172607 is running but I cannot find this application ID when I do a yarn logs -applicationId application_1647980654756_172607.
Could not locate application logs for application_1647980654756_172607
How do I fix this issue?
