oozie workflow error

[#1]

I am trying to run oozie workflow from Hue. I am trying to run a hive job in the workflow and that seem to be working but overall I am getting error in the oozie workflow.

Error:

Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [10096]

I have a jdbc driver in hdfs and

HDP 2.5.3

Hue version: 2.6.1-37

Log:

2017-03-06 15:47:46,557  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@:start:] Start action [0000005-170304185753616-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-03-06 15:47:46,557  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@:start:] [***0000005-170304185753616-oozie-oozi-W@:start:***]Action status=DONE
2017-03-06 15:47:46,557  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@:start:] [***0000005-170304185753616-oozie-oozi-W@:start:***]Action updated in DB!
2017-03-06 15:47:46,611  INFO WorkflowNotificationXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000005-170304185753616-oozie-oozi-W@:start:
2017-03-06 15:47:46,611  INFO WorkflowNotificationXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000005-170304185753616-oozie-oozi-W
2017-03-06 15:47:46,627  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] Start action [0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-03-06 15:47:48,278  INFO HiveActionExecutor:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] checking action, hadoop job ID [job_1488578860595_0014] status [RUNNING]
2017-03-06 15:47:48,281  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] [***0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean***]Action status=RUNNING
2017-03-06 15:47:48,281  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] [***0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean***]Action updated in DB!
2017-03-06 15:47:48,287  INFO WorkflowNotificationXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] No Notification URL is defined. Therefore nothing to notify for job 0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean
2017-03-06 15:48:51,019  INFO CallbackServlet:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] callback for action [0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean]
2017-03-06 15:48:51,151  INFO HiveActionExecutor:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] Trying to get job [job_1488578860595_0014], attempt [1]
2017-03-06 15:48:51,239  INFO HiveActionExecutor:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] Hadoop Jobs launched : [job_1488578860595_0015]
2017-03-06 15:48:51,241  INFO HiveActionExecutor:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] action completed, external ID [job_1488578860595_0014]
2017-03-06 15:48:51,255  WARN HiveActionExecutor:523 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [10096]
2017-03-06 15:48:51,281  INFO ActionEndXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] ERROR is considered as FAILED for SLA
2017-03-06 15:48:51,320  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@kill] Start action [0000005-170304185753616-oozie-oozi-W@kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-03-06 15:48:51,320  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@kill] [***0000005-170304185753616-oozie-oozi-W@kill***]Action status=DONE
2017-03-06 15:48:51,320  INFO ActionStartXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[eedc] GROUP[-] TOKEN[] APP[test] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@kill] [***0000005-170304185753616-oozie-oozi-W@kill***]Action updated in DB!
2017-03-06 15:48:51,373  INFO WorkflowNotificationXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@kill] No Notification URL is defined. Therefore nothing to notify for job 0000005-170304185753616-oozie-oozi-W@kill
2017-03-06 15:48:51,374  INFO WorkflowNotificationXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000005-170304185753616-oozie-oozi-W
2017-03-06 15:48:51,374  INFO WorkflowNotificationXCommand:520 - SERVER[phcv-hddev01.corp.incresearch.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170304185753616-oozie-oozi-W] ACTION[0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean] No Notification URL is defined. Therefore nothing to notify for job 0000005-170304185753616-oozie-oozi-W@daily_snapshot_clean

workflow.xml

<workflow-app name="test" xmlns="uri:oozie:workflow:0.4">
    <start to="daily_snapshot_clean"/>
    <action name="daily_snapshot_clean">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <script>/user/eedc/oozie/sbi/hive/daily/daily_snapshot_clean_sbi_master_start.hql</script>
        </hive>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

[#2 · Accepted solution]

@Prabhat Ratnala The error is coming from Hive rather than oozie.

Error 10096 is a SemanticException: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict

You can verify this by running your hive script without oozie. Once that work, then try running with oozie.

[#3]

You are right. It was error with hive. Setting hive.exec.dynamic.partition.mode=nonstrict resolved the issue. Thank you

[#4]

Hi All

I am new to the CDH and the similar error I am also getting when run Oozie job through Java client application. The error Logs displays like below

Main error is " No Notification URL is defined. Therefore nothing to notify for job 0000040-231010100945447-oozie-oozi-W@spark-node"

USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[] No results found
2023-12-22 03:33:19,121 INFO THREAD[qtp236002428-766348] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@:start:] Start action [0000040-231010100945447-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2023-12-22 03:33:19,125 INFO THREAD[qtp236002428-766348] org.apache.oozie.action.control.StartActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@:start:] Starting action
2023-12-22 03:33:19,130 INFO THREAD[qtp236002428-766348] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@:start:] [***0000040-231010100945447-oozie-oozi-W@:start:***]Action status=DONE
2023-12-22 03:33:19,131 INFO THREAD[qtp236002428-766348] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@:start:] [***0000040-231010100945447-oozie-oozi-W@:start:***]Action updated in DB!
2023-12-22 03:33:19,146 INFO THREAD[qtp236002428-766348] org.apache.oozie.action.control.StartActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@:start:] Action ended with external status [OK]
2023-12-22 03:33:19,173 INFO THREAD[qtp236002428-766348] org.apache.oozie.service.JPAService: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@:start:] No results found
2023-12-22 03:33:19,186 INFO THREAD[CallableQueue-7] org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[ln000xhstm0002.federated.fds] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000040-231010100945447-oozie-oozi-W@:start:
2023-12-22 03:33:19,186 INFO THREAD[CallableQueue-37] org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[ln000xhstm0002.federated.fds] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000040-231010100945447-oozie-oozi-W
2023-12-22 03:33:19,234 INFO THREAD[CallableQueue-7] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Start action [0000040-231010100945447-oozie-oozi-W@spark-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2023-12-22 03:33:19,238 INFO THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Starting action. Getting Action File System
2023-12-22 03:33:19,249 INFO THREAD[CallableQueue-7] org.apache.oozie.service.SparkConfigurationService: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] The atlas-application.properties does not exists for *: /etc/spark/conf/atlas-application.properties
2023-12-22 03:33:19,256 INFO THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Initializing sharelib excluder for action if specified.
2023-12-22 03:33:19,259 INFO THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.ShareLibExcluder: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Sharelib exclude pattern not configured, skipping.
2023-12-22 03:33:19,266 INFO THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Initializing sharelib excluder for action if specified.
2023-12-22 03:33:19,266 INFO THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.ShareLibExcluder: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Sharelib exclude pattern not configured, skipping.
2023-12-22 03:33:19,282 WARN THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] No credential properties found for action : 0000040-231010100945447-oozie-oozi-W@spark-node, cred : null
2023-12-22 03:33:19,287 WARN THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Invalid configuration value [null] defined for launcher max attempts count, using default [2].
2023-12-22 03:33:19,287 INFO THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.YarnACLHandler: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Not setting ACLs because mapreduce.cluster.acls.enabled is set to false
2023-12-22 03:33:19,652 INFO THREAD[CallableQueue-7] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] checking action, hadoop job ID [application_1696946942137_0018] status [RUNNING]
2023-12-22 03:33:19,654 INFO THREAD[CallableQueue-7] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] [***0000040-231010100945447-oozie-oozi-W@spark-node***]Action status=RUNNING
2023-12-22 03:33:19,654 INFO THREAD[CallableQueue-7] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP[${INDEXER_PARAMS}-ReindexJob-${USER_ROLE}-on-${USER_ENTITY_HOST}] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] [***0000040-231010100945447-oozie-oozi-W@spark-node***]Action updated in DB!
2023-12-22 03:33:19,666 INFO THREAD[CallableQueue-4] org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[ln000xhstm0002.federated.fds] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] No Notification URL is defined. Therefore nothing to notify for job 0000040-231010100945447-oozie-oozi-W@spark-node
2023-12-22 03:43:40,218 INFO THREAD[CallableQueue-6] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] action completed, external ID [application_1696946942137_0018]
2023-12-22 03:43:40,218 WARN THREAD[CallableQueue-6] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Launcher AM died, check Hadoop LOG for job [ln000xhstm0002.federated.fds:8032:application_1696946942137_0018]
2023-12-22 03:43:40,235 INFO THREAD[CallableQueue-6] org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] Action ended with external status [FAILED/KILLED]
2023-12-22 03:43:40,241 INFO THREAD[CallableQueue-6] org.apache.oozie.command.wf.ActionEndXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] ERROR is considered as FAILED for SLA
2023-12-22 03:43:40,262 INFO THREAD[CallableQueue-6] org.apache.oozie.service.JPAService: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] No results found
2023-12-22 03:43:40,280 INFO THREAD[CallableQueue-6] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@fail] Start action [0000040-231010100945447-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2023-12-22 03:43:40,284 INFO THREAD[CallableQueue-6] org.apache.oozie.action.control.KillActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@fail] Starting action
2023-12-22 03:43:40,288 INFO THREAD[CallableQueue-6] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@fail] [***0000040-231010100945447-oozie-oozi-W@fail***]Action status=DONE
2023-12-22 03:43:40,288 INFO THREAD[CallableQueue-6] org.apache.oozie.command.wf.ActionStartXCommand: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@fail] [***0000040-231010100945447-oozie-oozi-W@fail***]Action updated in DB!
2023-12-22 03:43:40,300 INFO THREAD[CallableQueue-6] org.apache.oozie.action.control.KillActionExecutor: SERVER[ln000xhstm0002.federated.fds] USER[myapp] GROUP[-] TOKEN[] APP["UPC"-ReindexJob-ASYNC-on-http://USER_ENTITY-mcom-mycell2.tbe.zeus.fds.com/UI] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@fail] Action ended with external status [OK]
2023-12-22 03:43:40,353 INFO THREAD[CallableQueue-5] org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[ln000xhstm0002.federated.fds] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@fail] No Notification URL is defined. Therefore nothing to notify for job 0000040-231010100945447-oozie-oozi-W@fail
2023-12-22 03:43:40,353 INFO THREAD[CallableQueue-8] org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[ln000xhstm0002.federated.fds] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[0000040-231010100945447-oozie-oozi-W@spark-node] No Notification URL is defined. Therefore nothing to notify for job 0000040-231010100945447-oozie-oozi-W@spark-node
2023-12-22 03:43:40,353 INFO THREAD[CallableQueue-5] org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[ln000xhstm0002.federated.fds] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000040-231010100945447-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000040-231010100945447-oozie-oozi-W

[#5]

@Raghunath_CDH As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.