Reply
New Contributor
Posts: 2
Registered: ‎10-23-2017

How to Schedule Spark Streaming Jobs with Oozie

[ Edited ]

Hi,

 

Iam trying to schedule a script file in Oozie.In which I need to kill the existing spark streaming  job and restart the job after interval of 5mins so as to capture new records in the output directory.Iam using CDH 5.13(parcels)

 

Can someone help me to resolve this issue ?

 

The error is 

017-11-15 15:30:35,962 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2017-11-15 15:30:36,029 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] ERROR is considered as FAILED for SLA

 

Entire Log File

2017-11-15 15:30:01,575 INFO org.apache.oozie.service.JPAService: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[] No results found
2017-11-15 15:30:01,629 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@:start:] Start action [0000003-171115152053257-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-11-15 15:30:01,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@:start:] [***0000003-171115152053257-oozie-oozi-W@:start:***]Action status=DONE
2017-11-15 15:30:01,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@:start:] [***0000003-171115152053257-oozie-oozi-W@:start:***]Action updated in DB!
2017-11-15 15:30:01,751 INFO org.apache.oozie.service.JPAService: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@:start:] No results found
2017-11-15 15:30:01,791 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[master1.mldev.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000003-171115152053257-oozie-oozi-W@:start:
2017-11-15 15:30:01,791 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[master1.mldev.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000003-171115152053257-oozie-oozi-W
2017-11-15 15:30:01,875 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] Start action [0000003-171115152053257-oozie-oozi-W@shell-baab] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-11-15 15:30:03,636 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] checking action, hadoop job ID [job_1510759169185_0003] status [RUNNING]
2017-11-15 15:30:03,644 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] [***0000003-171115152053257-oozie-oozi-W@shell-baab***]Action status=RUNNING
2017-11-15 15:30:03,644 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] [***0000003-171115152053257-oozie-oozi-W@shell-baab***]Action updated in DB!
2017-11-15 15:30:03,664 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[master1.mldev.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] No Notification URL is defined. Therefore nothing to notify for job 0000003-171115152053257-oozie-oozi-W@shell-baab
2017-11-15 15:30:35,431 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[master1.mldev.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] callback for action [0000003-171115152053257-oozie-oozi-W@shell-baab]
2017-11-15 15:30:35,934 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] action completed, external ID [job_1510759169185_0003]
2017-11-15 15:30:35,962 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2017-11-15 15:30:36,029 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] ERROR is considered as FAILED for SLA
2017-11-15 15:30:36,103 INFO org.apache.oozie.service.JPAService: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] No results found
2017-11-15 15:30:36,146 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@Kill] Start action [0000003-171115152053257-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-11-15 15:30:36,147 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@Kill] [***0000003-171115152053257-oozie-oozi-W@Kill***]Action status=DONE
2017-11-15 15:30:36,147 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[master1.mldev.local] USER[jjohnson] GROUP[-] TOKEN[] APP[Oozie Test] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@Kill] [***0000003-171115152053257-oozie-oozi-W@Kill***]Action updated in DB!
2017-11-15 15:30:36,335 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[master1.mldev.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000003-171115152053257-oozie-oozi-W@Kill
2017-11-15 15:30:36,335 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[master1.mldev.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000003-171115152053257-oozie-oozi-W
2017-11-15 15:30:36,336 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[master1.mldev.local] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-171115152053257-oozie-oozi-W] ACTION[0000003-171115152053257-oozie-oozi-W@shell-baab] No Notification URL is defined. Therefore nothing to notify for job 0000003-171115152053257-oozie-oozi-W@shell-baab

 

Highlighted
New Contributor
Posts: 1
Registered: ‎09-20-2018

Re: How to Schedule Spark Streaming Jobs with Oozie

You can use Spark Action or shell Action which has spark-submit command