Support Questions

JSenzier · ‎06-29-2018

Hi,

I try to create a workflow into oozie with a spark job, I read the documentation with the two files, job.properties and workflow.xml, but I have a problem :

My spark job use local file, so I don't want to use HDFS to execute it.

export HADOOP_USER_NAME=hdfs;spark-submit --master yarn-cluster --class=com.budgetbox.decisionnel.aggregator.launcher.batch.AggregatorDataFlowBatch --conf "spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote -XX:+UseG1GC " --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties -XX:+UseG1GC -XX:+UseCompressedOops" --num-executors 3 --executor-cores 4 --executor-memory 30GB --driver-memory 8g /root/scripts/aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar --action FILE_CONSOLIDATOR --pathAction /data/bbox/realtime/action --pathSessionActivity /data/bbox/realtime/session-activity --pathFlags /data/bbox/flags --consolidateSuffix -consolidate --writeRepartition 48

We've created a bash script and placed it in crontab, but we are thinking that oozie is a better solution.

It's hard to understand how to complete job.properties ans workflow.xml files because of local files used by my spark job.

Do I need to create a new Spark Workflow or can I just create a new shell action workflow and execute the script ?

Can you help me about this subject ?

JSenzier · ‎07-12-2018

Hi GeKas,

Regarding the rest. First of all you don't have to be a scala developer to schedule a script in oozie 🙂

Right, but as System / Big Data administrator it's usually not in my scope, but it's better to know for sure 😛

So, now It works with this xml syntax (workflow.xml), I found the correct way with shell action two days ago, and i've implemented several jobs with this workflow as it's a generic syntax as well with variables :

<workflow-app name="${wfname}" xmlns="uri:oozie:workflow:0.5">
    <start to="shellAction"/>
    <action name="shellAction">
        <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <exec>${command}</exec>
        <file>${scriptfile}</file>
        <file>${jarfile}</file>
        <capture-output/>
        </shell>
        <ok to="end"/>
        <error to="killAction"/>
    </action>
    <kill name="killAction">
    <message>"Killed job due to error"</message>
    </kill>
    <end name="end"/>
</workflow-app>

I have my job.properties with my variables locally, I put the workflow.xml in HDFS in the specified directory, and the jar into a subdirectory named "lib". It execute the spark shell without error, it's perfect !

Thank you for your support for the workflow, it's not easy to understand at the beginning.

I still don't understand all the syntax, for example I need to add an action into my workflow for send emails, I found email actions but it does not work (I don't have a smtp server for the moment, but it can fail on email sending it doesn't matter, I juste want to have the correct syntax without oozie errors first).

Now, I try to schedule a job with a coordinator, but it doesn't work, I have an issue with timezone, I put "Europe/Paris" in the timezone field, but it's not the correct time for execution, always a difference between what I want and the time printed.

I've already configured the timezone in hue configuration, but there is a difference, Hue and Oozie not printed the same time (I think Oozie is UTC, and probably HUE in my timezone)

And my last question, is there a way to record my submitted job into Hue ? For example, I've write on my own the workflow and I want to submit it to Hue (easier for the customer).

View solution in original post

GeKas · ‎06-29-2018

Just to understand, you don't want the spark job to use these files from HDFS but from your local system.

If that is the case, then you can create a shell action as you have mentioned.

First of all, put all required files on HDFS.

Then define these files in the shell action. Oozie will automatically download these files to the working directory of the node that job will be executed.

You don't have to manually distribute anything in advance to the nodes, oozie will take care of it, you just have to define these files in the job.

JSenzier · ‎06-29-2018

Yes exactly, I need to execute my script (which contain spark job execution command) with Oozie, but my spark job (as mentionned before) use some files which are on local filesystem (not in hdfs)

In HDFS, which files need I to put ?

I tried to put my script into /tmp under hdfs and execute it, but I get this error message :

2018-06-29 12:08:44,611 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No results found
2018-06-29 12:08:44,629 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] Start action [0000002-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action status=DONE
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action updated in DB!
2018-06-29 12:08:44,683 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No results found
2018-06-29 12:08:44,695 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@:start:
2018-06-29 12:08:44,696 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:44,724 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Start action [0000002-180627191420891-oozie-oozi-W@shell-a54d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:45,149 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] checking action, hadoop job ID [job_1530119712021_0014] status [RUNNING]
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action status=RUNNING
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action updated in DB!
2018-06-29 12:08:45,163 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d
2018-06-29 12:08:59,064 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] callback for action [0000002-180627191420891-oozie-oozi-W@shell-a54d]
2018-06-29 12:08:59,159 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] action completed, external ID [job_1530119712021_0014]
2018-06-29 12:08:59,164 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-06-29 12:08:59,185 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] ERROR is considered as FAILED for SLA
2018-06-29 12:08:59,208 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No results found
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] Start action [0000002-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action status=DONE
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action updated in DB!
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@Kill
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d

My job crashed (it's written killed in status failed) and I don't know how to make it work

JSenzier · ‎06-29-2018

Yes exactly, I need to execute my script (which contain spark job execution command) with Oozie, but my spark job (as mentionned before) use some files which are on local filesystem (not in hdfs)

In HDFS, which files need I to put ?

I tried to put my script into /tmp under hdfs and execute it, but I get this error message :

2018-06-29 12:08:44,611 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No results found
2018-06-29 12:08:44,629 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] Start action [0000002-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action status=DONE
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action updated in DB!
2018-06-29 12:08:44,683 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No results found
2018-06-29 12:08:44,695 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@:start:
2018-06-29 12:08:44,696 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:44,724 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Start action [0000002-180627191420891-oozie-oozi-W@shell-a54d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:45,149 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] checking action, hadoop job ID [job_1530119712021_0014] status [RUNNING]
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action status=RUNNING
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action updated in DB!
2018-06-29 12:08:45,163 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d
2018-06-29 12:08:59,064 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] callback for action [0000002-180627191420891-oozie-oozi-W@shell-a54d]
2018-06-29 12:08:59,159 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] action completed, external ID [job_1530119712021_0014]
2018-06-29 12:08:59,164 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-06-29 12:08:59,185 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] ERROR is considered as FAILED for SLA
2018-06-29 12:08:59,208 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No results found
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] Start action [0000002-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action status=DONE
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action updated in DB!
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@Kill
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d

My job crashed (it's written killed in status failed) and I don't know how to make it work

JSenzier · ‎06-29-2018

Hi GeKas,

Thank you for your help

Yes, I need to execute this job (as spark action or into a script with shell action) which is on local filesystem. As you can see some of the arguments passed to the job are local directories.

Which files need I to put to HDFS ?

This is the error message I get when I submit the workflow into a shell action :

2018-06-29 12:08:44,611 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No results found
2018-06-29 12:08:44,629 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] Start action [0000002-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action status=DONE
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action updated in DB!
2018-06-29 12:08:44,683 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No results found
2018-06-29 12:08:44,695 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@:start:
2018-06-29 12:08:44,696 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:44,724 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Start action [0000002-180627191420891-oozie-oozi-W@shell-a54d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:45,149 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] checking action, hadoop job ID [job_1530119712021_0014] status [RUNNING]
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action status=RUNNING
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action updated in DB!
2018-06-29 12:08:45,163 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d
2018-06-29 12:08:59,064 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] callback for action [0000002-180627191420891-oozie-oozi-W@shell-a54d]
2018-06-29 12:08:59,159 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] action completed, external ID [job_1530119712021_0014]
2018-06-29 12:08:59,164 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-06-29 12:08:59,185 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] ERROR is considered as FAILED for SLA
2018-06-29 12:08:59,208 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No results found
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] Start action [0000002-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action status=DONE
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action updated in DB!
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@Kill
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d

I don't understand how I can make this work

JSenzier · ‎07-03-2018

Any idea ? I need help about this subject, I'm not sure I can use only Hue or if I must write workflow.xml and job.properties.

For me it's simple, I have a shell script with spark-submit, but I don't know how to execute it with oozie outside ofHDFS (as oozie use hdfs)

GeKas · ‎07-03-2018

Hi,

sorry if the reply is not very clear. It is written in hurry. I will try to expand it later.

First of all HUE provides a very good interface to write oozie jobs and hardly you will need to write the job xml your own.

You have a shell script with spark-submit. Spark submit will execute something (a jar or python file). When you define a "shell action", all files (used as parameters) should exist on the local filesystem (not in HDFS).

If for example in a shel action you try "cat /etc/hosts" you will get the /etc/hosts file of the YARN node, where this shell action will be executed.

If you have a file in HDFS (e.g. my_hosts) and you define it as "File" in shell action then oozie will download this file automatically into the working directory. Working directory is a random directory in YARN node's filesystem, which leaves while this yarn job is being executed. So, if you use the command "cat ./my_hosts", then wou will get the contents of the "my_hosts" which is downloaded to the working directory.

In general is not very good idea to work with files on Slave Nodes, because you don't know on which yarn node this command will be executed each time. Unless you are sure tha you control it and you have deployed all required files to all nodes. Of course we are not discussing about temporary files that you may create during the execution, but for files with like metadata or configuration or files with results that you want to use after.

IMHO it is always better to have these files in HDFS and send the results back to HDFS, so they will be easily accessible by other actions.

JSenzier · ‎07-04-2018

Thanks for the answer, don't worry, you help me as you can and it's very cool 😉

So, I understand what you explained, and I want to try with "Shell action" first, it will be easier for my customer, and if needed I will try with Spark action.

But it's difficult as we don't work with "best practices", let me explain :

We have our Cloudera cluster, on each node we have a Purestorage storage bay mounted in NFS (on /data)

I put my files into the storage bay :

[12:00:46]root@mst01:/data/tmp/oozie_test$ ll
total 76409
-rwxr-xr-x 1 hdfs hadoop 78241636 Jun 13 16:47 aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar*
-rwxrwxr-x 1 hdfs hadoop      708 Jul  4 11:59 runConsolidator.sh*

As you said, when it's executed, we don't know on which yarn node the command will be executed. So, with this storage bay mounted on each node, it doesn't matter on which node it's executed (I think :p)

In the spark-submit bash script, we have (as mentionned before) :

[12:08:59]root@mst01:/data/tmp/oozie_test$ cat runConsolidator.sh
export HADOOP_USER_NAME=hdfs;spark-submit --master yarn-cluster --class=com.budgetbox.decisionnel.aggregator.launcher.batch.AggregatorDataFlowBatch --conf "spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote -XX:+UseG1GC " --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties  -XX:+UseG1GC -XX:+UseCompressedOops" --num-executors 3 --executor-cores 4 --executor-memory 30GB  --driver-memory 8g /data/tmp/oozie_test/aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar --action FILE_CONSOLIDATOR --pathAction /data/bbox/realtime/action --pathSessionActivity /data/bbox/realtime/session-activity --pathFlags /data/bbox/flags --consolidateSuffix -consolidate --writeRepartition 48

On Hue, I tried to create a new Shell action, but I don't know what's the input :

What's the input ?

I tried to type "runConsolidator.sh" (my spark submit bash script), and add "files" property with the absolute local path of my script, but it failed with this error message :

2018-07-04 12:05:53,304 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[] No results found
2018-07-04 12:05:53,322 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] Start action [0000004-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-07-04 12:05:53,322 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] [***0000004-180627191420891-oozie-oozi-W@:start:***]Action status=DONE
2018-07-04 12:05:53,322 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] [***0000004-180627191420891-oozie-oozi-W@:start:***]Action updated in DB!
2018-07-04 12:05:53,359 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] No results found
2018-07-04 12:05:53,375 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@:start:
2018-07-04 12:05:53,375 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W
2018-07-04 12:05:53,393 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] Start action [0000004-180627191420891-oozie-oozi-W@shell-942e] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-07-04 12:05:53,843 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] checking action, hadoop job ID [job_1530119712021_0042] status [RUNNING]
2018-07-04 12:05:53,845 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] [***0000004-180627191420891-oozie-oozi-W@shell-942e***]Action status=RUNNING
2018-07-04 12:05:53,845 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] [***0000004-180627191420891-oozie-oozi-W@shell-942e***]Action updated in DB!
2018-07-04 12:05:53,853 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@shell-942e
2018-07-04 12:06:10,670 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] callback for action [0000004-180627191420891-oozie-oozi-W@shell-942e]
2018-07-04 12:06:10,768 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] action completed, external ID [job_1530119712021_0042]
2018-07-04 12:06:10,775 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-07-04 12:06:10,796 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] ERROR is considered as FAILED for SLA
2018-07-04 12:06:10,817 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] No results found
2018-07-04 12:06:10,837 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] Start action [0000004-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-07-04 12:06:10,837 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] [***0000004-180627191420891-oozie-oozi-W@Kill***]Action status=DONE
2018-07-04 12:06:10,837 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] [***0000004-180627191420891-oozie-oozi-W@Kill***]Action updated in DB!
2018-07-04 12:06:10,917 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@Kill
2018-07-04 12:06:10,917 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@shell-942e
2018-07-04 12:06:10,917 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W

Any idea ?

Can you help me to implemente my spark-submit action (with or without shell action, I just want to make it work lol) ?

As I'm big data administrator, I know how the oozie process work, but I've never schedule a job as I'm not a scala developer unfortunately.

JSenzier · ‎07-04-2018

I worked on the subject this afternoon and read oozie official documentation, I tried to use xml file to define my spark workflow, I've called my configuration file job.properties (it's just a try, I know I can have a job.properties file and a workflow.xml file which get properties from the first and use them) :

<workflow-app name="Consolidator-WF" xmlns="uri:oozie:workflow:0.3">
    <action name="[NODE-NAME]">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>dec-bb-dl01.bbox-dec.lab.oxv.fr:8021</job-tracker>
            <name-node>dec-bb-dl01.bbox-dec.lab.oxv.fr:8020</name-node>
            <configuration>
                <property>
                    <name>mapred.compress.map.output</name>
                    <value>true</value>
                </property>
            </configuration>
            <master>yarn-cluster</master>
            <mode>client</mode>
            <name>Consolidator</name>
            <class>com.budgetbox.decisionnel.aggregator.launcher.batch.AggregatorDataFlowBatch</class>
            <jar>/data/tmp/oozie_test/aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar</jar>
            <spark-opts>--conf spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote -XX:+UseG1GC --conf spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties -XX:+UseG1GC -XX:+UseCompressedOops --spark.yarn.historyServer.address=http://dec-bb-dl01.bbox-dec.lab.oxv.fr:18088 --spark.eventLog.dir=hdfs://dec-bb-dl01.bbox-dec.lab.oxv.fr:8020/user/spark/applicationHistory --spark.eventLog.enabled=true --num-executors 3 --executor-cores 4 --executor-memory 30GB  --driver-memory 8g --action FILE_CONSOLIDATOR --pathAction /data/bbox/realtime/action --pathSessionActivity /data/bbox/realtime/session-activity --pathFlags /data/bbox/flags --consolidateSuffix -consolidate --writeRepartition 48</spark-opts>
        </spark>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Oozie job Consolidator failed ...</message>
    </kill>
    <end name="end"/>
</workflow-app>

Now, I have an error :

[16:54:12]root@mst01:/data/tmp/oozie_test$ oozie job -oozie http://$(hostname -f):11000/oozie -config ./job.properties -run
Error: E0405 : E0405: Submission request doesn''t have any application or lib path

I don't know what is this error, any idea ?

JSenzier · ‎07-09-2018

Hi,

I changed my jar and the workflow :

<workflow-app name="DirectSales-WF" xmlns="uri:oozie:workflow:0.5">
    <start to="spark-4383"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="spark-4383">
        <spark xmlns="uri:oozie:spark-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.compress.map.output</name>
                    <value>true</value>
                </property>
            </configuration>
            <master>yarn-cluster</master>
            <mode>client</mode>
            <name>DirectSales</name>
              <class>com.budgetbox.decisionnel.aggregator.launcher.batch.AggregatorDataFlowBatch</class>
            <jar>aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar</jar>
              <spark-opts>--conf spark.ui.retainedJobs=10 --conf spark.ui.retainedStages=10 --conf spark.worker.ui.retainedExecutors=10 --conf spark.worker.ui.retainedDrivers=10 --conf spark.sql.ui.retainedExecutions=10 --conf spark.streaming.ui.retainedBatches=10 --conf spark.streaming.ui.retainedBatches=100 --conf spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:+UseCompressedOops -Dlog4j.configuration=log4j.xml --driver-memory 8g --num-executors 6 --executor-cores 12 --executor-memory 30G --conf spark.yarn.historyServer.address=http://dec-bb-dl01.bbox-dec.lab.oxv.fr:18088 --conf spark.eventLog.dir=hdfs://dec-bb-dl01.bbox-dec.lab.oxv.fr:8020/user/spark/applicationHistory --conf spark.eventLog.enabled=true</spark-opts>
              <arg>--action</arg>
              <arg>DIRECT_SALE_CONTACTS_UC</arg>
              <arg>--mongoHost</arg>
              <arg>dec-bb-dm01</arg>
              <arg>--mongoUser</arg>
              <arg>XXXXXXXXXXXXXXXX</arg>
              <arg>--mongoPassword</arg>
              <arg>XXXXXXXXXXXXXXXX</arg>
              <arg>--mongoDB</arg>
              <arg>XXXXXXXXXXXXXXXX</arg>
              <arg>--mongoCollection</arg>
              <arg>agg.action</arg>
              <arg>--dayToReplay</arg>
              <arg>9999</arg>
              <arg>--hiveSchema</arg>
              <arg>decisionnel</arg>
            <file>/user/hdfs/oozie_test/workflow.xml#workflow.xml</file>
        </spark>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

Now I can start the workflow with oozie :

[16:36:21]hdfs@mst01:/data/tmp/oozie_test$ oozie job -oozie http://$(hostname -f):11000/oozie -config ./job.properties -run
job: 0000017-180627191420891-oozie-oozi-W

Now, I have an error, the oozie job appear in "RUNNING" state, but when i open the YARN Resource Manager, it appears in FAILED state.

Is it normal to see MAPREDUCE type in Application Type section ?

Here is the log of the fail status :

16:37:17.619 [main] DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JvmMetrics, JVM related metrics etc.
16:37:17.630 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsSubmitted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsCompleted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsFailed with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsKilled with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsPreparing with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsRunning with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsLaunched with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsCompleted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsFailed with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsKilled with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsRunning with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsWaiting with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesLaunched with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesCompleted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesFailed with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesKilled with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.643 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesRunning with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.643 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesWaiting with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.644 [main] DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl - MRAppMetrics, MR App Metrics
16:37:17.650 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Created MRAppMaster for application appattempt_1530119712021_0062_000001
16:37:17.672 [main] DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:303)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:328)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:641)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1441)
16:37:17.683 [main] DEBUG org.apache.hadoop.util.Shell - setsid exited with exit code 0
16:37:17.760 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
16:37:17.761 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
16:37:17.761 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
16:37:17.762 [main] DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl - UgiMetrics, User and group related metrics
16:37:17.791 [main] DEBUG org.apache.hadoop.security.Groups -  Creating new Groups object
16:37:17.812 [main] DEBUG org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
16:37:17.815 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login
16:37:17.815 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login commit
16:37:17.817 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - using local user:UnixPrincipal: yarn
16:37:17.817 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - Using user: "UnixPrincipal: yarn" with name yarn
16:37:17.817 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - User entry: "yarn"
16:37:17.823 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - UGI loginUser:yarn (auth:SIMPLE)
16:37:17.823 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Executing with tokens:
16:37:18.034 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Kind: YARN_AM_RM_TOKEN, Service: , Ident: 00 00 01 64 42 3d ed 15 00 00 00 3e 00 00 00 01 91 fd 1a 75
16:37:18.035 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Kind: RM_DELEGATION_TOKEN, Service: 10.100.101.11:8032,10.100.101.12:8032, Ident: 00 04 68 64 66 73 0e 6f 6f 7a 69 65 20 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 64 7f 79 77 b9 8a 01 64 a3 85 fb b9 70 8f e5
16:37:18.036 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:hdfs (auth:SIMPLE) from:org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
16:37:18.037 [main] DEBUG org.apache.hadoop.service.AbstractService - Service: org.apache.hadoop.mapreduce.v2.app.MRAppMaster entered state INITED
16:37:18.073 [main] DEBUG  - address: dec-bb-dl05.bbox-dec.lab.oxv.fr/10.100.101.15 isLoopbackAddress: false, with host 10.100.101.15 dec-bb-dl05.bbox-dec.lab.oxv.fr
16:37:18.080 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - Java version: 8
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - Javassist: unavailable
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes.  Please check the configuration for better performance.
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /yarn/nm/usercache/hdfs/appcache/application_1530119712021_0062/container_e42_1530119712021_0062_01_000001/./tmp (java.io.tmpdir)
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
16:37:18.088 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
16:37:18.089 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.tmpdir: /yarn/nm/usercache/hdfs/appcache/application_1530119712021_0062/container_e42_1530119712021_0062_01_000001/./tmp (java.io.tmpdir)
16:37:18.089 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.netty.workdir: /yarn/nm/usercache/hdfs/appcache/application_1530119712021_0062/container_e42_1530119712021_0062_01_000001/./tmp (io.netty.tmpdir)
16:37:18.092 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 127.0.0.1)
16:37:18.092 [main] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 1024
16:37:18.194 [main] ERROR org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Error starting MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf(Ljava/lang/String;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/htrace/core/HTraceConfiguration;
	at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:696)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:680)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:158)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:503)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:287)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
16:37:18.196 [main] INFO org.apache.hadoop.util.ExitUtil - Exiting with status 1
16:37:18.217 [Thread-1] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.
16:37:18.218 [Thread-1] DEBUG org.apache.hadoop.service.AbstractService - Service: org.apache.hadoop.mapreduce.v2.app.MRAppMaster entered state STOPPED
16:37:18.218 [Thread-1] DEBUG org.apache.hadoop.service.CompositeService - org.apache.hadoop.mapreduce.v2.app.MRAppMaster: stopping services, size=0
16:37:18.218 [Finalizer] DEBUG org.apache.hadoop.fs.azure.NativeAzureFileSystem - finalize() called.

Any idea ? I don't understand why I get this exception, (I think this is the root cause, but it's not very easy to understand ... Java's exceptions are always a pleasure to read ='D )

Cloudera Community

Support Questions

Oozie - Create a Spark workflow