Member since
04-06-2018
21
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 8159 | 07-12-2018 03:06 AM |
07-13-2018
02:05 AM
The developer (customer side) who work with me on the cluster try to use Apache Airflow, and after one week, he can do what we need (workflow, emailing / alerting, re-run, ...) without the load of files into hdfs, Apache airflow is running in standalone mode and the web UI is better than Oozie UI. It seems a better solution than oozie, what do you think about this ? As it is an incubating project, I don't know if it's a good idea, but the web UI is good, it looks easy to manage, I didn't know this new project but I think Oozie is outdated compare to Airflow. For the moment Oozie is in stand-by, they will make a choice between oozie and airflow, but I must admit that Airflow looks a better solution.
... View more
07-12-2018
06:09 AM
Cool, so easy 🙂 With my workflow, do you have any idea about email sending ? I know we have an emailAction on Oozie, but it don't work, maybe a bad syntax, I think we must edit the "kill" and "end" section ?
... View more
07-12-2018
03:06 AM
Hi GeKas, Regarding the rest. First of all you don't have to be a scala developer to schedule a script in oozie :) Right, but as System / Big Data administrator it's usually not in my scope, but it's better to know for sure 😛 So, now It works with this xml syntax (workflow.xml), I found the correct way with shell action two days ago, and i've implemented several jobs with this workflow as it's a generic syntax as well with variables : <workflow-app name="${wfname}" xmlns="uri:oozie:workflow:0.5">
<start to="shellAction"/>
<action name="shellAction">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>${command}</exec>
<file>${scriptfile}</file>
<file>${jarfile}</file>
<capture-output/>
</shell>
<ok to="end"/>
<error to="killAction"/>
</action>
<kill name="killAction">
<message>"Killed job due to error"</message>
</kill>
<end name="end"/>
</workflow-app> I have my job.properties with my variables locally, I put the workflow.xml in HDFS in the specified directory, and the jar into a subdirectory named "lib". It execute the spark shell without error, it's perfect ! Thank you for your support for the workflow, it's not easy to understand at the beginning. I still don't understand all the syntax, for example I need to add an action into my workflow for send emails, I found email actions but it does not work (I don't have a smtp server for the moment, but it can fail on email sending it doesn't matter, I juste want to have the correct syntax without oozie errors first). Now, I try to schedule a job with a coordinator, but it doesn't work, I have an issue with timezone, I put "Europe/Paris" in the timezone field, but it's not the correct time for execution, always a difference between what I want and the time printed. I've already configured the timezone in hue configuration, but there is a difference, Hue and Oozie not printed the same time (I think Oozie is UTC, and probably HUE in my timezone) And my last question, is there a way to record my submitted job into Hue ? For example, I've write on my own the workflow and I want to submit it to Hue (easier for the customer).
... View more
07-09-2018
08:05 AM
Hi, I changed my jar and the workflow : <workflow-app name="DirectSales-WF" xmlns="uri:oozie:workflow:0.5">
<start to="spark-4383"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-4383">
<spark xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
<master>yarn-cluster</master>
<mode>client</mode>
<name>DirectSales</name>
<class>com.budgetbox.decisionnel.aggregator.launcher.batch.AggregatorDataFlowBatch</class>
<jar>aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar</jar>
<spark-opts>--conf spark.ui.retainedJobs=10 --conf spark.ui.retainedStages=10 --conf spark.worker.ui.retainedExecutors=10 --conf spark.worker.ui.retainedDrivers=10 --conf spark.sql.ui.retainedExecutions=10 --conf spark.streaming.ui.retainedBatches=10 --conf spark.streaming.ui.retainedBatches=100 --conf spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:+UseCompressedOops -Dlog4j.configuration=log4j.xml --driver-memory 8g --num-executors 6 --executor-cores 12 --executor-memory 30G --conf spark.yarn.historyServer.address=http://dec-bb-dl01.bbox-dec.lab.oxv.fr:18088 --conf spark.eventLog.dir=hdfs://dec-bb-dl01.bbox-dec.lab.oxv.fr:8020/user/spark/applicationHistory --conf spark.eventLog.enabled=true</spark-opts>
<arg>--action</arg>
<arg>DIRECT_SALE_CONTACTS_UC</arg>
<arg>--mongoHost</arg>
<arg>dec-bb-dm01</arg>
<arg>--mongoUser</arg>
<arg>XXXXXXXXXXXXXXXX</arg>
<arg>--mongoPassword</arg>
<arg>XXXXXXXXXXXXXXXX</arg>
<arg>--mongoDB</arg>
<arg>XXXXXXXXXXXXXXXX</arg>
<arg>--mongoCollection</arg>
<arg>agg.action</arg>
<arg>--dayToReplay</arg>
<arg>9999</arg>
<arg>--hiveSchema</arg>
<arg>decisionnel</arg>
<file>/user/hdfs/oozie_test/workflow.xml#workflow.xml</file>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app> Now I can start the workflow with oozie : [16:36:21]hdfs@mst01:/data/tmp/oozie_test$ oozie job -oozie http://$(hostname -f):11000/oozie -config ./job.properties -run
job: 0000017-180627191420891-oozie-oozi-W Now, I have an error, the oozie job appear in "RUNNING" state, but when i open the YARN Resource Manager, it appears in FAILED state. Is it normal to see MAPREDUCE type in Application Type section ? Here is the log of the fail status : 16:37:17.619 [main] DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JvmMetrics, JVM related metrics etc.
16:37:17.630 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsSubmitted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsCompleted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsFailed with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsKilled with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.637 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsPreparing with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.jobsRunning with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsLaunched with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsCompleted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsFailed with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsKilled with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsRunning with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.641 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.mapsWaiting with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesLaunched with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesCompleted with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesFailed with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.642 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableCounterInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesKilled with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.643 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesRunning with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.643 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.reducesWaiting with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[])
16:37:17.644 [main] DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl - MRAppMetrics, MR App Metrics
16:37:17.650 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Created MRAppMaster for application appattempt_1530119712021_0062_000001
16:37:17.672 [main] DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:303)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:328)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:641)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1441)
16:37:17.683 [main] DEBUG org.apache.hadoop.util.Shell - setsid exited with exit code 0
16:37:17.760 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
16:37:17.761 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
16:37:17.761 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
16:37:17.762 [main] DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl - UgiMetrics, User and group related metrics
16:37:17.791 [main] DEBUG org.apache.hadoop.security.Groups - Creating new Groups object
16:37:17.812 [main] DEBUG org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
16:37:17.815 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login
16:37:17.815 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login commit
16:37:17.817 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - using local user:UnixPrincipal: yarn
16:37:17.817 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - Using user: "UnixPrincipal: yarn" with name yarn
16:37:17.817 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - User entry: "yarn"
16:37:17.823 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - UGI loginUser:yarn (auth:SIMPLE)
16:37:17.823 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Executing with tokens:
16:37:18.034 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Kind: YARN_AM_RM_TOKEN, Service: , Ident: 00 00 01 64 42 3d ed 15 00 00 00 3e 00 00 00 01 91 fd 1a 75
16:37:18.035 [main] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Kind: RM_DELEGATION_TOKEN, Service: 10.100.101.11:8032,10.100.101.12:8032, Ident: 00 04 68 64 66 73 0e 6f 6f 7a 69 65 20 6d 72 20 74 6f 6b 65 6e 05 6f 6f 7a 69 65 8a 01 64 7f 79 77 b9 8a 01 64 a3 85 fb b9 70 8f e5
16:37:18.036 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:hdfs (auth:SIMPLE) from:org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
16:37:18.037 [main] DEBUG org.apache.hadoop.service.AbstractService - Service: org.apache.hadoop.mapreduce.v2.app.MRAppMaster entered state INITED
16:37:18.073 [main] DEBUG - address: dec-bb-dl05.bbox-dec.lab.oxv.fr/10.100.101.15 isLoopbackAddress: false, with host 10.100.101.15 dec-bb-dl05.bbox-dec.lab.oxv.fr
16:37:18.080 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
16:37:18.085 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - Java version: 8
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
16:37:18.086 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - Javassist: unavailable
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /yarn/nm/usercache/hdfs/appcache/application_1530119712021_0062/container_e42_1530119712021_0062_01_000001/./tmp (java.io.tmpdir)
16:37:18.087 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
16:37:18.088 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
16:37:18.089 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.tmpdir: /yarn/nm/usercache/hdfs/appcache/application_1530119712021_0062/container_e42_1530119712021_0062_01_000001/./tmp (java.io.tmpdir)
16:37:18.089 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.netty.workdir: /yarn/nm/usercache/hdfs/appcache/application_1530119712021_0062/container_e42_1530119712021_0062_01_000001/./tmp (io.netty.tmpdir)
16:37:18.092 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 127.0.0.1)
16:37:18.092 [main] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 1024
16:37:18.194 [main] ERROR org.apache.hadoop.mapreduce.v2.app.MRAppMaster - Error starting MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf(Ljava/lang/String;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/htrace/core/HTraceConfiguration;
at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:696)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:680)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:158)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:503)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:287)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
16:37:18.196 [main] INFO org.apache.hadoop.util.ExitUtil - Exiting with status 1
16:37:18.217 [Thread-1] INFO org.apache.hadoop.mapreduce.v2.app.MRAppMaster - MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.
16:37:18.218 [Thread-1] DEBUG org.apache.hadoop.service.AbstractService - Service: org.apache.hadoop.mapreduce.v2.app.MRAppMaster entered state STOPPED
16:37:18.218 [Thread-1] DEBUG org.apache.hadoop.service.CompositeService - org.apache.hadoop.mapreduce.v2.app.MRAppMaster: stopping services, size=0
16:37:18.218 [Finalizer] DEBUG org.apache.hadoop.fs.azure.NativeAzureFileSystem - finalize() called. Any idea ? I don't understand why I get this exception, (I think this is the root cause, but it's not very easy to understand ... Java's exceptions are always a pleasure to read ='D )
... View more
07-04-2018
07:58 AM
I worked on the subject this afternoon and read oozie official documentation, I tried to use xml file to define my spark workflow, I've called my configuration file job.properties (it's just a try, I know I can have a job.properties file and a workflow.xml file which get properties from the first and use them) : <workflow-app name="Consolidator-WF" xmlns="uri:oozie:workflow:0.3">
<action name="[NODE-NAME]">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>dec-bb-dl01.bbox-dec.lab.oxv.fr:8021</job-tracker>
<name-node>dec-bb-dl01.bbox-dec.lab.oxv.fr:8020</name-node>
<configuration>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
<master>yarn-cluster</master>
<mode>client</mode>
<name>Consolidator</name>
<class>com.budgetbox.decisionnel.aggregator.launcher.batch.AggregatorDataFlowBatch</class>
<jar>/data/tmp/oozie_test/aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar</jar>
<spark-opts>--conf spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote -XX:+UseG1GC --conf spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties -XX:+UseG1GC -XX:+UseCompressedOops --spark.yarn.historyServer.address=http://dec-bb-dl01.bbox-dec.lab.oxv.fr:18088 --spark.eventLog.dir=hdfs://dec-bb-dl01.bbox-dec.lab.oxv.fr:8020/user/spark/applicationHistory --spark.eventLog.enabled=true --num-executors 3 --executor-cores 4 --executor-memory 30GB --driver-memory 8g --action FILE_CONSOLIDATOR --pathAction /data/bbox/realtime/action --pathSessionActivity /data/bbox/realtime/session-activity --pathFlags /data/bbox/flags --consolidateSuffix -consolidate --writeRepartition 48</spark-opts>
</spark>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Oozie job Consolidator failed ...</message>
</kill>
<end name="end"/>
</workflow-app> Now, I have an error : [16:54:12]root@mst01:/data/tmp/oozie_test$ oozie job -oozie http://$(hostname -f):11000/oozie -config ./job.properties -run
Error: E0405 : E0405: Submission request doesn''t have any application or lib path I don't know what is this error, any idea ?
... View more
07-04-2018
03:18 AM
Thanks for the answer, don't worry, you help me as you can and it's very cool 😉 So, I understand what you explained, and I want to try with "Shell action" first, it will be easier for my customer, and if needed I will try with Spark action. But it's difficult as we don't work with "best practices", let me explain : We have our Cloudera cluster, on each node we have a Purestorage storage bay mounted in NFS (on /data) I put my files into the storage bay : [12:00:46]root@mst01:/data/tmp/oozie_test$ ll
total 76409
-rwxr-xr-x 1 hdfs hadoop 78241636 Jun 13 16:47 aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar*
-rwxrwxr-x 1 hdfs hadoop 708 Jul 4 11:59 runConsolidator.sh* As you said, when it's executed, we don't know on which yarn node the command will be executed. So, with this storage bay mounted on each node, it doesn't matter on which node it's executed (I think :p) In the spark-submit bash script, we have (as mentionned before) : [12:08:59]root@mst01:/data/tmp/oozie_test$ cat runConsolidator.sh
export HADOOP_USER_NAME=hdfs;spark-submit --master yarn-cluster --class=com.budgetbox.decisionnel.aggregator.launcher.batch.AggregatorDataFlowBatch --conf "spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote -XX:+UseG1GC " --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties -XX:+UseG1GC -XX:+UseCompressedOops" --num-executors 3 --executor-cores 4 --executor-memory 30GB --driver-memory 8g /data/tmp/oozie_test/aggregator-dataflow-assembly-3.0.7-SNAPSHOT.jar --action FILE_CONSOLIDATOR --pathAction /data/bbox/realtime/action --pathSessionActivity /data/bbox/realtime/session-activity --pathFlags /data/bbox/flags --consolidateSuffix -consolidate --writeRepartition 48 On Hue, I tried to create a new Shell action, but I don't know what's the input : What's the input ? I tried to type "runConsolidator.sh" (my spark submit bash script), and add "files" property with the absolute local path of my script, but it failed with this error message : 2018-07-04 12:05:53,304 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[] No results found
2018-07-04 12:05:53,322 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] Start action [0000004-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-07-04 12:05:53,322 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] [***0000004-180627191420891-oozie-oozi-W@:start:***]Action status=DONE
2018-07-04 12:05:53,322 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] [***0000004-180627191420891-oozie-oozi-W@:start:***]Action updated in DB!
2018-07-04 12:05:53,359 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] No results found
2018-07-04 12:05:53,375 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@:start:
2018-07-04 12:05:53,375 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W
2018-07-04 12:05:53,393 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] Start action [0000004-180627191420891-oozie-oozi-W@shell-942e] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-07-04 12:05:53,843 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] checking action, hadoop job ID [job_1530119712021_0042] status [RUNNING]
2018-07-04 12:05:53,845 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] [***0000004-180627191420891-oozie-oozi-W@shell-942e***]Action status=RUNNING
2018-07-04 12:05:53,845 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] [***0000004-180627191420891-oozie-oozi-W@shell-942e***]Action updated in DB!
2018-07-04 12:05:53,853 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@shell-942e
2018-07-04 12:06:10,670 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] callback for action [0000004-180627191420891-oozie-oozi-W@shell-942e]
2018-07-04 12:06:10,768 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] action completed, external ID [job_1530119712021_0042]
2018-07-04 12:06:10,775 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-07-04 12:06:10,796 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] ERROR is considered as FAILED for SLA
2018-07-04 12:06:10,817 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] No results found
2018-07-04 12:06:10,837 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] Start action [0000004-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-07-04 12:06:10,837 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] [***0000004-180627191420891-oozie-oozi-W@Kill***]Action status=DONE
2018-07-04 12:06:10,837 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] [***0000004-180627191420891-oozie-oozi-W@Kill***]Action updated in DB!
2018-07-04 12:06:10,917 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@Kill
2018-07-04 12:06:10,917 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[0000004-180627191420891-oozie-oozi-W@shell-942e] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W@shell-942e
2018-07-04 12:06:10,917 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000004-180627191420891-oozie-oozi-W Any idea ? Can you help me to implemente my spark-submit action (with or without shell action, I just want to make it work lol) ? As I'm big data administrator, I know how the oozie process work, but I've never schedule a job as I'm not a scala developer unfortunately.
... View more
07-03-2018
01:37 AM
Any idea ? I need help about this subject, I'm not sure I can use only Hue or if I must write workflow.xml and job.properties. For me it's simple, I have a shell script with spark-submit, but I don't know how to execute it with oozie outside ofHDFS (as oozie use hdfs)
... View more
06-29-2018
04:53 AM
Hi GeKas, Thank you for your help Yes, I need to execute this job (as spark action or into a script with shell action) which is on local filesystem. As you can see some of the arguments passed to the job are local directories. Which files need I to put to HDFS ? This is the error message I get when I submit the workflow into a shell action : 2018-06-29 12:08:44,611 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No results found
2018-06-29 12:08:44,629 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] Start action [0000002-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action status=DONE
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action updated in DB!
2018-06-29 12:08:44,683 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No results found
2018-06-29 12:08:44,695 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@:start:
2018-06-29 12:08:44,696 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:44,724 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Start action [0000002-180627191420891-oozie-oozi-W@shell-a54d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:45,149 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] checking action, hadoop job ID [job_1530119712021_0014] status [RUNNING]
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action status=RUNNING
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action updated in DB!
2018-06-29 12:08:45,163 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d
2018-06-29 12:08:59,064 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] callback for action [0000002-180627191420891-oozie-oozi-W@shell-a54d]
2018-06-29 12:08:59,159 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] action completed, external ID [job_1530119712021_0014]
2018-06-29 12:08:59,164 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-06-29 12:08:59,185 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] ERROR is considered as FAILED for SLA
2018-06-29 12:08:59,208 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No results found
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] Start action [0000002-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action status=DONE
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action updated in DB!
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@Kill
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d I don't understand how I can make this work
... View more
06-29-2018
03:22 AM
Yes exactly, I need to execute my script (which contain spark job execution command) with Oozie, but my spark job (as mentionned before) use some files which are on local filesystem (not in hdfs) In HDFS, which files need I to put ? I tried to put my script into /tmp under hdfs and execute it, but I get this error message : 2018-06-29 12:08:44,611 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No results found 2018-06-29 12:08:44,629 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] Start action [0000002-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action status=DONE 2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action updated in DB! 2018-06-29 12:08:44,683 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No results found 2018-06-29 12:08:44,695 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@:start: 2018-06-29 12:08:44,696 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W 2018-06-29 12:08:44,724 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Start action [0000002-180627191420891-oozie-oozi-W@shell-a54d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-06-29 12:08:45,149 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] checking action, hadoop job ID [job_1530119712021_0014] status [RUNNING] 2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action status=RUNNING 2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action updated in DB! 2018-06-29 12:08:45,163 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d 2018-06-29 12:08:59,064 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] callback for action [0000002-180627191420891-oozie-oozi-W@shell-a54d] 2018-06-29 12:08:59,159 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] action completed, external ID [job_1530119712021_0014] 2018-06-29 12:08:59,164 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] 2018-06-29 12:08:59,185 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] ERROR is considered as FAILED for SLA 2018-06-29 12:08:59,208 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No results found 2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] Start action [0000002-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action status=DONE 2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action updated in DB! 2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@Kill 2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W 2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d My job crashed (it's written killed in status failed) and I don't know how to make it work
... View more
06-29-2018
03:22 AM
Yes exactly, I need to execute my script (which contain spark job execution command) with Oozie, but my spark job (as mentionned before) use some files which are on local filesystem (not in hdfs) In HDFS, which files need I to put ? I tried to put my script into /tmp under hdfs and execute it, but I get this error message : 2018-06-29 12:08:44,611 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No results found
2018-06-29 12:08:44,629 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] Start action [0000002-180627191420891-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action status=DONE
2018-06-29 12:08:44,630 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] [***0000002-180627191420891-oozie-oozi-W@:start:***]Action updated in DB!
2018-06-29 12:08:44,683 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No results found
2018-06-29 12:08:44,695 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@:start:
2018-06-29 12:08:44,696 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:44,724 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Start action [0000002-180627191420891-oozie-oozi-W@shell-a54d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:45,149 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] checking action, hadoop job ID [job_1530119712021_0014] status [RUNNING]
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action status=RUNNING
2018-06-29 12:08:45,151 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] [***0000002-180627191420891-oozie-oozi-W@shell-a54d***]Action updated in DB!
2018-06-29 12:08:45,163 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d
2018-06-29 12:08:59,064 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] callback for action [0000002-180627191420891-oozie-oozi-W@shell-a54d]
2018-06-29 12:08:59,159 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] action completed, external ID [job_1530119712021_0014]
2018-06-29 12:08:59,164 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-06-29 12:08:59,185 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] ERROR is considered as FAILED for SLA
2018-06-29 12:08:59,208 INFO org.apache.oozie.service.JPAService: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No results found
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] Start action [0000002-180627191420891-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action status=DONE
2018-06-29 12:08:59,231 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[admin] GROUP[-] TOKEN[] APP[My Workflow] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] [***0000002-180627191420891-oozie-oozi-W@Kill***]Action updated in DB!
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@Kill
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W
2018-06-29 12:08:59,298 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[dec-bb-dl01.bbox-dec.lab.oxv.fr] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-180627191420891-oozie-oozi-W] ACTION[0000002-180627191420891-oozie-oozi-W@shell-a54d] No Notification URL is defined. Therefore nothing to notify for job 0000002-180627191420891-oozie-oozi-W@shell-a54d My job crashed (it's written killed in status failed) and I don't know how to make it work
... View more