11-28-2015 08:44 AM
I wrote a data dependency coordinator job that should name the job output in the date format mentioned in dataset uri. but the job is failing saying that outputDir variable is not resolved. I am posting my coordinator,workflow & job.properties. Let me know where things are going bad. Thanks in advance.
error:
2015-11-28 17:40:20,946 INFO ActionStartXCommand:539 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[0000033-151128171830337-oozie-bigd-W@:start:] Start action [0000033-151128171830337-oozie-bigd-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2015-11-28 17:40:20,954 WARN ActionStartXCommand:542 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[0000033-151128171830337-oozie-bigd-W@:start:] [***0000033-151128171830337-oozie-bigd-W@:start:***]Action status=DONE 2015-11-28 17:40:20,954 WARN ActionStartXCommand:542 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[0000033-151128171830337-oozie-bigd-W@:start:] [***0000033-151128171830337-oozie-bigd-W@:start:***]Action updated in DB! 2015-11-28 17:40:21,336 WARN ActionStartXCommand:542 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[0000033-151128171830337-oozie-bigd-W@mapreduce-wordcount-example] ELException in ActionStartXCommand javax.servlet.jsp.el.ELException: variable [outputDir] cannot be resolved at org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106) at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124) at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114) at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274) at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190) at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:175) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:60) at org.apache.oozie.command.XCommand.call(XCommand.java:280) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-11-28 17:40:21,338 WARN ActionStartXCommand:542 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[0000033-151128171830337-oozie-bigd-W@mapreduce-wordcount-example] Failing Job due to failed action [mapreduce-wordcount-example] 2015-11-28 17:40:21,340 WARN LiteWorkflowInstance:542 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[0000033-151128171830337-oozie-bigd-W@mapreduce-wordcount-example] Workflow Failed. Failing node [mapreduce-wordcount-example] 2015-11-28 17:40:21,431 INFO KillXCommand:539 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[-] STARTED WorkflowKillXCommand for jobId=0000033-151128171830337-oozie-bigd-W 2015-11-28 17:40:21,488 INFO KillXCommand:539 - USER[bigdata] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000033-151128171830337-oozie-bigd-W] ACTION[-] ENDED WorkflowKillXCommand for jobId=0000033-151128171830337-oozie-bigd-W
coordinator:
<coordinator-app name="coordinator1" frequency="${frequency}" start="${startTime}" end="${endTime}" timezone="${timezone}" xmlns="uri:oozie:coordinator:0.1"> <datasets> <dataset name="OUT" frequency="${coord:minutes(7)}" initial-instance="2015-11-28T04:00Z" timezone="UTC"> <uri-template> hdfs://localhost:8020/user/bigdata/TimeBasedCoord/Output/${YEAR}-${MONTH}-${DAY}-${HOUR} </uri-template> <done-flag>_SUCCESS</done-flag> </dataset> </datasets> <output-events> <data-out name="coordOutData" dataset="OUT"> <instance>${coord:current(0)}</instance> </data-out> </output-events> <action> <workflow> <app-path>${workflowPath}</app-path> <configuration> <property> <name>outputDir</name> <value>${coord:dataOut('coordOutData')}</value> </property> </configuration> </workflow> </action> </coordinator-app>
Workflow:
<workflow-app xmlns='uri:oozie:workflow:0.1' name='java-main-wf'> <start to='mapreduce-wordcount-example' /> <action name='mapreduce-wordcount-example'> <java> <job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>default</value> </property> </configuration> <main-class>com.test.mypack.WordCountDriver</main-class> <arg>${inputDir}</arg> <arg>${outputDir}</arg> </java> <ok to="end" /> <error to="fail" />
</action> <kill name="fail"> <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name='end' /> </workflow-app>
job.properties:
frequency=10 startTime=2015-11-28T02:00Z endTime=2015-11-30T10:00Z timezone=UTC workflowPath=${nameNode}/user/bigdata/TimeBasedCoord/workflow/ nameNode=hdfs://localhost:8020 jobTracker=localhost:8032 queueName=default inputDir=${nameNode}/user/${user.name}/TimeBasedCoord/Input #outputDir=${nameNode}/user/${user.name}/examples/input-data/rawLogs/ oozie.coord.application.path=${nameNode}/user/${user.name}/TimeBasedCoord/coordinator.xml
03-14-2017 10:37 AM
Hi, could you make it work? Thanks