Reply
New Contributor
Posts: 1
Registered: ‎03-22-2016

Oozie MR Yarn Exception When Running Workflow

[ Edited ]

I am trying to run a simple workflow I created in Hue that just calls a Spark job with no arguments, but am getting an exception when running it. This is running on a Google DataProc cluser and using Oozie 4.2.0 with Hue 3.9.

 

Here is the code for my workflow:

 

<workflow-app name="Drug_File_Price_Generation" xmlns="uri:oozie:workflow:0.5">
    <start to="spark-22ca"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="spark-22ca">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>local[*]</master>
            <mode>client</mode>
            <name>MySpark</name>
              <class>com.company.drug.GenerateDrugPriceFile</class>
            <jar>gs://customers/jar/DrugFileImport-assembly-1.0.jar</jar>
        </spark>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

 

 

Here is the output from the log I am receiving:

 

 

2016-03-22 19:27:27,927  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] Start action [0000011-160322181838855-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action status=DONE
2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action updated in DB!
2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W
2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W@:start:
2016-03-22 19:27:27,976  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:27:32,670  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 10 more
2016-03-22 19:27:32,671  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Next Retry, Attempt Number [1] in [10,000] milliseconds
2016-03-22 19:27:42,688  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:27:47,343  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 8 more
2016-03-22 19:27:47,344  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Next Retry, Attempt Number [2] in [10,000] milliseconds
2016-03-22 19:27:57,420  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:28:02,172  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 8 more
2016-03-22 19:28:02,173  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Next Retry, Attempt Number [3] in [10,000] milliseconds
2016-03-22 19:28:12,212  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:28:16,947  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 8 more
2016-03-22 19:28:16,948  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Exceeded max retry count [3]. Suspending Job
2016-03-22 19:28:16,948  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Suspending Workflow Job id=0000011-160322181838855-oozie-oozi-W

 

I have made sure that the mapreduce.framework.name is for yarn. I found a few solutions online so far, but all of those revolved around issues with the workflow XML. I have rebuilt by hand and it is doing the same thing that way as well.

 

Anybody ever seen this or have a solution for it?

Contributor
Posts: 50
Registered: ‎08-05-2015

Re: Oozie MR Yarn Exception When Running Workflow

Your workflow XML is incomplete, so I'm thinking you may not have pasted it correctly?

  1. There is now closing workflow-app tag
  2. There is no "spark-22ca" action from which the workflow will start to
  3. There is no "ok to" tag
  4. etc.

Perhaps take a look at the workflow specification for some guidance: (link)

Here's another link to the specification for the Spark action: (link)

 

Hope this helps,

Tyler

Announcements