Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie MR Yarn Exception When Running Workflow

Oozie MR Yarn Exception When Running Workflow

New Contributor

I am trying to run a simple workflow I created in Hue that just calls a Spark job with no arguments, but am getting an exception when running it. This is running on a Google DataProc cluser and using Oozie 4.2.0 with Hue 3.9.

 

Here is the code for my workflow:

 

<workflow-app name="Drug_File_Price_Generation" xmlns="uri:oozie:workflow:0.5">
    <start to="spark-22ca"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="spark-22ca">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>local[*]</master>
            <mode>client</mode>
            <name>MySpark</name>
              <class>com.company.drug.GenerateDrugPriceFile</class>
            <jar>gs://customers/jar/DrugFileImport-assembly-1.0.jar</jar>
        </spark>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

 

 

Here is the output from the log I am receiving:

 

 

2016-03-22 19:27:27,927  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] Start action [0000011-160322181838855-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action status=DONE
2016-03-22 19:27:27,930  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] [***0000011-160322181838855-oozie-oozi-W@:start:***]Action updated in DB!
2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W
2016-03-22 19:27:27,963  INFO WorkflowNotificationXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000011-160322181838855-oozie-oozi-W@:start:
2016-03-22 19:27:27,976  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:27:32,670  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
	at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 10 more
2016-03-22 19:27:32,671  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Next Retry, Attempt Number [1] in [10,000] milliseconds
2016-03-22 19:27:42,688  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:27:47,343  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 8 more
2016-03-22 19:27:47,344  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Next Retry, Attempt Number [2] in [10,000] milliseconds
2016-03-22 19:27:57,420  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:28:02,172  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 8 more
2016-03-22 19:28:02,173  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Next Retry, Attempt Number [3] in [10,000] milliseconds
2016-03-22 19:28:12,212  INFO ActionStartXCommand:520 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Start action [0000011-160322181838855-oozie-oozi-W@spark-22ca] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-22 19:28:16,947  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Error starting action [spark-22ca]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
	at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
	at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:454)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:437)
	at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:435)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:435)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1329)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1080)
	... 8 more
2016-03-22 19:28:16,948  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Exceeded max retry count [3]. Suspending Job
2016-03-22 19:28:16,948  WARN ActionStartXCommand:523 - SERVER[cluster-1-m.c.server-name.internal] USER[sayle.matthews] GROUP[-] TOKEN[] APP[Drug_File_Price_Generation] JOB[0000011-160322181838855-oozie-oozi-W] ACTION[0000011-160322181838855-oozie-oozi-W@spark-22ca] Suspending Workflow Job id=0000011-160322181838855-oozie-oozi-W

 

I have made sure that the mapreduce.framework.name is for yarn. I found a few solutions online so far, but all of those revolved around issues with the workflow XML. I have rebuilt by hand and it is doing the same thing that way as well.

 

Anybody ever seen this or have a solution for it?

1 REPLY 1
Highlighted

Re: Oozie MR Yarn Exception When Running Workflow

Rising Star

Your workflow XML is incomplete, so I'm thinking you may not have pasted it correctly?

  1. There is now closing workflow-app tag
  2. There is no "spark-22ca" action from which the workflow will start to
  3. There is no "ok to" tag
  4. etc.

Perhaps take a look at the workflow specification for some guidance: (link)

Here's another link to the specification for the Spark action: (link)

 

Hope this helps,

Tyler

Don't have an account?
Coming from Hortonworks? Activate your account here