Reply
New Contributor
Posts: 2
Registered: ‎03-16-2017

Spark Sql job can't run on Hue Oozie

Hi everyone, In my company, we use command line mode to install cdh components. Everything works fine but Spark Sql job can't run on Hue Oozie. And I tried spark-submit, it worked. It seems oozie can't read hive context or hadoop context while it starts a spark job. So I wonder whether there are some configuration for oozie to recognize hadoop hive context. Here is my environment: CDH: 5.9.0 OS: CentOS 7 Hadoop, Oozie, Hive, Spark version is related to cdh 5.9.0. Oozie is installed on the namenode. Hue is installed outside of hadoop cluster, but it have all necessary hadoop environment parameters. In Hue, i config the spark job as follows: Spark Master: yarn Mode: client App name: test Jar/py name: spark-test.jar Main class: com.ron.Main Files: /user/ron/spark-test.jar Options list: --file ${nameNode}/user/ron/hive-site.xml Here is the detail log when I execute the workflow: 2017-03-17 13:26:17,027 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@:start:] Start action [0000005-170317113628323-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2017-03-17 13:26:17,028 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@:start:] [***0000005-170317113628323-oozie-oozi-W@:start:***]Action status=DONE 2017-03-17 13:26:17,028 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@:start:] [***0000005-170317113628323-oozie-oozi-W@:start:***]Action updated in DB! 2017-03-17 13:26:17,155 INFO WorkflowNotificationXCommand:520 - SERVER[master01.yscredit.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000005-170317113628323-oozie-oozi-W 2017-03-17 13:26:17,155 INFO WorkflowNotificationXCommand:520 - SERVER[master01.yscredit.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000005-170317113628323-oozie-oozi-W@:start: 2017-03-17 13:26:17,199 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] Start action [0000005-170317113628323-oozie-oozi-W@spark-7e65] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2017-03-17 13:26:19,228 INFO SparkActionExecutor:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] checking action, hadoop job ID [job_1487813931557_1131] status [RUNNING] 2017-03-17 13:26:19,232 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] [***0000005-170317113628323-oozie-oozi-W@spark-7e65***]Action status=RUNNING 2017-03-17 13:26:19,232 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] [***0000005-170317113628323-oozie-oozi-W@spark-7e65***]Action updated in DB! 2017-03-17 13:26:19,254 INFO WorkflowNotificationXCommand:520 - SERVER[master01.yscredit.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] No Notification URL is defined. Therefore nothing to notify for job 0000005-170317113628323-oozie-oozi-W@spark-7e65 2017-03-17 13:26:37,816 INFO CallbackServlet:520 - SERVER[master01.yscredit.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] callback for action [0000005-170317113628323-oozie-oozi-W@spark-7e65] 2017-03-17 13:26:38,225 INFO SparkActionExecutor:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] action completed, external ID [job_1487813931557_1131] 2017-03-17 13:26:38,235 WARN SparkActionExecutor:523 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Yarn application has already ended! It might have been killed or unable to launch application master. 2017-03-17 13:26:38,235 WARN SparkActionExecutor:523 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] Launcher exception: Yarn application has already ended! It might have been killed or unable to launch application master. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:157) at org.apache.spark.SparkContext.(SparkContext.scala:542) at com.yscredit.Main$.main(Main.scala:21) at com.yscredit.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:256) at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:207) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49) at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:231) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2017-03-17 13:26:38,318 INFO ActionEndXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] ERROR is considered as FAILED for SLA 2017-03-17 13:26:38,416 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@Kill] Start action [0000005-170317113628323-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2017-03-17 13:26:38,416 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@Kill] [***0000005-170317113628323-oozie-oozi-W@Kill***]Action status=DONE 2017-03-17 13:26:38,416 INFO ActionStartXCommand:520 - SERVER[master01.yscredit.com] USER[linshirong] GROUP[-] TOKEN[] APP[spark-test] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@Kill] [***0000005-170317113628323-oozie-oozi-W@Kill***]Action updated in DB! 2017-03-17 13:26:38,619 INFO WorkflowNotificationXCommand:520 - SERVER[master01.yscredit.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000005-170317113628323-oozie-oozi-W@Kill 2017-03-17 13:26:38,619 INFO WorkflowNotificationXCommand:520 - SERVER[master01.yscredit.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000005-170317113628323-oozie-oozi-W 2017-03-17 13:26:38,619 INFO WorkflowNotificationXCommand:520 - SERVER[master01.yscredit.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000005-170317113628323-oozie-oozi-W] ACTION[0000005-170317113628323-oozie-oozi-W@spark-7e65] No Notification URL is defined. Therefore nothing to notify for job 0000005-170317113628323-oozie-oozi-W@spark-7e65 And here is the workflow definition: Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}] ${jobTracker} ${nameNode} yarn client test com.ron.Main spark-test.jar --files ${nameNode}/user/ron/hive-site.xml /user/ron/spark-test.jar#spark-test.jar
Announcements