Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issue on running spark application in Yarn-cluster mode

Issue on running spark application in Yarn-cluster mode

Explorer

When I execute the following in yarn-client mode its working fine and giving the result properly, but when i try to run in Yarn-cluster mode i am getting error

 

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client /home/abc/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar 10

 The above code works fine, but when i execute the same code in yarn cluster mode i amgetting the following error.

 

14/10/07 09:40:24 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1150
         appId: 1150
         clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service:  }
         appDiagnostics:
         appMasterHost: N/A
         appQueue: root.default
         appMasterRpcPort: -1
         appStartTime: 1412689195537
         yarnAppState: ACCEPTED
         distributedFinalState: UNDEFINED
         appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1150/
         appUser: abc
14/10/07 09:40:25 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1150
         appId: 1150
         clientToAMToken: null
         appDiagnostics: Application application_1412117173893_1150 failed 2 times due to AM Container for appattempt_1412117173893_1150_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
        at org.apache.hadoop.util.Shell.run(Shell.java:424)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

main : command provided 1
main : user is abc
main : requested yarn user is abc

Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
         appMasterHost: N/A
         appQueue: root.default
         appMasterRpcPort: -1
         appStartTime: 1412689195537
         yarnAppState: FAILED
         distributedFinalState: FAILED
         appTrackingUrl: spark.abcd.com:8088/cluster/app/application_1412117173893_1150
         appUser: abc

 Where may be the problem? sometimes when i try to execute in yarn-cluster mode i am getting the following , but i dint see any result

14/10/08 01:51:57 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1442
         appId: 1442
         clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service:  }
         appDiagnostics:
         appMasterHost: spark.abcd.com
         appQueue: root.default
         appMasterRpcPort: 0
         appStartTime: 1412747485673
         yarnAppState: FINISHED
         distributedFinalState: SUCCEEDED
         appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1442/A
         appUser: abc

 Thanks

13 REPLIES 13

Re: Issue on running spark application in Yarn-cluster mode

New Contributor

I met the same issue now. Have you find out the way to solve it? 

Re: Issue on running spark application in Yarn-cluster mode

Explorer

try the below command to get the detailed log.

 

$HADOOP_HOME/bin/yarn logs -applicationId application_1419229907721_0010

 

 

Re: Issue on running spark application in Yarn-cluster mode

Contributor

You are not specifying the jar that contains that class (the examples jar).

It could be that the jar is included automatically in local mode but not in the yarn classpath.

 

Have a loot at the nodemanager log for the node that tried to run it to veryfy if it's a classpath issue

Re: Issue on running spark application in Yarn-cluster mode

New Contributor
Faced this issue and your comment helped to fix. Out of curiosity, how from logs, you suspect that this could be the issue.

Re: Issue on running spark application in Yarn-cluster mode

Contributor

An exit code -1 means Java crashed, normally that is due to classpath or memory settings

Re: Issue on running spark application in Yarn-cluster mode

New Contributor

How was this fixed?

Re: Issue on running spark application in Yarn-cluster mode

Super Collaborator

For the application ownload the application logs and check what the error is in the logs:

  yarn application -applicationId APPID -appOwner USERID

Check the exit codes of the application and you should be able to tell in a bit more detail what is going on.

 

Wilfred

Re: Issue on running spark application in Yarn-cluster mode

Super Collaborator

This error:

 

main : command provided 1
main : user is abc
main : requested yarn user is abc

Container exited with a non-zero exit code 1

Looks like the exit code from the linux container executor.

In cluster mode the driver runs inside the same container as the Application Master which makes a difference.

 

As other people have said already get the logs from the containers by running:

  yarn logs -applicationId APPID

Make sure that you run it as the user "abc" (same as the user that executes the spark command).

 

Wilfred

Highlighted

Re: Issue on running spark application in Yarn-cluster mode

New Contributor

I am running my spark streaming application using spark-submit on yarn-cluster. When I run it on local mode it is working fine. But when I try to run it on yarn-cluster using spark-submit, it runs for some time and then exits with following execption.

 

Diagnostics:Exception from container-launch.Container id: container_1435576266959_1208_02_000002Exit code:13Stack trace:ExitCodeException exitCode=13: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

 

Any help will be appreciated.