Issue on running spark application in Yarn-cluster mode

[#1]

When I execute the following in yarn-client mode its working fine and giving the result properly, but when i try to run in Yarn-cluster mode i am getting error

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client /home/abc/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar 10

 The above code works fine, but when i execute the same code in yarn cluster mode i amgetting the following error.

14/10/07 09:40:24 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1150
         appId: 1150
         clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service:  }
         appDiagnostics:
         appMasterHost: N/A
         appQueue: root.default
         appMasterRpcPort: -1
         appStartTime: 1412689195537
         yarnAppState: ACCEPTED
         distributedFinalState: UNDEFINED
         appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1150/
         appUser: abc
14/10/07 09:40:25 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1150
         appId: 1150
         clientToAMToken: null
         appDiagnostics: Application application_1412117173893_1150 failed 2 times due to AM Container for appattempt_1412117173893_1150_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
        at org.apache.hadoop.util.Shell.run(Shell.java:424)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

main : command provided 1
main : user is abc
main : requested yarn user is abc

Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
         appMasterHost: N/A
         appQueue: root.default
         appMasterRpcPort: -1
         appStartTime: 1412689195537
         yarnAppState: FAILED
         distributedFinalState: FAILED
         appTrackingUrl: spark.abcd.com:8088/cluster/app/application_1412117173893_1150
         appUser: abc

 Where may be the problem? sometimes when i try to execute in yarn-cluster mode i am getting the following , but i dint see any result

14/10/08 01:51:57 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1442
         appId: 1442
         clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service:  }
         appDiagnostics:
         appMasterHost: spark.abcd.com
         appQueue: root.default
         appMasterRpcPort: 0
         appStartTime: 1412747485673
         yarnAppState: FINISHED
         distributedFinalState: SUCCEEDED
         appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1442/A
         appUser: abc

 Thanks

[#2]

I met the same issue now. Have you find out the way to solve it? 

[#3]

try the below command to get the detailed log.

$HADOOP_HOME/bin/yarn logs -applicationId application_1419229907721_0010

[#4]

You are not specifying the jar that contains that class (the examples jar).

It could be that the jar is included automatically in local mode but not in the yarn classpath.

Have a loot at the nodemanager log for the node that tried to run it to veryfy if it's a classpath issue

[#5]

Faced this issue and your comment helped to fix. Out of curiosity, how from logs, you suspect that this could be the issue.

[#6]

An exit code -1 means Java crashed, normally that is due to classpath or memory settings

[#7]

How was this fixed?

[#8]

For the application ownload the application logs and check what the error is in the logs:

  yarn application -applicationId APPID -appOwner USERID

Check the exit codes of the application and you should be able to tell in a bit more detail what is going on.

Wilfred

[#9]

This error:

main : command provided 1
main : user is abc
main : requested yarn user is abc

Container exited with a non-zero exit code 1

Looks like the exit code from the linux container executor.

In cluster mode the driver runs inside the same container as the Application Master which makes a difference.

As other people have said already get the logs from the containers by running:

  yarn logs -applicationId APPID

Make sure that you run it as the user "abc" (same as the user that executes the spark command).

Wilfred

[#10]

I am running my spark streaming application using spark-submit on yarn-cluster. When I run it on local mode it is working fine. But when I try to run it on yarn-cluster using spark-submit, it runs for some time and then exits with following execption.

Diagnostics:Exception from container-launch.Container id: container_1435576266959_1208_02_000002Exit code:13Stack trace:ExitCodeException exitCode=13: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

Any help will be appreciated.