Reply
Explorer
Posts: 62
Registered: ‎01-22-2014

Issue on running spark application in Yarn-cluster mode

[ Edited ]

When I execute the following in yarn-client mode its working fine and giving the result properly, but when i try to run in Yarn-cluster mode i am getting error

 

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client /home/abc/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar 10

 The above code works fine, but when i execute the same code in yarn cluster mode i amgetting the following error.

 

14/10/07 09:40:24 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1150
         appId: 1150
         clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service:  }
         appDiagnostics:
         appMasterHost: N/A
         appQueue: root.default
         appMasterRpcPort: -1
         appStartTime: 1412689195537
         yarnAppState: ACCEPTED
         distributedFinalState: UNDEFINED
         appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1150/
         appUser: abc
14/10/07 09:40:25 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1150
         appId: 1150
         clientToAMToken: null
         appDiagnostics: Application application_1412117173893_1150 failed 2 times due to AM Container for appattempt_1412117173893_1150_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
        at org.apache.hadoop.util.Shell.run(Shell.java:424)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

main : command provided 1
main : user is abc
main : requested yarn user is abc

Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
         appMasterHost: N/A
         appQueue: root.default
         appMasterRpcPort: -1
         appStartTime: 1412689195537
         yarnAppState: FAILED
         distributedFinalState: FAILED
         appTrackingUrl: spark.abcd.com:8088/cluster/app/application_1412117173893_1150
         appUser: abc

 Where may be the problem? sometimes when i try to execute in yarn-cluster mode i am getting the following , but i dint see any result

14/10/08 01:51:57 INFO Client: Application report from ASM:
         application identifier: application_1412117173893_1442
         appId: 1442
         clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service:  }
         appDiagnostics:
         appMasterHost: spark.abcd.com
         appQueue: root.default
         appMasterRpcPort: 0
         appStartTime: 1412747485673
         yarnAppState: FINISHED
         distributedFinalState: SUCCEEDED
         appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1442/A
         appUser: abc

 Thanks

New Contributor
Posts: 7
Registered: ‎11-12-2014

Re: Issue on running spark application in Yarn-cluster mode

I met the same issue now. Have you find out the way to solve it? 

Explorer
Posts: 8
Registered: ‎12-21-2014

Re: Issue on running spark application in Yarn-cluster mode

try the below command to get the detailed log.

 

$HADOOP_HOME/bin/yarn logs -applicationId application_1419229907721_0010

 

 

Contributor
Posts: 31
Registered: ‎10-31-2014

Re: Issue on running spark application in Yarn-cluster mode

You are not specifying the jar that contains that class (the examples jar).

It could be that the jar is included automatically in local mode but not in the yarn classpath.

 

Have a loot at the nodemanager log for the node that tried to run it to veryfy if it's a classpath issue

Cloudera Employee
Posts: 306
Registered: ‎01-16-2014

Re: Issue on running spark application in Yarn-cluster mode

This error:

 

main : command provided 1
main : user is abc
main : requested yarn user is abc

Container exited with a non-zero exit code 1

Looks like the exit code from the linux container executor.

In cluster mode the driver runs inside the same container as the Application Master which makes a difference.

 

As other people have said already get the logs from the containers by running:

  yarn logs -applicationId APPID

Make sure that you run it as the user "abc" (same as the user that executes the spark command).

 

Wilfred

New Contributor
Posts: 3
Registered: ‎10-08-2015

Re: Issue on running spark application in Yarn-cluster mode

[ Edited ]

I am running my spark streaming application using spark-submit on yarn-cluster. When I run it on local mode it is working fine. But when I try to run it on yarn-cluster using spark-submit, it runs for some time and then exits with following execption.

 

Diagnostics:Exception from container-launch.Container id: container_1435576266959_1208_02_000002Exit code:13Stack trace:ExitCodeException exitCode=13: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

 

Any help will be appreciated.

Contributor
Posts: 31
Registered: ‎10-31-2014

Re: Issue on running spark application in Yarn-cluster mode

You'll have to view the logs of the YARN node running the executor, it's not very obvious how to see the logs in the YARN console.

If I had to make a wild guess, I would say the user you are running the job with doesn't exit in the node running the executor.

New Contributor
Posts: 1
Registered: ‎01-29-2016

Re: Issue on running spark application in Yarn-cluster mode

Faced this issue and your comment helped to fix. Out of curiosity, how from logs, you suspect that this could be the issue.
Contributor
Posts: 31
Registered: ‎10-31-2014

Re: Issue on running spark application in Yarn-cluster mode

An exit code -1 means Java crashed, normally that is due to classpath or memory settings

New Contributor
Posts: 1
Registered: ‎06-12-2016

Re: Issue on running spark application in Yarn-cluster mode

How was this fixed?