Created on 10-07-2014 11:20 PM - edited 09-16-2022 02:09 AM
When I execute the following in yarn-client mode its working fine and giving the result properly, but when i try to run in Yarn-cluster mode i am getting error
spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client /home/abc/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar 10
The above code works fine, but when i execute the same code in yarn cluster mode i amgetting the following error.
14/10/07 09:40:24 INFO Client: Application report from ASM:
application identifier: application_1412117173893_1150
appId: 1150
clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service: }
appDiagnostics:
appMasterHost: N/A
appQueue: root.default
appMasterRpcPort: -1
appStartTime: 1412689195537
yarnAppState: ACCEPTED
distributedFinalState: UNDEFINED
appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1150/
appUser: abc
14/10/07 09:40:25 INFO Client: Application report from ASM:
application identifier: application_1412117173893_1150
appId: 1150
clientToAMToken: null
appDiagnostics: Application application_1412117173893_1150 failed 2 times due to AM Container for appattempt_1412117173893_1150_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
main : command provided 1
main : user is abc
main : requested yarn user is abc
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
appMasterHost: N/A
appQueue: root.default
appMasterRpcPort: -1
appStartTime: 1412689195537
yarnAppState: FAILED
distributedFinalState: FAILED
appTrackingUrl: spark.abcd.com:8088/cluster/app/application_1412117173893_1150
appUser: abcWhere may be the problem? sometimes when i try to execute in yarn-cluster mode i am getting the following , but i dint see any result
14/10/08 01:51:57 INFO Client: Application report from ASM:
application identifier: application_1412117173893_1442
appId: 1442
clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service: }
appDiagnostics:
appMasterHost: spark.abcd.com
appQueue: root.default
appMasterRpcPort: 0
appStartTime: 1412747485673
yarnAppState: FINISHED
distributedFinalState: SUCCEEDED
appTrackingUrl: http://spark.abcd.com:8088/proxy/application_1412117173893_1442/A
appUser: abcThanks