question Re: couldn't able to execute spark jar in cluster mode in Archives of Support Questions (Read Only)

couldn't able to execute spark jar in cluster mode

iamsparkman — Wed, 18 Jan 2017 20:08:22 GMT

I am working on Spark with scala , I have created a jar and tried to execute in the cluster mode as below spark-submit --class com.test.sales.LoadSalesData \ --master yarn \ --deploy-mode cluster \ --driver-memory 4g \

--executor-memory 2g \

--executor-cores 1 \

test_salesdata_2.10-1.0.jar

but I am getting below error ,

17/01/18 05:15:16 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:17 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:18 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:19 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:20 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:21 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:22 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:23 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED)

client token: N/A diagnostics: Application application_1483728394968_0141 failed 2 times due to AM Container for appattempt_1483728394968_0141_000002 exited with exitCode: 1 3

i could able to execute the same code in spark-shell with out any issues.. could you please let me know ,if I am missing anything to run the spark job in cluster mode. thank you !

Re: couldn't able to execute spark jar in cluster mode

PranayV — Thu, 19 Jan 2017 00:47:25 GMT

Looks like your job was submitted to Yarn. Can you log information from spark history UI or from Yarn

Regards

Pranay Vyas

Re: couldn't able to execute spark jar in cluster mode

skurup — Sat, 21 Jan 2017 05:34:10 GMT

Please check from the Yarn RM UI why the application master failed? The AM logs would give you indication depending on exceptions as to what caused the failures . Once that is identified, you can proceed to fix it.

Re: couldn't able to execute spark jar in cluster mode

sandyy006 — Wed, 25 Jan 2017 04:36:52 GMT

@Satya

yarn application log could give the actual error.

Can you also check if you have set master inside the code?

Re: couldn't able to execute spark jar in cluster mode

iamsparkman — Thu, 26 Jan 2017 20:46:52 GMT

Thank you.. It has been resolved now... i forgot to remove .setMaster("local") in the spark jar.. after removing that job went fine..

Re: couldn't able to execute spark jar in cluster mode

skurup — Thu, 26 Jan 2017 23:34:14 GMT

Please accept the answer by Sandeep so that this question can be marked as addressed.