Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

couldn't able to execute spark jar in cluster mode

avatar
New Contributor

I am working on Spark with scala , I have created a jar and tried to execute in the cluster mode as below spark-submit --class com.test.sales.LoadSalesData \ --master yarn \ --deploy-mode cluster \ --driver-memory 4g \

--executor-memory 2g \

--executor-cores 1 \

test_salesdata_2.10-1.0.jar

but I am getting below error ,

17/01/18 05:15:16 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:17 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:18 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:19 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:20 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:21 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:22 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED) 17/01/18 05:15:23 INFO Client: Application report for application_1483728394968_0141 (state: ACCEPTED)

client token: N/A diagnostics: Application application_1483728394968_0141 failed 2 times due to AM Container for appattempt_1483728394968_0141_000002 exited with exitCode: 1 3

i could able to execute the same code in spark-shell with out any issues.. could you please let me know ,if I am missing anything to run the spark job in cluster mode. thank you !

1 ACCEPTED SOLUTION

avatar
@Satya

yarn application log could give the actual error.

Can you also check if you have set master inside the code?

View solution in original post

5 REPLIES 5

avatar
Super Collaborator

Looks like your job was submitted to Yarn. Can you log information from spark history UI or from Yarn

Regards

Pranay Vyas

avatar
Super Collaborator

Please check from the Yarn RM UI why the application master failed? The AM logs would give you indication depending on exceptions as to what caused the failures . Once that is identified, you can proceed to fix it.

avatar
@Satya

yarn application log could give the actual error.

Can you also check if you have set master inside the code?

avatar
New Contributor

Thank you.. It has been resolved now... i forgot to remove .setMaster("local") in the spark jar.. after removing that job went fine..

avatar
Super Collaborator

Please accept the answer by Sandeep so that this question can be marked as addressed.